Oct 31 00:11:10.528237 ip-10-0-137-56 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Oct 31 00:11:10.528247 ip-10-0-137-56 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Oct 31 00:11:10.528254 ip-10-0-137-56 systemd[1]: kubelet.service: Failed with result 'resources'. Oct 31 00:11:10.528482 ip-10-0-137-56 systemd[1]: Failed to start Kubernetes Kubelet. Oct 31 00:11:20.733012 ip-10-0-137-56 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Oct 31 00:11:20.733034 ip-10-0-137-56 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot af8b0a00c27e4d39bc3a3bf1c043eb45 -- Oct 31 00:13:52.649698 ip-10-0-137-56 systemd[1]: Starting Kubernetes Kubelet... Oct 31 00:13:53.039529 ip-10-0-137-56 kubenswrapper[2581]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 00:13:53.039529 ip-10-0-137-56 kubenswrapper[2581]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 31 00:13:53.039529 ip-10-0-137-56 kubenswrapper[2581]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 00:13:53.039529 ip-10-0-137-56 kubenswrapper[2581]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 00:13:53.042836 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.042717 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:13:53.042836 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.042836 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:13:53.042836 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.042841 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:13:53.042928 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.042844 2581 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:13:53.042928 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.042847 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:13:53.042928 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.042849 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:13:53.042928 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.042852 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:13:53.042928 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.042855 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:13:53.042928 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.042857 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:13:53.042928 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.042860 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:13:53.042928 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.042863 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:13:53.042928 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.042866 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:13:53.042928 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.042868 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:13:53.042928 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.042871 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:13:53.042928 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.042873 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:13:53.042928 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.042876 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:13:53.042928 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.042885 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:13:53.042928 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.042887 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:13:53.042928 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.042890 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:13:53.042928 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.042893 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:13:53.042928 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.042895 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:13:53.042928 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.042898 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:13:53.042928 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.042900 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:13:53.043415 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.042903 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:13:53.043415 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.042905 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:13:53.043415 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.042908 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:13:53.043415 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.042910 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:13:53.043415 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.042913 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:13:53.043415 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.042916 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:13:53.043415 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.042919 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:13:53.043415 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.042922 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:13:53.043415 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.042924 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:13:53.043415 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.042927 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:13:53.043415 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.042930 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:13:53.043415 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.042933 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:13:53.043415 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.042936 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:13:53.043415 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.042939 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:13:53.043415 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.042941 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:13:53.043415 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.042944 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:13:53.043415 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.042947 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:13:53.043415 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.042950 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:13:53.043415 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.042954 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:13:53.043854 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.042958 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:13:53.043854 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.042961 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:13:53.043854 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.042964 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:13:53.043854 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.042967 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:13:53.043854 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.042970 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:13:53.043854 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.042972 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:13:53.043854 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.042975 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:13:53.043854 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.042978 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:13:53.043854 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.042980 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:13:53.043854 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.042983 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:13:53.043854 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.042985 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:13:53.043854 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.042988 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:13:53.043854 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.042990 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:13:53.043854 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.042993 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:13:53.043854 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.042996 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:13:53.043854 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.042998 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:13:53.043854 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.043001 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:13:53.043854 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.043003 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:13:53.043854 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.043006 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:13:53.043854 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.043008 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:13:53.044328 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.043011 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:13:53.044328 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.043013 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:13:53.044328 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.043016 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:13:53.044328 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.043019 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:13:53.044328 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.043022 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:13:53.044328 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.043024 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:13:53.044328 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.043027 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:13:53.044328 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.043029 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:13:53.044328 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.043032 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:13:53.044328 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.043034 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:13:53.044328 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.043037 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:13:53.044328 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.043040 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:13:53.044328 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.043042 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:13:53.044328 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.043045 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:13:53.044328 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.043047 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:13:53.044328 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.043050 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:13:53.044328 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.043053 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:13:53.044328 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.043056 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:13:53.044328 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.043059 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:13:53.044796 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.043061 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:13:53.044796 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.043064 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:13:53.044796 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.043066 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:13:53.044796 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.043069 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:13:53.044796 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.043071 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:13:53.044796 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.043074 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:13:53.044796 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.043076 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:13:53.044796 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.043079 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:13:53.044796 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.043081 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:13:53.044796 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.043083 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:13:53.044796 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.043086 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:13:53.044796 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.043088 2581 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:13:53.044796 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.043091 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:13:53.044796 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.043093 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:13:53.044796 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.043098 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:13:53.044796 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.043102 2581 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:13:53.044796 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.043108 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:13:53.044796 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.043112 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:13:53.044796 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.043115 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:13:53.044796 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.043118 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:13:53.045298 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.043121 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:13:53.045298 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.043124 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:13:53.045298 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.043127 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:13:53.045298 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.043130 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:13:53.045298 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.043132 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:13:53.045298 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.043966 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:13:53.045298 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.043973 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:13:53.045298 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.043977 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:13:53.045298 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.043979 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:13:53.045298 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.043983 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:13:53.045298 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.043986 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:13:53.045298 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.043989 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:13:53.045298 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.043992 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:13:53.045298 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.043994 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:13:53.045298 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.043997 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:13:53.045298 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044000 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:13:53.045298 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044003 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:13:53.045298 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044005 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:13:53.045298 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044008 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:13:53.045776 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044011 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:13:53.045776 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044013 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:13:53.045776 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044016 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:13:53.045776 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044018 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:13:53.045776 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044021 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:13:53.045776 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044023 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:13:53.045776 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044026 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:13:53.045776 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044029 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:13:53.045776 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044031 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:13:53.045776 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044034 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:13:53.045776 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044036 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:13:53.045776 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044039 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:13:53.045776 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044041 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:13:53.045776 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044044 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:13:53.045776 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044047 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:13:53.045776 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044049 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:13:53.045776 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044053 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:13:53.045776 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044056 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:13:53.045776 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044058 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:13:53.045776 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044061 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:13:53.046249 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044063 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:13:53.046249 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044066 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:13:53.046249 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044068 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:13:53.046249 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044071 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:13:53.046249 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044073 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:13:53.046249 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044076 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:13:53.046249 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044078 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:13:53.046249 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044081 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:13:53.046249 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044083 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:13:53.046249 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044086 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:13:53.046249 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044088 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:13:53.046249 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044091 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:13:53.046249 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044093 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:13:53.046249 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044095 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:13:53.046249 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044098 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:13:53.046249 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044101 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:13:53.046249 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044103 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:13:53.046249 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044106 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:13:53.046249 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044109 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:13:53.046708 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044112 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:13:53.046708 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044115 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:13:53.046708 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044117 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:13:53.046708 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044119 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:13:53.046708 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044122 2581 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:13:53.046708 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044124 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:13:53.046708 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044127 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:13:53.046708 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044130 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:13:53.046708 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044142 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:13:53.046708 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044145 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:13:53.046708 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044148 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:13:53.046708 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044151 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:13:53.046708 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044153 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:13:53.046708 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044156 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:13:53.046708 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044158 2581 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:13:53.046708 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044161 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:13:53.046708 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044165 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:13:53.046708 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044169 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:13:53.046708 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044173 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:13:53.046708 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044176 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:13:53.047183 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044179 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:13:53.047183 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044181 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:13:53.047183 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044184 2581 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:13:53.047183 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044187 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:13:53.047183 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044189 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:13:53.047183 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044192 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:13:53.047183 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044194 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:13:53.047183 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044197 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:13:53.047183 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044199 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:13:53.047183 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044202 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:13:53.047183 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044204 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:13:53.047183 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044207 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:13:53.047183 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044213 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:13:53.047183 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044215 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:13:53.047183 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044218 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:13:53.047183 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044220 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:13:53.047183 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044223 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:13:53.047183 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044225 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:13:53.047183 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044228 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:13:53.047652 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044230 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:13:53.047652 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044233 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:13:53.047652 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044235 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:13:53.047652 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044238 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:13:53.047652 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044241 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:13:53.047652 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044243 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:13:53.047652 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044246 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:13:53.047652 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044250 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:13:53.047652 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044255 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:13:53.047652 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044258 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:13:53.047652 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044261 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:13:53.047652 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044264 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:13:53.047652 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044268 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:13:53.047652 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044271 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:13:53.047652 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044856 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:13:53.047652 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044863 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:13:53.047652 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044867 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:13:53.047652 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044870 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:13:53.047652 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044872 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:13:53.048224 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044875 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:13:53.048224 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044878 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:13:53.048224 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044881 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:13:53.048224 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044883 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:13:53.048224 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044886 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:13:53.048224 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044888 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:13:53.048224 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044892 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:13:53.048224 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044894 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:13:53.048224 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044897 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:13:53.048224 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044899 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:13:53.048224 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044902 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:13:53.048224 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044904 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:13:53.048224 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044906 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:13:53.048224 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044909 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:13:53.048224 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044911 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:13:53.048224 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044914 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:13:53.048224 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044917 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:13:53.048224 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044919 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:13:53.048224 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044922 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:13:53.048687 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044924 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:13:53.048687 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044927 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:13:53.048687 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044929 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:13:53.048687 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044932 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:13:53.048687 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044934 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:13:53.048687 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044936 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:13:53.048687 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044939 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:13:53.048687 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044942 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:13:53.048687 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044944 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:13:53.048687 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044946 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:13:53.048687 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044949 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:13:53.048687 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044951 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:13:53.048687 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044954 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:13:53.048687 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044956 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:13:53.048687 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044958 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:13:53.048687 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044961 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:13:53.048687 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044963 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:13:53.048687 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044965 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:13:53.048687 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044968 2581 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:13:53.048687 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044971 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:13:53.049155 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044973 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:13:53.049155 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044976 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:13:53.049155 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044980 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:13:53.049155 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044984 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:13:53.049155 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044987 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:13:53.049155 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044990 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:13:53.049155 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044993 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:13:53.049155 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.044997 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:13:53.049155 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.045002 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:13:53.049155 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.045005 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:13:53.049155 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.045007 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:13:53.049155 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.045010 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:13:53.049155 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.045012 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:13:53.049155 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.045015 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:13:53.049155 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.045017 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:13:53.049155 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.045020 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:13:53.049155 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.045022 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:13:53.049155 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.045024 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:13:53.049155 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.045027 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:13:53.049640 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.045037 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:13:53.049640 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.045039 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:13:53.049640 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.045042 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:13:53.049640 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.045044 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:13:53.049640 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.045047 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:13:53.049640 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.045049 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:13:53.049640 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.045051 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:13:53.049640 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.045054 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:13:53.049640 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.045056 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:13:53.049640 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.045059 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:13:53.049640 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.045061 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:13:53.049640 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.045063 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:13:53.049640 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.045066 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:13:53.049640 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.045069 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:13:53.049640 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.045071 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:13:53.049640 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.045074 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:13:53.049640 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.045076 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:13:53.049640 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.045079 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:13:53.049640 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.045081 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:13:53.049640 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.045084 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:13:53.050157 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.045086 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:13:53.050157 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.045090 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:13:53.050157 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.045093 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:13:53.050157 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.045095 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:13:53.050157 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.045097 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:13:53.050157 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.045100 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:13:53.050157 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.045102 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:13:53.050157 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.045105 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:13:53.050157 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.045107 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:13:53.050157 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.045110 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:13:53.050157 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.045112 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:13:53.050157 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.045115 2581 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:13:53.050157 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.045117 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:13:53.050157 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.045127 2581 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:13:53.050157 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.045130 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:13:53.050157 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.045133 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:13:53.050157 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.045135 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:13:53.050157 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.045137 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:13:53.050157 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.045140 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:13:53.050157 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.045142 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:13:53.050653 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.045147 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:13:53.050653 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.045150 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:13:53.050653 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.045153 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:13:53.050653 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046052 2581 flags.go:64] FLAG: --address="0.0.0.0" Oct 31 00:13:53.050653 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046063 2581 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 31 00:13:53.050653 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046069 2581 flags.go:64] FLAG: --anonymous-auth="true" Oct 31 00:13:53.050653 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046074 2581 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 31 00:13:53.050653 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046079 2581 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 31 00:13:53.050653 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046083 2581 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 31 00:13:53.050653 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046087 2581 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 31 00:13:53.050653 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046093 2581 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 31 00:13:53.050653 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046096 2581 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 31 00:13:53.050653 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046099 2581 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 31 00:13:53.050653 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046104 2581 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 31 00:13:53.050653 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046107 2581 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 31 00:13:53.050653 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046110 2581 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 31 00:13:53.050653 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046113 2581 flags.go:64] FLAG: --cgroup-root="" Oct 31 00:13:53.050653 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046116 2581 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 31 00:13:53.050653 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046119 2581 flags.go:64] FLAG: --client-ca-file="" Oct 31 00:13:53.050653 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046122 2581 flags.go:64] FLAG: --cloud-provider="external" Oct 31 00:13:53.050653 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046125 2581 flags.go:64] FLAG: --cluster-dns="[]" Oct 31 00:13:53.050653 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046130 2581 flags.go:64] FLAG: --cluster-domain="" Oct 31 00:13:53.050653 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046133 2581 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 31 00:13:53.051202 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046136 2581 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 31 00:13:53.051202 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046139 2581 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 31 00:13:53.051202 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046142 2581 flags.go:64] FLAG: --container-log-max-files="5" Oct 31 00:13:53.051202 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046147 2581 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 31 00:13:53.051202 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046150 2581 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 31 00:13:53.051202 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046153 2581 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 31 00:13:53.051202 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046156 2581 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 31 00:13:53.051202 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046159 2581 flags.go:64] FLAG: --contention-profiling="false" Oct 31 00:13:53.051202 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046162 2581 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 31 00:13:53.051202 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046165 2581 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 31 00:13:53.051202 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046170 2581 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 31 00:13:53.051202 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046173 2581 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 31 00:13:53.051202 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046177 2581 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 31 00:13:53.051202 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046180 2581 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 31 00:13:53.051202 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046183 2581 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 31 00:13:53.051202 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046186 2581 flags.go:64] FLAG: --enable-load-reader="false" Oct 31 00:13:53.051202 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046189 2581 flags.go:64] FLAG: --enable-server="true" Oct 31 00:13:53.051202 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046191 2581 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 31 00:13:53.051202 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046196 2581 flags.go:64] FLAG: --event-burst="100" Oct 31 00:13:53.051202 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046199 2581 flags.go:64] FLAG: --event-qps="50" Oct 31 00:13:53.051202 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046202 2581 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 31 00:13:53.051202 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046205 2581 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 31 00:13:53.051202 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046209 2581 flags.go:64] FLAG: --eviction-hard="" Oct 31 00:13:53.051202 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046213 2581 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 31 00:13:53.051817 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046216 2581 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 31 00:13:53.051817 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046219 2581 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 31 00:13:53.051817 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046223 2581 flags.go:64] FLAG: --eviction-soft="" Oct 31 00:13:53.051817 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046226 2581 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 31 00:13:53.051817 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046229 2581 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 31 00:13:53.051817 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046232 2581 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 31 00:13:53.051817 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046234 2581 flags.go:64] FLAG: --experimental-mounter-path="" Oct 31 00:13:53.051817 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046237 2581 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 31 00:13:53.051817 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046240 2581 flags.go:64] FLAG: --fail-swap-on="true" Oct 31 00:13:53.051817 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046243 2581 flags.go:64] FLAG: --feature-gates="" Oct 31 00:13:53.051817 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046247 2581 flags.go:64] FLAG: --file-check-frequency="20s" Oct 31 00:13:53.051817 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046250 2581 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 31 00:13:53.051817 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046254 2581 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 31 00:13:53.051817 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046258 2581 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 31 00:13:53.051817 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046261 2581 flags.go:64] FLAG: --healthz-port="10248" Oct 31 00:13:53.051817 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046264 2581 flags.go:64] FLAG: --help="false" Oct 31 00:13:53.051817 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046267 2581 flags.go:64] FLAG: --hostname-override="ip-10-0-137-56.us-west-2.compute.internal" Oct 31 00:13:53.051817 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046270 2581 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 31 00:13:53.051817 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046275 2581 flags.go:64] FLAG: --http-check-frequency="20s" Oct 31 00:13:53.051817 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046278 2581 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 31 00:13:53.051817 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046281 2581 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 31 00:13:53.051817 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046285 2581 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 31 00:13:53.051817 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046287 2581 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 31 00:13:53.052390 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046290 2581 flags.go:64] FLAG: --image-service-endpoint="" Oct 31 00:13:53.052390 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046293 2581 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 31 00:13:53.052390 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046296 2581 flags.go:64] FLAG: --kube-api-burst="100" Oct 31 00:13:53.052390 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046299 2581 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 31 00:13:53.052390 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046302 2581 flags.go:64] FLAG: --kube-api-qps="50" Oct 31 00:13:53.052390 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046305 2581 flags.go:64] FLAG: --kube-reserved="" Oct 31 00:13:53.052390 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046308 2581 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 31 00:13:53.052390 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046311 2581 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 31 00:13:53.052390 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046314 2581 flags.go:64] FLAG: --kubelet-cgroups="" Oct 31 00:13:53.052390 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046316 2581 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 31 00:13:53.052390 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046320 2581 flags.go:64] FLAG: --lock-file="" Oct 31 00:13:53.052390 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046322 2581 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 31 00:13:53.052390 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046325 2581 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 31 00:13:53.052390 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046328 2581 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 31 00:13:53.052390 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046334 2581 flags.go:64] FLAG: --log-json-split-stream="false" Oct 31 00:13:53.052390 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046337 2581 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 31 00:13:53.052390 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046340 2581 flags.go:64] FLAG: --log-text-split-stream="false" Oct 31 00:13:53.052390 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046342 2581 flags.go:64] FLAG: --logging-format="text" Oct 31 00:13:53.052390 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046345 2581 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 31 00:13:53.052390 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046348 2581 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 31 00:13:53.052390 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046351 2581 flags.go:64] FLAG: --manifest-url="" Oct 31 00:13:53.052390 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046354 2581 flags.go:64] FLAG: --manifest-url-header="" Oct 31 00:13:53.052390 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046359 2581 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 31 00:13:53.052390 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046362 2581 flags.go:64] FLAG: --max-open-files="1000000" Oct 31 00:13:53.052390 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046378 2581 flags.go:64] FLAG: --max-pods="110" Oct 31 00:13:53.052995 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046382 2581 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 31 00:13:53.052995 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046385 2581 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 31 00:13:53.052995 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046390 2581 flags.go:64] FLAG: --memory-manager-policy="None" Oct 31 00:13:53.052995 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046393 2581 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 31 00:13:53.052995 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046396 2581 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 31 00:13:53.052995 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046399 2581 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 31 00:13:53.052995 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046402 2581 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 31 00:13:53.052995 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046409 2581 flags.go:64] FLAG: --node-status-max-images="50" Oct 31 00:13:53.052995 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046413 2581 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 31 00:13:53.052995 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046416 2581 flags.go:64] FLAG: --oom-score-adj="-999" Oct 31 00:13:53.052995 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046419 2581 flags.go:64] FLAG: --pod-cidr="" Oct 31 00:13:53.052995 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046421 2581 flags.go:64] FLAG: --pod-manifest-path="" Oct 31 00:13:53.052995 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046424 2581 flags.go:64] FLAG: --pod-max-pids="-1" Oct 31 00:13:53.052995 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046427 2581 flags.go:64] FLAG: --pods-per-core="0" Oct 31 00:13:53.052995 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046430 2581 flags.go:64] FLAG: --port="10250" Oct 31 00:13:53.052995 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046433 2581 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 31 00:13:53.052995 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046436 2581 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-024d89d5d0a1f47d6" Oct 31 00:13:53.052995 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046439 2581 flags.go:64] FLAG: --qos-reserved="" Oct 31 00:13:53.052995 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046442 2581 flags.go:64] FLAG: --read-only-port="10255" Oct 31 00:13:53.052995 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046445 2581 flags.go:64] FLAG: --register-node="true" Oct 31 00:13:53.052995 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046448 2581 flags.go:64] FLAG: --register-with-taints="" Oct 31 00:13:53.052995 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046452 2581 flags.go:64] FLAG: --registry-burst="10" Oct 31 00:13:53.052995 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046455 2581 flags.go:64] FLAG: --registry-qps="5" Oct 31 00:13:53.052995 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046458 2581 flags.go:64] FLAG: --reserved-cpus="" Oct 31 00:13:53.052995 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046460 2581 flags.go:64] FLAG: --reserved-memory="" Oct 31 00:13:53.052995 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046465 2581 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 31 00:13:53.053659 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046468 2581 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 31 00:13:53.053659 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046471 2581 flags.go:64] FLAG: --rotate-certificates="false" Oct 31 00:13:53.053659 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046474 2581 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 31 00:13:53.053659 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046479 2581 flags.go:64] FLAG: --runonce="false" Oct 31 00:13:53.053659 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046482 2581 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 31 00:13:53.053659 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046485 2581 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 31 00:13:53.053659 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046488 2581 flags.go:64] FLAG: --seccomp-default="false" Oct 31 00:13:53.053659 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046491 2581 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 31 00:13:53.053659 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046495 2581 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 31 00:13:53.053659 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046499 2581 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 31 00:13:53.053659 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046502 2581 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 31 00:13:53.053659 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046505 2581 flags.go:64] FLAG: --storage-driver-password="root" Oct 31 00:13:53.053659 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046508 2581 flags.go:64] FLAG: --storage-driver-secure="false" Oct 31 00:13:53.053659 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046511 2581 flags.go:64] FLAG: --storage-driver-table="stats" Oct 31 00:13:53.053659 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046514 2581 flags.go:64] FLAG: --storage-driver-user="root" Oct 31 00:13:53.053659 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046516 2581 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 31 00:13:53.053659 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046519 2581 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 31 00:13:53.053659 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046523 2581 flags.go:64] FLAG: --system-cgroups="" Oct 31 00:13:53.053659 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046525 2581 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 31 00:13:53.053659 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046536 2581 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 31 00:13:53.053659 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046539 2581 flags.go:64] FLAG: --tls-cert-file="" Oct 31 00:13:53.053659 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046541 2581 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 31 00:13:53.053659 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046549 2581 flags.go:64] FLAG: --tls-min-version="" Oct 31 00:13:53.053659 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046552 2581 flags.go:64] FLAG: --tls-private-key-file="" Oct 31 00:13:53.053659 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046554 2581 flags.go:64] FLAG: --topology-manager-policy="none" Oct 31 00:13:53.054259 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046557 2581 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 31 00:13:53.054259 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046560 2581 flags.go:64] FLAG: --topology-manager-scope="container" Oct 31 00:13:53.054259 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046563 2581 flags.go:64] FLAG: --v="2" Oct 31 00:13:53.054259 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046568 2581 flags.go:64] FLAG: --version="false" Oct 31 00:13:53.054259 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046574 2581 flags.go:64] FLAG: --vmodule="" Oct 31 00:13:53.054259 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046578 2581 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 31 00:13:53.054259 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.046582 2581 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 31 00:13:53.054259 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.046742 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:13:53.054259 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.046746 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:13:53.054259 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.046749 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:13:53.054259 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.046754 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:13:53.054259 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.046756 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:13:53.054259 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.046759 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:13:53.054259 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.046762 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:13:53.054259 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.046764 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:13:53.054259 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.046769 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:13:53.054259 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.046772 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:13:53.054259 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.046774 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:13:53.054259 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.046777 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:13:53.054259 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.046780 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:13:53.054259 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.046783 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:13:53.054765 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.046785 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:13:53.054765 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.046788 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:13:53.054765 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.046790 2581 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:13:53.054765 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.046793 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:13:53.054765 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.046795 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:13:53.054765 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.046798 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:13:53.054765 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.046800 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:13:53.054765 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.046803 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:13:53.054765 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.046805 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:13:53.054765 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.046808 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:13:53.054765 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.046811 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:13:53.054765 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.046813 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:13:53.054765 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.046816 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:13:53.054765 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.046818 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:13:53.054765 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.046821 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:13:53.054765 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.046823 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:13:53.054765 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.046826 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:13:53.054765 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.046828 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:13:53.054765 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.046831 2581 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:13:53.054765 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.046833 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:13:53.055243 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.046835 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:13:53.055243 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.046839 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:13:53.055243 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.046842 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:13:53.055243 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.046844 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:13:53.055243 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.046847 2581 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:13:53.055243 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.046850 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:13:53.055243 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.046854 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:13:53.055243 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.046856 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:13:53.055243 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.046859 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:13:53.055243 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.046861 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:13:53.055243 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.046864 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:13:53.055243 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.046866 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:13:53.055243 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.046869 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:13:53.055243 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.046871 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:13:53.055243 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.046874 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:13:53.055243 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.046877 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:13:53.055243 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.046879 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:13:53.055243 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.046881 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:13:53.055243 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.046884 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:13:53.055243 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.046886 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:13:53.055810 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.046889 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:13:53.055810 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.046891 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:13:53.055810 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.046894 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:13:53.055810 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.046896 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:13:53.055810 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.046899 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:13:53.055810 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.046901 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:13:53.055810 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.046904 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:13:53.055810 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.046906 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:13:53.055810 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.046910 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:13:53.055810 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.046914 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:13:53.055810 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.046916 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:13:53.055810 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.046919 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:13:53.055810 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.046921 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:13:53.055810 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.046925 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:13:53.055810 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.046927 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:13:53.055810 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.046930 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:13:53.055810 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.046932 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:13:53.055810 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.046935 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:13:53.055810 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.046939 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:13:53.056285 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.046942 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:13:53.056285 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.046945 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:13:53.056285 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.046947 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:13:53.056285 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.046950 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:13:53.056285 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.046952 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:13:53.056285 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.046955 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:13:53.056285 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.046957 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:13:53.056285 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.046960 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:13:53.056285 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.046962 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:13:53.056285 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.046965 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:13:53.056285 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.046968 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:13:53.056285 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.046970 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:13:53.056285 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.046973 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:13:53.056285 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.046975 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:13:53.056285 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.046977 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:13:53.056285 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.046980 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:13:53.056285 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.046983 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:13:53.056285 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.046985 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:13:53.056285 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.046988 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:13:53.056753 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.046990 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:13:53.056753 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.046993 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:13:53.056753 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.046996 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:13:53.056753 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.047000 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:13:53.056753 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.047003 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:13:53.056753 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.047006 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:13:53.056753 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.047009 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:13:53.056753 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.047013 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:13:53.056753 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.047017 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:13:53.056753 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.047019 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:13:53.056753 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.047022 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:13:53.056753 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.047025 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:13:53.056753 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.047030 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:13:53.056753 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.047033 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:13:53.056753 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.047062 2581 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 00:13:53.057114 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.047100 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:13:53.057114 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.047106 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:13:53.057114 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.047110 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:13:53.057114 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.047113 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:13:53.057114 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.047116 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:13:53.057114 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.047119 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:13:53.057114 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.047121 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:13:53.057114 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.047124 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:13:53.057114 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.047127 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:13:53.057114 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.047129 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:13:53.057114 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.047132 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:13:53.057114 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.047134 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:13:53.057114 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.047137 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:13:53.057114 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.047140 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:13:53.057114 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.047142 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:13:53.057114 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.047145 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:13:53.057114 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.047147 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:13:53.057114 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.047150 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:13:53.057114 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.047152 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:13:53.057582 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.047155 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:13:53.057582 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.047157 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:13:53.057582 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.047159 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:13:53.057582 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.047162 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:13:53.057582 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.047164 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:13:53.057582 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.047168 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:13:53.057582 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.047170 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:13:53.057582 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.047173 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:13:53.057582 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.047175 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:13:53.057582 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.047177 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:13:53.057582 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.047180 2581 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:13:53.057582 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.047185 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:13:53.057582 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.047188 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:13:53.057582 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.047190 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:13:53.057582 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.047193 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:13:53.057582 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.047196 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:13:53.057582 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.047198 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:13:53.057582 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.047201 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:13:53.057582 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.047203 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:13:53.057582 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.047206 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:13:53.058055 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.047208 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:13:53.058055 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.047211 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:13:53.058055 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.047213 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:13:53.058055 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.047216 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:13:53.058055 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.047218 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:13:53.058055 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.047221 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:13:53.058055 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.047223 2581 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:13:53.058055 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.047226 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:13:53.058055 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.047228 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:13:53.058055 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.047231 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:13:53.058055 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.047233 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:13:53.058055 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.047236 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:13:53.058055 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.047238 2581 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:13:53.058055 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.047241 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:13:53.058055 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.047243 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:13:53.058055 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.047246 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:13:53.058055 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.047248 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:13:53.058055 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.047252 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:13:53.058055 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.047255 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:13:53.058055 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.047257 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:13:53.058552 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.047260 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:13:53.058552 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.047262 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:13:53.058552 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.047265 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:13:53.058552 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.047269 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:13:53.058552 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.047272 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:13:53.058552 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.047275 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:13:53.058552 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.047277 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:13:53.058552 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.047280 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:13:53.058552 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.047282 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:13:53.058552 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.047284 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:13:53.058552 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.047302 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:13:53.058552 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.047306 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:13:53.058552 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.047310 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:13:53.058552 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.047313 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:13:53.058552 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.047315 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:13:53.058552 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.047318 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:13:53.058552 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.047321 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:13:53.058552 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.047325 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:13:53.058552 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.047328 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:13:53.059042 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.047331 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:13:53.059042 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.047334 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:13:53.059042 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.047337 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:13:53.059042 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.047340 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:13:53.059042 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.047342 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:13:53.059042 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.047344 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:13:53.059042 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.047347 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:13:53.059042 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.047349 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:13:53.059042 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.047351 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:13:53.059042 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.047354 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:13:53.059042 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.047358 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:13:53.059042 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.047360 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:13:53.059042 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.047363 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:13:53.059042 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.047378 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:13:53.059042 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.047381 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:13:53.059042 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.047383 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:13:53.059042 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.047386 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:13:53.059042 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.047388 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:13:53.059042 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.047391 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:13:53.059510 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.047394 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:13:53.059510 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.047397 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:13:53.059510 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.047399 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:13:53.059510 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.047402 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:13:53.059510 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.047405 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:13:53.059510 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.047408 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:13:53.059510 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.047411 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:13:53.059510 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.047413 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:13:53.059510 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.047415 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:13:53.059510 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.054473 2581 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 31 00:13:53.059510 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.054496 2581 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 31 00:13:53.059510 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.054514 2581 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 31 00:13:53.059510 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.054519 2581 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 31 00:13:53.059510 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.054577 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:13:53.059510 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.054583 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:13:53.059510 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.054586 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:13:53.059510 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.054589 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:13:53.059510 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.054592 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:13:53.059510 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.054594 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:13:53.059971 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.054597 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:13:53.059971 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.054599 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:13:53.059971 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.054602 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:13:53.059971 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.054604 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:13:53.059971 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.054607 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:13:53.059971 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.054610 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:13:53.059971 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.054613 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:13:53.059971 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.054616 2581 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:13:53.059971 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.054619 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:13:53.059971 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.054621 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:13:53.059971 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.054624 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:13:53.059971 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.054626 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:13:53.059971 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.054629 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:13:53.059971 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.054631 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:13:53.059971 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.054634 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:13:53.059971 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.054636 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:13:53.059971 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.054639 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:13:53.059971 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.054641 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:13:53.059971 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.054644 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:13:53.059971 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.054646 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:13:53.060462 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.054649 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:13:53.060462 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.054651 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:13:53.060462 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.054654 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:13:53.060462 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.054658 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:13:53.060462 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.054660 2581 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:13:53.060462 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.054663 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:13:53.060462 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.054665 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:13:53.060462 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.054668 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:13:53.060462 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.054670 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:13:53.060462 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.054673 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:13:53.060462 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.054675 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:13:53.060462 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.054677 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:13:53.060462 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.054680 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:13:53.060462 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.054682 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:13:53.060462 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.054685 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:13:53.060462 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.054687 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:13:53.060462 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.054689 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:13:53.060462 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.054692 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:13:53.060462 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.054694 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:13:53.060462 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.054697 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:13:53.060949 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.054699 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:13:53.060949 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.054702 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:13:53.060949 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.054704 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:13:53.060949 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.054708 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:13:53.060949 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.054712 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:13:53.060949 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.054716 2581 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:13:53.060949 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.054719 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:13:53.060949 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.054721 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:13:53.060949 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.054725 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:13:53.060949 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.054729 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:13:53.060949 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.054732 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:13:53.060949 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.054734 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:13:53.060949 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.054737 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:13:53.060949 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.054739 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:13:53.060949 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.054742 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:13:53.060949 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.054745 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:13:53.060949 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.054748 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:13:53.060949 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.054751 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:13:53.060949 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.054753 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:13:53.061432 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.054755 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:13:53.061432 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.054758 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:13:53.061432 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.054760 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:13:53.061432 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.054763 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:13:53.061432 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.054766 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:13:53.061432 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.054768 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:13:53.061432 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.054771 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:13:53.061432 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.054774 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:13:53.061432 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.054776 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:13:53.061432 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.054779 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:13:53.061432 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.054781 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:13:53.061432 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.054784 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:13:53.061432 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.054786 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:13:53.061432 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.054788 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:13:53.061432 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.054791 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:13:53.061432 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.054794 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:13:53.061432 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.054796 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:13:53.061432 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.054799 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:13:53.061432 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.054802 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:13:53.061432 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.054804 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:13:53.061921 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.054807 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:13:53.061921 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.054809 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:13:53.061921 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.054812 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:13:53.061921 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.054814 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:13:53.061921 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.054816 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:13:53.061921 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.054819 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:13:53.061921 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.054821 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:13:53.061921 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.054824 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:13:53.061921 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.054827 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:13:53.061921 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.054830 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:13:53.061921 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.054833 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:13:53.061921 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.054835 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:13:53.061921 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.054838 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:13:53.061921 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.054840 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:13:53.061921 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.054843 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:13:53.061921 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.054845 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:13:53.061921 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.054847 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:13:53.061921 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.054850 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:13:53.061921 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.054852 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:13:53.062464 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.054855 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:13:53.062464 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.054857 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:13:53.062464 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.054888 2581 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 00:13:53.062464 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055037 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:13:53.062464 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055042 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:13:53.062464 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055045 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:13:53.062464 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055047 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:13:53.062464 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055050 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:13:53.062464 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055052 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:13:53.062464 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055055 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:13:53.062464 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055058 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:13:53.062464 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055060 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:13:53.062464 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055063 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:13:53.062464 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055065 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:13:53.062464 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055067 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:13:53.062464 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055070 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:13:53.062846 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055072 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:13:53.062846 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055075 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:13:53.062846 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055078 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:13:53.062846 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055080 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:13:53.062846 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055082 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:13:53.062846 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055085 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:13:53.062846 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055088 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:13:53.062846 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055091 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:13:53.062846 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055094 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:13:53.062846 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055097 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:13:53.062846 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055099 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:13:53.062846 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055101 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:13:53.062846 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055104 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:13:53.062846 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055106 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:13:53.062846 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055109 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:13:53.062846 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055111 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:13:53.062846 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055113 2581 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:13:53.062846 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055116 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:13:53.062846 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055118 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:13:53.063551 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055121 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:13:53.063551 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055123 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:13:53.063551 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055126 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:13:53.063551 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055128 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:13:53.063551 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055131 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:13:53.063551 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055133 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:13:53.063551 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055135 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:13:53.063551 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055138 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:13:53.063551 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055141 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:13:53.063551 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055143 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:13:53.063551 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055146 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:13:53.063551 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055148 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:13:53.063551 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055150 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:13:53.063551 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055153 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:13:53.063551 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055155 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:13:53.063551 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055158 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:13:53.063551 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055160 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:13:53.063551 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055162 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:13:53.063551 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055165 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:13:53.064031 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055168 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:13:53.064031 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055170 2581 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:13:53.064031 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055173 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:13:53.064031 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055176 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:13:53.064031 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055179 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:13:53.064031 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055181 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:13:53.064031 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055184 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:13:53.064031 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055186 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:13:53.064031 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055188 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:13:53.064031 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055191 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:13:53.064031 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055193 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:13:53.064031 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055196 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:13:53.064031 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055198 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:13:53.064031 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055201 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:13:53.064031 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055203 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:13:53.064031 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055206 2581 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:13:53.064031 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055208 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:13:53.064031 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055210 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:13:53.064031 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055213 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:13:53.064031 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055215 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:13:53.064563 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055218 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:13:53.064563 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055220 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:13:53.064563 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055223 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:13:53.064563 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055225 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:13:53.064563 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055228 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:13:53.064563 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055230 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:13:53.064563 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055233 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:13:53.064563 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055235 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:13:53.064563 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055238 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:13:53.064563 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055240 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:13:53.064563 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055243 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:13:53.064563 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055246 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:13:53.064563 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055249 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:13:53.064563 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055252 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:13:53.064563 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055256 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:13:53.064563 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055259 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:13:53.064563 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055262 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:13:53.064563 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055265 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:13:53.064563 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055267 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:13:53.065016 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055270 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:13:53.065016 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055272 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:13:53.065016 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055275 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:13:53.065016 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055277 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:13:53.065016 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055279 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:13:53.065016 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055282 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:13:53.065016 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055284 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:13:53.065016 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055286 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:13:53.065016 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055289 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:13:53.065016 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055291 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:13:53.065016 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055293 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:13:53.065016 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055296 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:13:53.065016 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055298 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:13:53.065016 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055302 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:13:53.065016 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055305 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:13:53.065016 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055308 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:13:53.065503 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.055331 2581 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 00:13:53.065503 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055362 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:13:53.065503 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055384 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:13:53.065503 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055388 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:13:53.065503 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055392 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:13:53.065503 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055394 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:13:53.065503 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055397 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:13:53.065503 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055399 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:13:53.065503 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055402 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:13:53.065503 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055405 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:13:53.065503 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055408 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:13:53.065503 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055411 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:13:53.065503 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055415 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:13:53.065503 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055418 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:13:53.065503 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055421 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:13:53.065503 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055424 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:13:53.065932 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055426 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:13:53.065932 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055429 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:13:53.065932 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055431 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:13:53.065932 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055434 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:13:53.065932 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055436 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:13:53.065932 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055438 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:13:53.065932 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055441 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:13:53.065932 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055443 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:13:53.065932 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055445 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:13:53.065932 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055448 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:13:53.065932 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055450 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:13:53.065932 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055453 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:13:53.065932 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055455 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:13:53.065932 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055458 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:13:53.065932 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055462 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:13:53.065932 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055465 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:13:53.065932 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055468 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:13:53.065932 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055470 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:13:53.065932 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055473 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:13:53.066385 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055476 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:13:53.066385 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055478 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:13:53.066385 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055480 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:13:53.066385 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055483 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:13:53.066385 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055488 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:13:53.066385 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055491 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:13:53.066385 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055494 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:13:53.066385 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055496 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:13:53.066385 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055499 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:13:53.066385 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055501 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:13:53.066385 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055504 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:13:53.066385 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055506 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:13:53.066385 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055509 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:13:53.066385 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055511 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:13:53.066385 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055514 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:13:53.066385 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055517 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:13:53.066385 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055519 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:13:53.066385 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055521 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:13:53.066385 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055524 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:13:53.066891 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055526 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:13:53.066891 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055529 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:13:53.066891 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055531 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:13:53.066891 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055534 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:13:53.066891 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055536 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:13:53.066891 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055538 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:13:53.066891 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055541 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:13:53.066891 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055543 2581 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:13:53.066891 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055546 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:13:53.066891 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055548 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:13:53.066891 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055551 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:13:53.066891 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055554 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:13:53.066891 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055556 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:13:53.066891 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055559 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:13:53.066891 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055561 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:13:53.066891 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055564 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:13:53.066891 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055566 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:13:53.066891 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055570 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:13:53.066891 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055573 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:13:53.066891 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055575 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:13:53.067432 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055577 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:13:53.067432 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055580 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:13:53.067432 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055583 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:13:53.067432 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055585 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:13:53.067432 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055587 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:13:53.067432 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055590 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:13:53.067432 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055593 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:13:53.067432 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055596 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:13:53.067432 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055598 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:13:53.067432 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055601 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:13:53.067432 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055603 2581 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:13:53.067432 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055606 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:13:53.067432 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055608 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:13:53.067432 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055610 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:13:53.067432 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055613 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:13:53.067432 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055615 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:13:53.067432 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055617 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:13:53.067432 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055620 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:13:53.067432 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055622 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:13:53.067432 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055625 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:13:53.067894 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055627 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:13:53.067894 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055630 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:13:53.067894 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055633 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:13:53.067894 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055635 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:13:53.067894 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055638 2581 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:13:53.067894 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055640 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:13:53.067894 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055642 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:13:53.067894 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055645 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:13:53.067894 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055648 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:13:53.067894 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055653 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:13:53.067894 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055655 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:13:53.067894 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055658 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:13:53.067894 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:53.055660 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:13:53.067894 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.055720 2581 server.go:957] "Client rotation is on, will bootstrap in background" Oct 31 00:13:53.067894 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.058727 2581 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 31 00:13:53.067894 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.059550 2581 server.go:1014] "Starting client certificate rotation" Oct 31 00:13:53.067894 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.059641 2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 00:13:53.067894 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.059672 2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 00:13:53.068347 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.061261 2581 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 00:13:53.068347 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.066621 2581 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 00:13:53.076307 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.076283 2581 log.go:25] "Validated CRI v1 runtime API" Oct 31 00:13:53.080501 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.080485 2581 log.go:25] "Validated CRI v1 image API" Oct 31 00:13:53.082492 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.082456 2581 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 31 00:13:53.085009 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.084980 2581 fs.go:135] Filesystem UUIDs: map[23ee767d-b28d-4825-ae3e-a07f0a1c007a:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 cc168d45-902b-4a82-8b93-fb101dccc663:/dev/nvme0n1p3] Oct 31 00:13:53.085096 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.085006 2581 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 31 00:13:53.090490 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.090329 2581 manager.go:217] Machine: {Timestamp:2026-10-31 00:13:53.088755927 +0000 UTC m=+0.321291366 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099086 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2a538df4d8ddd6a1300a4438b9b447 SystemUUID:ec2a538d-f4d8-ddd6-a130-0a4438b9b447 BootID:af8b0a00-c27e-4d39-bc3a-3bf1c043eb45 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:d2:28:5e:33 Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:d2:28:5e:33 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:4a:86:01:71:a3:2b Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 31 00:13:53.090490 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.090489 2581 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 31 00:13:53.090651 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.090636 2581 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 31 00:13:53.093211 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.093172 2581 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 31 00:13:53.093435 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.093207 2581 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-56.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 31 00:13:53.093536 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.093439 2581 topology_manager.go:143] "Creating topology manager with none policy" Oct 31 00:13:53.093536 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.093453 2581 container_manager_linux.go:311] "Creating device plugin manager" Oct 31 00:13:53.093536 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.093507 2581 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 00:13:53.093536 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.093532 2581 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 00:13:53.094342 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.094328 2581 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 31 00:13:53.097637 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.097620 2581 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 31 00:13:53.097803 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.097791 2581 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 31 00:13:53.098429 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.098418 2581 kubelet.go:493] "Attempting to sync node with API server" Oct 31 00:13:53.098493 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.098442 2581 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 31 00:13:53.098493 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.098457 2581 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 31 00:13:53.098493 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.098473 2581 kubelet.go:405] "Adding apiserver pod source" Oct 31 00:13:53.098493 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.098485 2581 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 31 00:13:53.098880 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.098861 2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 00:13:53.106760 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.106737 2581 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 31 00:13:53.111119 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.111105 2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 00:13:53.111901 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.111888 2581 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 31 00:13:53.111940 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.111919 2581 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 31 00:13:53.112086 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.112075 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 31 00:13:53.112116 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.112093 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 31 00:13:53.112116 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.112104 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 31 00:13:53.112116 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.112113 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 31 00:13:53.112219 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.112122 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 31 00:13:53.112219 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.112130 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 31 00:13:53.112643 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.112633 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 31 00:13:53.112678 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.112645 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 31 00:13:53.112678 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.112653 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 31 00:13:53.112678 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.112658 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 31 00:13:53.112678 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.112674 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 31 00:13:53.112794 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.112683 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 31 00:13:53.113246 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.113238 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 31 00:13:53.113274 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.113247 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 31 00:13:53.117064 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.117042 2581 server.go:1263] "Started kubelet" Oct 31 00:13:53.117220 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.117147 2581 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 31 00:13:53.117310 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.117248 2581 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 31 00:13:53.117310 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.117249 2581 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 31 00:13:53.117872 ip-10-0-137-56 systemd[1]: Started Kubernetes Kubelet. Oct 31 00:13:53.122198 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.122175 2581 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 31 00:13:53.125356 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.125331 2581 server.go:317] "Adding debug handlers to kubelet server" Oct 31 00:13:53.128331 ip-10-0-137-56 kubenswrapper[2581]: E1031 00:13:53.128301 2581 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 31 00:13:53.128993 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.128972 2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 31 00:13:53.129918 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.129893 2581 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 31 00:13:53.130052 ip-10-0-137-56 kubenswrapper[2581]: E1031 00:13:53.129993 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-56.us-west-2.compute.internal\" not found" Oct 31 00:13:53.130415 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.130404 2581 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 31 00:13:53.130415 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.130416 2581 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 31 00:13:53.130568 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.130504 2581 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 31 00:13:53.130568 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.130551 2581 reconstruct.go:97] "Volume reconstruction finished" Oct 31 00:13:53.130568 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.130559 2581 reconciler.go:29] "Reconciler: start to sync state" Oct 31 00:13:53.130864 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.130850 2581 factory.go:55] Registering systemd factory Oct 31 00:13:53.130947 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.130868 2581 factory.go:223] Registration of the systemd container factory successfully Oct 31 00:13:53.131225 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.131209 2581 factory.go:153] Registering CRI-O factory Oct 31 00:13:53.131264 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.131231 2581 factory.go:223] Registration of the crio container factory successfully Oct 31 00:13:53.131327 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.131306 2581 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 31 00:13:53.131381 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.131356 2581 factory.go:103] Registering Raw factory Oct 31 00:13:53.131424 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.131400 2581 manager.go:1199] Started watching for new ooms in manager Oct 31 00:13:53.132401 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.132364 2581 manager.go:319] Starting recovery of all containers Oct 31 00:13:53.143046 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.143031 2581 manager.go:324] Recovery completed Oct 31 00:13:53.147112 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.147093 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 00:13:53.149223 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.149205 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-56.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 00:13:53.149318 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.149243 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-56.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 00:13:53.149318 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.149255 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-56.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 31 00:13:53.149816 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.149798 2581 cpu_manager.go:226] "Starting" policy="none" Oct 31 00:13:53.149816 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.149814 2581 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 31 00:13:53.149929 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.149835 2581 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 31 00:13:53.149929 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.149843 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 00:13:53.150579 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.150559 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-56.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 00:13:53.150640 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.150602 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-56.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 00:13:53.150640 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.150616 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-56.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 31 00:13:53.151033 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.151022 2581 policy_none.go:50] "Start" Oct 31 00:13:53.151063 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.151038 2581 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 31 00:13:53.151063 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.151048 2581 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 31 00:13:53.151984 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.151974 2581 policy_none.go:44] "Start" Oct 31 00:13:53.191188 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.190997 2581 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 31 00:13:53.196821 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.196806 2581 manager.go:341] "Starting Device Plugin manager" Oct 31 00:13:53.196900 ip-10-0-137-56 kubenswrapper[2581]: E1031 00:13:53.196840 2581 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 31 00:13:53.196900 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.196855 2581 server.go:87] "Starting device plugin registration server" Oct 31 00:13:53.197180 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.197167 2581 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 31 00:13:53.197228 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.197180 2581 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 31 00:13:53.197281 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.197264 2581 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 31 00:13:53.197393 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.197361 2581 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 31 00:13:53.197393 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.197390 2581 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 31 00:13:53.197824 ip-10-0-137-56 kubenswrapper[2581]: E1031 00:13:53.197801 2581 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 31 00:13:53.197900 ip-10-0-137-56 kubenswrapper[2581]: E1031 00:13:53.197859 2581 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-56.us-west-2.compute.internal\" not found" Oct 31 00:13:53.254627 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.254600 2581 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-56.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 31 00:13:53.255561 ip-10-0-137-56 kubenswrapper[2581]: E1031 00:13:53.254593 2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-56.us-west-2.compute.internal.18e374ebe41fbc28 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-56.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-56.us-west-2.compute.internal,},FirstTimestamp:2026-10-31 00:13:53.117010984 +0000 UTC m=+0.349546421,LastTimestamp:2026-10-31 00:13:53.117010984 +0000 UTC m=+0.349546421,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-56.us-west-2.compute.internal,}" Oct 31 00:13:53.271674 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.271643 2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 31 00:13:53.272964 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.272945 2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 31 00:13:53.273058 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.272969 2581 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 31 00:13:53.273058 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.272991 2581 kubelet.go:2521] "Starting kubelet main sync loop" Oct 31 00:13:53.273058 ip-10-0-137-56 kubenswrapper[2581]: E1031 00:13:53.273024 2581 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 31 00:13:53.297872 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.297795 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 00:13:53.298888 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.298863 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-56.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 00:13:53.298998 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.298905 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-56.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 00:13:53.298998 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.298916 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-56.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 31 00:13:53.298998 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.298941 2581 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-137-56.us-west-2.compute.internal" Oct 31 00:13:53.300588 ip-10-0-137-56 kubenswrapper[2581]: E1031 00:13:53.300556 2581 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-137-56.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 31 00:13:53.328244 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.328213 2581 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 31 00:13:53.328436 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.328276 2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-btjqb" Oct 31 00:13:53.373789 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.373749 2581 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-56.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-56.us-west-2.compute.internal"] Oct 31 00:13:53.373974 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.373833 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 00:13:53.375020 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.375001 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-56.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 00:13:53.375110 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.375040 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-56.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 00:13:53.375110 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.375055 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-56.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 31 00:13:53.376331 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.376318 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 00:13:53.376511 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.376495 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-56.us-west-2.compute.internal" Oct 31 00:13:53.376578 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.376543 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 00:13:53.377307 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.377293 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-56.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 00:13:53.377411 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.377324 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-56.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 00:13:53.377411 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.377337 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-56.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 31 00:13:53.377411 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.377296 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-56.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 00:13:53.377411 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.377400 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-56.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 00:13:53.377411 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.377410 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-56.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 31 00:13:53.378683 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.378667 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-56.us-west-2.compute.internal" Oct 31 00:13:53.378760 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.378698 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 00:13:53.379515 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.379499 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-56.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 00:13:53.379609 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.379528 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-56.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 00:13:53.379609 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.379543 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-56.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 31 00:13:53.412074 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.412054 2581 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:13:53.421545 ip-10-0-137-56 kubenswrapper[2581]: E1031 00:13:53.421521 2581 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-56.us-west-2.compute.internal\" not found" node="ip-10-0-137-56.us-west-2.compute.internal" Oct 31 00:13:53.425196 ip-10-0-137-56 kubenswrapper[2581]: E1031 00:13:53.425178 2581 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-56.us-west-2.compute.internal\" not found" node="ip-10-0-137-56.us-west-2.compute.internal" Oct 31 00:13:53.431233 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.431215 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f0db3b99b7954e19a6b4da51857943e0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-56.us-west-2.compute.internal\" (UID: \"f0db3b99b7954e19a6b4da51857943e0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-56.us-west-2.compute.internal" Oct 31 00:13:53.431288 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.431241 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f0db3b99b7954e19a6b4da51857943e0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-56.us-west-2.compute.internal\" (UID: \"f0db3b99b7954e19a6b4da51857943e0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-56.us-west-2.compute.internal" Oct 31 00:13:53.431288 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.431258 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e15cd53a0cf50b999ecad893fad25259-config\") pod \"kube-apiserver-proxy-ip-10-0-137-56.us-west-2.compute.internal\" (UID: \"e15cd53a0cf50b999ecad893fad25259\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-56.us-west-2.compute.internal" Oct 31 00:13:53.457343 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.457314 2581 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:13:53.503642 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.503609 2581 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:13:53.504760 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.504742 2581 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:13:53.505359 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.505343 2581 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-137-56.us-west-2.compute.internal" Oct 31 00:13:53.531107 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.531080 2581 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-56.us-west-2.compute.internal" Oct 31 00:13:53.532214 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.532196 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e15cd53a0cf50b999ecad893fad25259-config\") pod \"kube-apiserver-proxy-ip-10-0-137-56.us-west-2.compute.internal\" (UID: \"e15cd53a0cf50b999ecad893fad25259\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-56.us-west-2.compute.internal" Oct 31 00:13:53.532293 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.532223 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f0db3b99b7954e19a6b4da51857943e0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-56.us-west-2.compute.internal\" (UID: \"f0db3b99b7954e19a6b4da51857943e0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-56.us-west-2.compute.internal" Oct 31 00:13:53.532293 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.532254 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f0db3b99b7954e19a6b4da51857943e0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-56.us-west-2.compute.internal\" (UID: \"f0db3b99b7954e19a6b4da51857943e0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-56.us-west-2.compute.internal" Oct 31 00:13:53.532410 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.532310 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f0db3b99b7954e19a6b4da51857943e0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-56.us-west-2.compute.internal\" (UID: \"f0db3b99b7954e19a6b4da51857943e0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-56.us-west-2.compute.internal" Oct 31 00:13:53.532410 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.532323 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e15cd53a0cf50b999ecad893fad25259-config\") pod \"kube-apiserver-proxy-ip-10-0-137-56.us-west-2.compute.internal\" (UID: \"e15cd53a0cf50b999ecad893fad25259\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-56.us-west-2.compute.internal" Oct 31 00:13:53.532410 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.532334 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f0db3b99b7954e19a6b4da51857943e0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-56.us-west-2.compute.internal\" (UID: \"f0db3b99b7954e19a6b4da51857943e0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-56.us-west-2.compute.internal" Oct 31 00:13:53.589875 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.589802 2581 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 00:13:53.590008 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.589915 2581 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-56.us-west-2.compute.internal" Oct 31 00:13:53.645011 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.644987 2581 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 00:13:53.724438 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.724382 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-56.us-west-2.compute.internal" Oct 31 00:13:53.728916 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:53.728334 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-56.us-west-2.compute.internal" Oct 31 00:13:54.059921 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.059837 2581 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 31 00:13:54.060434 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.060031 2581 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 31 00:13:54.060434 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.060042 2581 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 31 00:13:54.060434 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.060034 2581 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 31 00:13:54.060434 ip-10-0-137-56 kubenswrapper[2581]: E1031 00:13:54.060107 2581 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a076d63db167a4a51863954deecefa98-6bdc7d033aceb6e1.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-137-56.us-west-2.compute.internal.18e374ebe60b46ee\": read tcp 10.0.137.56:34692->3.132.154.107:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-137-56.us-west-2.compute.internal.18e374ebe60b46ee default 5098 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-56.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-137-56.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-137-56.us-west-2.compute.internal,},FirstTimestamp:2026-10-31 00:13:53 +0000 UTC,LastTimestamp:2026-10-31 00:13:53.375021733 +0000 UTC m=+0.607557177,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-56.us-west-2.compute.internal,}" Oct 31 00:13:54.098840 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.098811 2581 apiserver.go:52] "Watching apiserver" Oct 31 00:13:54.129471 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.129438 2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 31 00:13:54.332773 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.332676 2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 00:08:53 +0000 UTC" deadline="2028-07-05 01:01:03.114885696 +0000 UTC" Oct 31 00:13:54.332773 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.332714 2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14712h47m8.782175762s" Oct 31 00:13:54.357667 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.357642 2581 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 31 00:13:54.357975 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.357959 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-rjfn2","openshift-network-operator/iptables-alerter-4nwn7","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ndd7n","openshift-multus/multus-additional-cni-plugins-jsctc","openshift-multus/multus-b6vpd","openshift-network-diagnostics/network-check-target-ml8n5","openshift-ovn-kubernetes/ovnkube-node-h9hsf","kube-system/global-pull-secret-syncer-fwgtx","kube-system/kube-apiserver-proxy-ip-10-0-137-56.us-west-2.compute.internal","openshift-dns/node-resolver-gkm2c","openshift-image-registry/node-ca-xlzv2","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-56.us-west-2.compute.internal","kube-system/konnectivity-agent-lvp2h","openshift-cluster-node-tuning-operator/tuned-vsfrc"] Oct 31 00:13:54.359507 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.359486 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:13:54.360621 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.360592 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-lvp2h" Oct 31 00:13:54.361659 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.361635 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ndd7n" Oct 31 00:13:54.362715 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.362699 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jsctc" Oct 31 00:13:54.363718 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.363701 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-b6vpd" Oct 31 00:13:54.364660 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.364648 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ml8n5" Oct 31 00:13:54.364714 ip-10-0-137-56 kubenswrapper[2581]: E1031 00:13:54.364698 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ml8n5" podUID="0cbf0042-931c-4135-9330-f9e798b7a567" Oct 31 00:13:54.365708 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.365695 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vsfrc" Oct 31 00:13:54.366782 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.366768 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rjfn2" Oct 31 00:13:54.366838 ip-10-0-137-56 kubenswrapper[2581]: E1031 00:13:54.366813 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rjfn2" podUID="a077dea4-b5a3-4335-bb6a-7618e794eb8f" Oct 31 00:13:54.368020 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.368007 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4nwn7" Oct 31 00:13:54.369017 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.368998 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gkm2c" Oct 31 00:13:54.370432 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.370419 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-xlzv2" Oct 31 00:13:54.371800 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.371784 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fwgtx" Oct 31 00:13:54.371857 ip-10-0-137-56 kubenswrapper[2581]: E1031 00:13:54.371833 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fwgtx" podUID="d02ae6b5-4860-45b3-8d16-4a49531b77d4" Oct 31 00:13:54.410532 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.410500 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 31 00:13:54.411131 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.411106 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 31 00:13:54.411247 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.411195 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 31 00:13:54.411247 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.411231 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-h9nxs\"" Oct 31 00:13:54.411354 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.411309 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 31 00:13:54.411435 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.411425 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 31 00:13:54.411494 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.411445 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-ds5xr\"" Oct 31 00:13:54.411674 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.411656 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 31 00:13:54.411793 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.411419 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 31 00:13:54.411852 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.411825 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 31 00:13:54.411852 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.411847 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 31 00:13:54.411947 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.411868 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 31 00:13:54.412060 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.412041 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 31 00:13:54.412116 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.412069 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 31 00:13:54.412263 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.412247 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-bcxbn\"" Oct 31 00:13:54.431696 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.431678 2581 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 31 00:13:54.437584 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.437563 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0eb4888f-40a8-4aba-85d8-3c845382bd02-etc-tuned\") pod \"tuned-vsfrc\" (UID: \"0eb4888f-40a8-4aba-85d8-3c845382bd02\") " pod="openshift-cluster-node-tuning-operator/tuned-vsfrc" Oct 31 00:13:54.437686 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.437592 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9801ff00-9e9a-4141-8212-ca32bf0251b3-host-run-netns\") pod \"multus-b6vpd\" (UID: \"9801ff00-9e9a-4141-8212-ca32bf0251b3\") " pod="openshift-multus/multus-b6vpd" Oct 31 00:13:54.437686 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.437607 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9801ff00-9e9a-4141-8212-ca32bf0251b3-host-var-lib-cni-bin\") pod \"multus-b6vpd\" (UID: \"9801ff00-9e9a-4141-8212-ca32bf0251b3\") " pod="openshift-multus/multus-b6vpd" Oct 31 00:13:54.437686 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.437621 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0f8c3c03-a1fc-4376-98c6-feeeefa08068-run-systemd\") pod \"ovnkube-node-h9hsf\" (UID: \"0f8c3c03-a1fc-4376-98c6-feeeefa08068\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:13:54.437686 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.437636 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0eb4888f-40a8-4aba-85d8-3c845382bd02-etc-modprobe-d\") pod \"tuned-vsfrc\" (UID: \"0eb4888f-40a8-4aba-85d8-3c845382bd02\") " pod="openshift-cluster-node-tuning-operator/tuned-vsfrc" Oct 31 00:13:54.437860 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.437683 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0eb4888f-40a8-4aba-85d8-3c845382bd02-host\") pod \"tuned-vsfrc\" (UID: \"0eb4888f-40a8-4aba-85d8-3c845382bd02\") " pod="openshift-cluster-node-tuning-operator/tuned-vsfrc" Oct 31 00:13:54.437860 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.437725 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbpb7\" (UniqueName: \"kubernetes.io/projected/978a9050-dddb-4869-85d8-84a001554c5b-kube-api-access-gbpb7\") pod \"iptables-alerter-4nwn7\" (UID: \"978a9050-dddb-4869-85d8-84a001554c5b\") " pod="openshift-network-operator/iptables-alerter-4nwn7" Oct 31 00:13:54.437860 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.437763 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nz8r2\" (UniqueName: \"kubernetes.io/projected/eadb6436-5e5b-4116-8ed7-26aa935fa749-kube-api-access-nz8r2\") pod \"node-ca-xlzv2\" (UID: \"eadb6436-5e5b-4116-8ed7-26aa935fa749\") " pod="openshift-image-registry/node-ca-xlzv2" Oct 31 00:13:54.437860 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.437831 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1bc3d212-b67c-4368-90a3-8e080a4f1452-device-dir\") pod \"aws-ebs-csi-driver-node-ndd7n\" (UID: \"1bc3d212-b67c-4368-90a3-8e080a4f1452\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ndd7n" Oct 31 00:13:54.438013 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.437864 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9801ff00-9e9a-4141-8212-ca32bf0251b3-cni-binary-copy\") pod \"multus-b6vpd\" (UID: \"9801ff00-9e9a-4141-8212-ca32bf0251b3\") " pod="openshift-multus/multus-b6vpd" Oct 31 00:13:54.438013 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.437897 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9801ff00-9e9a-4141-8212-ca32bf0251b3-host-var-lib-kubelet\") pod \"multus-b6vpd\" (UID: \"9801ff00-9e9a-4141-8212-ca32bf0251b3\") " pod="openshift-multus/multus-b6vpd" Oct 31 00:13:54.438013 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.437923 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzs4x\" (UniqueName: \"kubernetes.io/projected/9801ff00-9e9a-4141-8212-ca32bf0251b3-kube-api-access-xzs4x\") pod \"multus-b6vpd\" (UID: \"9801ff00-9e9a-4141-8212-ca32bf0251b3\") " pod="openshift-multus/multus-b6vpd" Oct 31 00:13:54.438013 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.437951 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8j6tz\" (UniqueName: \"kubernetes.io/projected/6a21f264-1e80-4eab-8c6e-27ca9ca19f0d-kube-api-access-8j6tz\") pod \"multus-additional-cni-plugins-jsctc\" (UID: \"6a21f264-1e80-4eab-8c6e-27ca9ca19f0d\") " pod="openshift-multus/multus-additional-cni-plugins-jsctc" Oct 31 00:13:54.438013 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.437977 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0eb4888f-40a8-4aba-85d8-3c845382bd02-run\") pod \"tuned-vsfrc\" (UID: \"0eb4888f-40a8-4aba-85d8-3c845382bd02\") " pod="openshift-cluster-node-tuning-operator/tuned-vsfrc" Oct 31 00:13:54.438013 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.438002 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9801ff00-9e9a-4141-8212-ca32bf0251b3-system-cni-dir\") pod \"multus-b6vpd\" (UID: \"9801ff00-9e9a-4141-8212-ca32bf0251b3\") " pod="openshift-multus/multus-b6vpd" Oct 31 00:13:54.438233 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.438021 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0f8c3c03-a1fc-4376-98c6-feeeefa08068-host-kubelet\") pod \"ovnkube-node-h9hsf\" (UID: \"0f8c3c03-a1fc-4376-98c6-feeeefa08068\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:13:54.438233 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.438035 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0f8c3c03-a1fc-4376-98c6-feeeefa08068-host-run-netns\") pod \"ovnkube-node-h9hsf\" (UID: \"0f8c3c03-a1fc-4376-98c6-feeeefa08068\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:13:54.438233 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.438075 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f8c3c03-a1fc-4376-98c6-feeeefa08068-var-lib-openvswitch\") pod \"ovnkube-node-h9hsf\" (UID: \"0f8c3c03-a1fc-4376-98c6-feeeefa08068\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:13:54.438233 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.438096 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6a21f264-1e80-4eab-8c6e-27ca9ca19f0d-system-cni-dir\") pod \"multus-additional-cni-plugins-jsctc\" (UID: \"6a21f264-1e80-4eab-8c6e-27ca9ca19f0d\") " pod="openshift-multus/multus-additional-cni-plugins-jsctc" Oct 31 00:13:54.438233 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.438110 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0eb4888f-40a8-4aba-85d8-3c845382bd02-var-lib-kubelet\") pod \"tuned-vsfrc\" (UID: \"0eb4888f-40a8-4aba-85d8-3c845382bd02\") " pod="openshift-cluster-node-tuning-operator/tuned-vsfrc" Oct 31 00:13:54.438233 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.438134 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/978a9050-dddb-4869-85d8-84a001554c5b-iptables-alerter-script\") pod \"iptables-alerter-4nwn7\" (UID: \"978a9050-dddb-4869-85d8-84a001554c5b\") " pod="openshift-network-operator/iptables-alerter-4nwn7" Oct 31 00:13:54.438233 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.438163 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/978a9050-dddb-4869-85d8-84a001554c5b-host-slash\") pod \"iptables-alerter-4nwn7\" (UID: \"978a9050-dddb-4869-85d8-84a001554c5b\") " pod="openshift-network-operator/iptables-alerter-4nwn7" Oct 31 00:13:54.438233 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.438197 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wm7wn\" (UniqueName: \"kubernetes.io/projected/1bc3d212-b67c-4368-90a3-8e080a4f1452-kube-api-access-wm7wn\") pod \"aws-ebs-csi-driver-node-ndd7n\" (UID: \"1bc3d212-b67c-4368-90a3-8e080a4f1452\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ndd7n" Oct 31 00:13:54.438233 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.438233 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9801ff00-9e9a-4141-8212-ca32bf0251b3-multus-cni-dir\") pod \"multus-b6vpd\" (UID: \"9801ff00-9e9a-4141-8212-ca32bf0251b3\") " pod="openshift-multus/multus-b6vpd" Oct 31 00:13:54.438604 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.438250 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9801ff00-9e9a-4141-8212-ca32bf0251b3-host-run-k8s-cni-cncf-io\") pod \"multus-b6vpd\" (UID: \"9801ff00-9e9a-4141-8212-ca32bf0251b3\") " pod="openshift-multus/multus-b6vpd" Oct 31 00:13:54.438604 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.438279 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9801ff00-9e9a-4141-8212-ca32bf0251b3-multus-daemon-config\") pod \"multus-b6vpd\" (UID: \"9801ff00-9e9a-4141-8212-ca32bf0251b3\") " pod="openshift-multus/multus-b6vpd" Oct 31 00:13:54.438604 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.438306 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9801ff00-9e9a-4141-8212-ca32bf0251b3-host-run-multus-certs\") pod \"multus-b6vpd\" (UID: \"9801ff00-9e9a-4141-8212-ca32bf0251b3\") " pod="openshift-multus/multus-b6vpd" Oct 31 00:13:54.438604 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.438325 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ms4cs\" (UniqueName: \"kubernetes.io/projected/0cbf0042-931c-4135-9330-f9e798b7a567-kube-api-access-ms4cs\") pod \"network-check-target-ml8n5\" (UID: \"0cbf0042-931c-4135-9330-f9e798b7a567\") " pod="openshift-network-diagnostics/network-check-target-ml8n5" Oct 31 00:13:54.438604 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.438381 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6a21f264-1e80-4eab-8c6e-27ca9ca19f0d-cni-binary-copy\") pod \"multus-additional-cni-plugins-jsctc\" (UID: \"6a21f264-1e80-4eab-8c6e-27ca9ca19f0d\") " pod="openshift-multus/multus-additional-cni-plugins-jsctc" Oct 31 00:13:54.438604 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.438410 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6a21f264-1e80-4eab-8c6e-27ca9ca19f0d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jsctc\" (UID: \"6a21f264-1e80-4eab-8c6e-27ca9ca19f0d\") " pod="openshift-multus/multus-additional-cni-plugins-jsctc" Oct 31 00:13:54.438604 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.438432 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d02ae6b5-4860-45b3-8d16-4a49531b77d4-kubelet-config\") pod \"global-pull-secret-syncer-fwgtx\" (UID: \"d02ae6b5-4860-45b3-8d16-4a49531b77d4\") " pod="kube-system/global-pull-secret-syncer-fwgtx" Oct 31 00:13:54.438604 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.438462 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9801ff00-9e9a-4141-8212-ca32bf0251b3-cnibin\") pod \"multus-b6vpd\" (UID: \"9801ff00-9e9a-4141-8212-ca32bf0251b3\") " pod="openshift-multus/multus-b6vpd" Oct 31 00:13:54.438604 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.438478 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9801ff00-9e9a-4141-8212-ca32bf0251b3-multus-conf-dir\") pod \"multus-b6vpd\" (UID: \"9801ff00-9e9a-4141-8212-ca32bf0251b3\") " pod="openshift-multus/multus-b6vpd" Oct 31 00:13:54.438604 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.438510 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f8c3c03-a1fc-4376-98c6-feeeefa08068-run-openvswitch\") pod \"ovnkube-node-h9hsf\" (UID: \"0f8c3c03-a1fc-4376-98c6-feeeefa08068\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:13:54.438604 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.438534 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6a21f264-1e80-4eab-8c6e-27ca9ca19f0d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jsctc\" (UID: \"6a21f264-1e80-4eab-8c6e-27ca9ca19f0d\") " pod="openshift-multus/multus-additional-cni-plugins-jsctc" Oct 31 00:13:54.438604 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.438568 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eadb6436-5e5b-4116-8ed7-26aa935fa749-host\") pod \"node-ca-xlzv2\" (UID: \"eadb6436-5e5b-4116-8ed7-26aa935fa749\") " pod="openshift-image-registry/node-ca-xlzv2" Oct 31 00:13:54.439096 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.438628 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9801ff00-9e9a-4141-8212-ca32bf0251b3-multus-socket-dir-parent\") pod \"multus-b6vpd\" (UID: \"9801ff00-9e9a-4141-8212-ca32bf0251b3\") " pod="openshift-multus/multus-b6vpd" Oct 31 00:13:54.439096 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.438655 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9801ff00-9e9a-4141-8212-ca32bf0251b3-hostroot\") pod \"multus-b6vpd\" (UID: \"9801ff00-9e9a-4141-8212-ca32bf0251b3\") " pod="openshift-multus/multus-b6vpd" Oct 31 00:13:54.439096 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.438673 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a077dea4-b5a3-4335-bb6a-7618e794eb8f-metrics-certs\") pod \"network-metrics-daemon-rjfn2\" (UID: \"a077dea4-b5a3-4335-bb6a-7618e794eb8f\") " pod="openshift-multus/network-metrics-daemon-rjfn2" Oct 31 00:13:54.439096 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.438688 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0f8c3c03-a1fc-4376-98c6-feeeefa08068-host-slash\") pod \"ovnkube-node-h9hsf\" (UID: \"0f8c3c03-a1fc-4376-98c6-feeeefa08068\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:13:54.439096 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.438702 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6a21f264-1e80-4eab-8c6e-27ca9ca19f0d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jsctc\" (UID: \"6a21f264-1e80-4eab-8c6e-27ca9ca19f0d\") " pod="openshift-multus/multus-additional-cni-plugins-jsctc" Oct 31 00:13:54.439096 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.438718 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0eb4888f-40a8-4aba-85d8-3c845382bd02-lib-modules\") pod \"tuned-vsfrc\" (UID: \"0eb4888f-40a8-4aba-85d8-3c845382bd02\") " pod="openshift-cluster-node-tuning-operator/tuned-vsfrc" Oct 31 00:13:54.439096 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.438748 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2ed1e615-cbca-482c-89cc-0fe7d8e7e597-konnectivity-ca\") pod \"konnectivity-agent-lvp2h\" (UID: \"2ed1e615-cbca-482c-89cc-0fe7d8e7e597\") " pod="kube-system/konnectivity-agent-lvp2h" Oct 31 00:13:54.439096 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.438779 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1bc3d212-b67c-4368-90a3-8e080a4f1452-registration-dir\") pod \"aws-ebs-csi-driver-node-ndd7n\" (UID: \"1bc3d212-b67c-4368-90a3-8e080a4f1452\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ndd7n" Oct 31 00:13:54.439096 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.438813 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0f8c3c03-a1fc-4376-98c6-feeeefa08068-host-cni-netd\") pod \"ovnkube-node-h9hsf\" (UID: \"0f8c3c03-a1fc-4376-98c6-feeeefa08068\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:13:54.439096 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.438858 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wrw4\" (UniqueName: \"kubernetes.io/projected/0f8c3c03-a1fc-4376-98c6-feeeefa08068-kube-api-access-8wrw4\") pod \"ovnkube-node-h9hsf\" (UID: \"0f8c3c03-a1fc-4376-98c6-feeeefa08068\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:13:54.439096 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.438882 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1bc3d212-b67c-4368-90a3-8e080a4f1452-sys-fs\") pod \"aws-ebs-csi-driver-node-ndd7n\" (UID: \"1bc3d212-b67c-4368-90a3-8e080a4f1452\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ndd7n" Oct 31 00:13:54.439096 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.438897 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9801ff00-9e9a-4141-8212-ca32bf0251b3-etc-kubernetes\") pod \"multus-b6vpd\" (UID: \"9801ff00-9e9a-4141-8212-ca32bf0251b3\") " pod="openshift-multus/multus-b6vpd" Oct 31 00:13:54.439096 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.438932 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0eb4888f-40a8-4aba-85d8-3c845382bd02-etc-sysctl-conf\") pod \"tuned-vsfrc\" (UID: \"0eb4888f-40a8-4aba-85d8-3c845382bd02\") " pod="openshift-cluster-node-tuning-operator/tuned-vsfrc" Oct 31 00:13:54.439096 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.438969 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/eadb6436-5e5b-4116-8ed7-26aa935fa749-serviceca\") pod \"node-ca-xlzv2\" (UID: \"eadb6436-5e5b-4116-8ed7-26aa935fa749\") " pod="openshift-image-registry/node-ca-xlzv2" Oct 31 00:13:54.439096 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.438987 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0f8c3c03-a1fc-4376-98c6-feeeefa08068-systemd-units\") pod \"ovnkube-node-h9hsf\" (UID: \"0f8c3c03-a1fc-4376-98c6-feeeefa08068\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:13:54.439096 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.439008 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0f8c3c03-a1fc-4376-98c6-feeeefa08068-log-socket\") pod \"ovnkube-node-h9hsf\" (UID: \"0f8c3c03-a1fc-4376-98c6-feeeefa08068\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:13:54.439657 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.439023 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0f8c3c03-a1fc-4376-98c6-feeeefa08068-host-cni-bin\") pod \"ovnkube-node-h9hsf\" (UID: \"0f8c3c03-a1fc-4376-98c6-feeeefa08068\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:13:54.439657 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.439055 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1bc3d212-b67c-4368-90a3-8e080a4f1452-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ndd7n\" (UID: \"1bc3d212-b67c-4368-90a3-8e080a4f1452\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ndd7n" Oct 31 00:13:54.439657 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.439085 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0f8c3c03-a1fc-4376-98c6-feeeefa08068-host-run-ovn-kubernetes\") pod \"ovnkube-node-h9hsf\" (UID: \"0f8c3c03-a1fc-4376-98c6-feeeefa08068\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:13:54.439657 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.439121 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0eb4888f-40a8-4aba-85d8-3c845382bd02-etc-sysctl-d\") pod \"tuned-vsfrc\" (UID: \"0eb4888f-40a8-4aba-85d8-3c845382bd02\") " pod="openshift-cluster-node-tuning-operator/tuned-vsfrc" Oct 31 00:13:54.439657 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.439157 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0eb4888f-40a8-4aba-85d8-3c845382bd02-tmp\") pod \"tuned-vsfrc\" (UID: \"0eb4888f-40a8-4aba-85d8-3c845382bd02\") " pod="openshift-cluster-node-tuning-operator/tuned-vsfrc" Oct 31 00:13:54.439657 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.439186 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrkcn\" (UniqueName: \"kubernetes.io/projected/0eb4888f-40a8-4aba-85d8-3c845382bd02-kube-api-access-nrkcn\") pod \"tuned-vsfrc\" (UID: \"0eb4888f-40a8-4aba-85d8-3c845382bd02\") " pod="openshift-cluster-node-tuning-operator/tuned-vsfrc" Oct 31 00:13:54.439657 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.439228 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2ed1e615-cbca-482c-89cc-0fe7d8e7e597-agent-certs\") pod \"konnectivity-agent-lvp2h\" (UID: \"2ed1e615-cbca-482c-89cc-0fe7d8e7e597\") " pod="kube-system/konnectivity-agent-lvp2h" Oct 31 00:13:54.439657 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.439251 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1bc3d212-b67c-4368-90a3-8e080a4f1452-etc-selinux\") pod \"aws-ebs-csi-driver-node-ndd7n\" (UID: \"1bc3d212-b67c-4368-90a3-8e080a4f1452\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ndd7n" Oct 31 00:13:54.439657 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.439267 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a7d0a920-91fe-4c4a-b9bc-67550c324e7f-hosts-file\") pod \"node-resolver-gkm2c\" (UID: \"a7d0a920-91fe-4c4a-b9bc-67550c324e7f\") " pod="openshift-dns/node-resolver-gkm2c" Oct 31 00:13:54.439657 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.439286 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0f8c3c03-a1fc-4376-98c6-feeeefa08068-run-ovn\") pod \"ovnkube-node-h9hsf\" (UID: \"0f8c3c03-a1fc-4376-98c6-feeeefa08068\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:13:54.439657 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.439312 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0eb4888f-40a8-4aba-85d8-3c845382bd02-etc-kubernetes\") pod \"tuned-vsfrc\" (UID: \"0eb4888f-40a8-4aba-85d8-3c845382bd02\") " pod="openshift-cluster-node-tuning-operator/tuned-vsfrc" Oct 31 00:13:54.439657 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.439360 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d02ae6b5-4860-45b3-8d16-4a49531b77d4-original-pull-secret\") pod \"global-pull-secret-syncer-fwgtx\" (UID: \"d02ae6b5-4860-45b3-8d16-4a49531b77d4\") " pod="kube-system/global-pull-secret-syncer-fwgtx" Oct 31 00:13:54.439657 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.439410 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9801ff00-9e9a-4141-8212-ca32bf0251b3-host-var-lib-cni-multus\") pod \"multus-b6vpd\" (UID: \"9801ff00-9e9a-4141-8212-ca32bf0251b3\") " pod="openshift-multus/multus-b6vpd" Oct 31 00:13:54.439657 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.439434 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7d0a920-91fe-4c4a-b9bc-67550c324e7f-tmp-dir\") pod \"node-resolver-gkm2c\" (UID: \"a7d0a920-91fe-4c4a-b9bc-67550c324e7f\") " pod="openshift-dns/node-resolver-gkm2c" Oct 31 00:13:54.439657 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.439455 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djjgz\" (UniqueName: \"kubernetes.io/projected/a7d0a920-91fe-4c4a-b9bc-67550c324e7f-kube-api-access-djjgz\") pod \"node-resolver-gkm2c\" (UID: \"a7d0a920-91fe-4c4a-b9bc-67550c324e7f\") " pod="openshift-dns/node-resolver-gkm2c" Oct 31 00:13:54.439657 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.439502 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0f8c3c03-a1fc-4376-98c6-feeeefa08068-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-h9hsf\" (UID: \"0f8c3c03-a1fc-4376-98c6-feeeefa08068\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:13:54.440167 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.439531 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0eb4888f-40a8-4aba-85d8-3c845382bd02-sys\") pod \"tuned-vsfrc\" (UID: \"0eb4888f-40a8-4aba-85d8-3c845382bd02\") " pod="openshift-cluster-node-tuning-operator/tuned-vsfrc" Oct 31 00:13:54.440167 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.439554 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d02ae6b5-4860-45b3-8d16-4a49531b77d4-dbus\") pod \"global-pull-secret-syncer-fwgtx\" (UID: \"d02ae6b5-4860-45b3-8d16-4a49531b77d4\") " pod="kube-system/global-pull-secret-syncer-fwgtx" Oct 31 00:13:54.440167 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.439576 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f8c3c03-a1fc-4376-98c6-feeeefa08068-etc-openvswitch\") pod \"ovnkube-node-h9hsf\" (UID: \"0f8c3c03-a1fc-4376-98c6-feeeefa08068\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:13:54.440167 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.439594 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6a21f264-1e80-4eab-8c6e-27ca9ca19f0d-cnibin\") pod \"multus-additional-cni-plugins-jsctc\" (UID: \"6a21f264-1e80-4eab-8c6e-27ca9ca19f0d\") " pod="openshift-multus/multus-additional-cni-plugins-jsctc" Oct 31 00:13:54.440167 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.439608 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xc9bh\" (UniqueName: \"kubernetes.io/projected/a077dea4-b5a3-4335-bb6a-7618e794eb8f-kube-api-access-xc9bh\") pod \"network-metrics-daemon-rjfn2\" (UID: \"a077dea4-b5a3-4335-bb6a-7618e794eb8f\") " pod="openshift-multus/network-metrics-daemon-rjfn2" Oct 31 00:13:54.440167 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.439650 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0f8c3c03-a1fc-4376-98c6-feeeefa08068-env-overrides\") pod \"ovnkube-node-h9hsf\" (UID: \"0f8c3c03-a1fc-4376-98c6-feeeefa08068\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:13:54.440167 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.439684 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0f8c3c03-a1fc-4376-98c6-feeeefa08068-ovnkube-script-lib\") pod \"ovnkube-node-h9hsf\" (UID: \"0f8c3c03-a1fc-4376-98c6-feeeefa08068\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:13:54.440167 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.439710 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0eb4888f-40a8-4aba-85d8-3c845382bd02-etc-systemd\") pod \"tuned-vsfrc\" (UID: \"0eb4888f-40a8-4aba-85d8-3c845382bd02\") " pod="openshift-cluster-node-tuning-operator/tuned-vsfrc" Oct 31 00:13:54.440167 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.439732 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1bc3d212-b67c-4368-90a3-8e080a4f1452-socket-dir\") pod \"aws-ebs-csi-driver-node-ndd7n\" (UID: \"1bc3d212-b67c-4368-90a3-8e080a4f1452\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ndd7n" Oct 31 00:13:54.440167 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.439756 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0f8c3c03-a1fc-4376-98c6-feeeefa08068-node-log\") pod \"ovnkube-node-h9hsf\" (UID: \"0f8c3c03-a1fc-4376-98c6-feeeefa08068\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:13:54.440167 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.439778 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0f8c3c03-a1fc-4376-98c6-feeeefa08068-ovnkube-config\") pod \"ovnkube-node-h9hsf\" (UID: \"0f8c3c03-a1fc-4376-98c6-feeeefa08068\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:13:54.440167 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.439803 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0f8c3c03-a1fc-4376-98c6-feeeefa08068-ovn-node-metrics-cert\") pod \"ovnkube-node-h9hsf\" (UID: \"0f8c3c03-a1fc-4376-98c6-feeeefa08068\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:13:54.440167 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.439838 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0eb4888f-40a8-4aba-85d8-3c845382bd02-etc-sysconfig\") pod \"tuned-vsfrc\" (UID: \"0eb4888f-40a8-4aba-85d8-3c845382bd02\") " pod="openshift-cluster-node-tuning-operator/tuned-vsfrc" Oct 31 00:13:54.457146 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.457125 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 31 00:13:54.457671 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.457647 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-7hvcc\"" Oct 31 00:13:54.457763 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.457689 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 31 00:13:54.457908 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.457893 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-qdpqj\"" Oct 31 00:13:54.458261 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.458249 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-bvv2r\"" Oct 31 00:13:54.458320 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.458265 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-5jgbw\"" Oct 31 00:13:54.458320 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.458290 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-xfr46\"" Oct 31 00:13:54.458475 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.458460 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-v2pb4\"" Oct 31 00:13:54.459060 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.459045 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 31 00:13:54.459060 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.459057 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 31 00:13:54.459180 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.459129 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 31 00:13:54.459180 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.459135 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 31 00:13:54.459280 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.459216 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 31 00:13:54.459280 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.459220 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 31 00:13:54.459280 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.459234 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 31 00:13:54.459280 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.459236 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 31 00:13:54.459280 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.459260 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 31 00:13:54.459280 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.459280 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 31 00:13:54.459499 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.459295 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 31 00:13:54.459499 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.459316 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 31 00:13:54.459499 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.459351 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 31 00:13:54.461107 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.461092 2581 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 31 00:13:54.461146 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.461141 2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-ql2vw" Oct 31 00:13:54.541088 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.541053 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d02ae6b5-4860-45b3-8d16-4a49531b77d4-original-pull-secret\") pod \"global-pull-secret-syncer-fwgtx\" (UID: \"d02ae6b5-4860-45b3-8d16-4a49531b77d4\") " pod="kube-system/global-pull-secret-syncer-fwgtx" Oct 31 00:13:54.541088 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.541092 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9801ff00-9e9a-4141-8212-ca32bf0251b3-host-var-lib-cni-multus\") pod \"multus-b6vpd\" (UID: \"9801ff00-9e9a-4141-8212-ca32bf0251b3\") " pod="openshift-multus/multus-b6vpd" Oct 31 00:13:54.541293 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.541107 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7d0a920-91fe-4c4a-b9bc-67550c324e7f-tmp-dir\") pod \"node-resolver-gkm2c\" (UID: \"a7d0a920-91fe-4c4a-b9bc-67550c324e7f\") " pod="openshift-dns/node-resolver-gkm2c" Oct 31 00:13:54.541293 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.541185 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djjgz\" (UniqueName: \"kubernetes.io/projected/a7d0a920-91fe-4c4a-b9bc-67550c324e7f-kube-api-access-djjgz\") pod \"node-resolver-gkm2c\" (UID: \"a7d0a920-91fe-4c4a-b9bc-67550c324e7f\") " pod="openshift-dns/node-resolver-gkm2c" Oct 31 00:13:54.541293 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.541193 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9801ff00-9e9a-4141-8212-ca32bf0251b3-host-var-lib-cni-multus\") pod \"multus-b6vpd\" (UID: \"9801ff00-9e9a-4141-8212-ca32bf0251b3\") " pod="openshift-multus/multus-b6vpd" Oct 31 00:13:54.541293 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.541201 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0f8c3c03-a1fc-4376-98c6-feeeefa08068-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-h9hsf\" (UID: \"0f8c3c03-a1fc-4376-98c6-feeeefa08068\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:13:54.541293 ip-10-0-137-56 kubenswrapper[2581]: E1031 00:13:54.541224 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 00:13:54.541293 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.541261 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0f8c3c03-a1fc-4376-98c6-feeeefa08068-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-h9hsf\" (UID: \"0f8c3c03-a1fc-4376-98c6-feeeefa08068\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:13:54.541293 ip-10-0-137-56 kubenswrapper[2581]: E1031 00:13:54.541296 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d02ae6b5-4860-45b3-8d16-4a49531b77d4-original-pull-secret podName:d02ae6b5-4860-45b3-8d16-4a49531b77d4 nodeName:}" failed. No retries permitted until 2026-10-31 00:13:55.041276309 +0000 UTC m=+2.273811757 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d02ae6b5-4860-45b3-8d16-4a49531b77d4-original-pull-secret") pod "global-pull-secret-syncer-fwgtx" (UID: "d02ae6b5-4860-45b3-8d16-4a49531b77d4") : object "kube-system"/"original-pull-secret" not registered Oct 31 00:13:54.541650 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.541226 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0eb4888f-40a8-4aba-85d8-3c845382bd02-sys\") pod \"tuned-vsfrc\" (UID: \"0eb4888f-40a8-4aba-85d8-3c845382bd02\") " pod="openshift-cluster-node-tuning-operator/tuned-vsfrc" Oct 31 00:13:54.541650 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.541310 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0eb4888f-40a8-4aba-85d8-3c845382bd02-sys\") pod \"tuned-vsfrc\" (UID: \"0eb4888f-40a8-4aba-85d8-3c845382bd02\") " pod="openshift-cluster-node-tuning-operator/tuned-vsfrc" Oct 31 00:13:54.541650 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.541338 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d02ae6b5-4860-45b3-8d16-4a49531b77d4-dbus\") pod \"global-pull-secret-syncer-fwgtx\" (UID: \"d02ae6b5-4860-45b3-8d16-4a49531b77d4\") " pod="kube-system/global-pull-secret-syncer-fwgtx" Oct 31 00:13:54.541650 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.541358 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f8c3c03-a1fc-4376-98c6-feeeefa08068-etc-openvswitch\") pod \"ovnkube-node-h9hsf\" (UID: \"0f8c3c03-a1fc-4376-98c6-feeeefa08068\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:13:54.541650 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.541397 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6a21f264-1e80-4eab-8c6e-27ca9ca19f0d-cnibin\") pod \"multus-additional-cni-plugins-jsctc\" (UID: \"6a21f264-1e80-4eab-8c6e-27ca9ca19f0d\") " pod="openshift-multus/multus-additional-cni-plugins-jsctc" Oct 31 00:13:54.541650 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.541422 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xc9bh\" (UniqueName: \"kubernetes.io/projected/a077dea4-b5a3-4335-bb6a-7618e794eb8f-kube-api-access-xc9bh\") pod \"network-metrics-daemon-rjfn2\" (UID: \"a077dea4-b5a3-4335-bb6a-7618e794eb8f\") " pod="openshift-multus/network-metrics-daemon-rjfn2" Oct 31 00:13:54.541650 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.541438 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f8c3c03-a1fc-4376-98c6-feeeefa08068-etc-openvswitch\") pod \"ovnkube-node-h9hsf\" (UID: \"0f8c3c03-a1fc-4376-98c6-feeeefa08068\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:13:54.541650 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.541445 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6a21f264-1e80-4eab-8c6e-27ca9ca19f0d-cnibin\") pod \"multus-additional-cni-plugins-jsctc\" (UID: \"6a21f264-1e80-4eab-8c6e-27ca9ca19f0d\") " pod="openshift-multus/multus-additional-cni-plugins-jsctc" Oct 31 00:13:54.541650 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.541445 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0f8c3c03-a1fc-4376-98c6-feeeefa08068-env-overrides\") pod \"ovnkube-node-h9hsf\" (UID: \"0f8c3c03-a1fc-4376-98c6-feeeefa08068\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:13:54.541650 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.541481 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0f8c3c03-a1fc-4376-98c6-feeeefa08068-ovnkube-script-lib\") pod \"ovnkube-node-h9hsf\" (UID: \"0f8c3c03-a1fc-4376-98c6-feeeefa08068\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:13:54.541650 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.541507 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0eb4888f-40a8-4aba-85d8-3c845382bd02-etc-systemd\") pod \"tuned-vsfrc\" (UID: \"0eb4888f-40a8-4aba-85d8-3c845382bd02\") " pod="openshift-cluster-node-tuning-operator/tuned-vsfrc" Oct 31 00:13:54.541650 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.541540 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d02ae6b5-4860-45b3-8d16-4a49531b77d4-dbus\") pod \"global-pull-secret-syncer-fwgtx\" (UID: \"d02ae6b5-4860-45b3-8d16-4a49531b77d4\") " pod="kube-system/global-pull-secret-syncer-fwgtx" Oct 31 00:13:54.541650 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.541532 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1bc3d212-b67c-4368-90a3-8e080a4f1452-socket-dir\") pod \"aws-ebs-csi-driver-node-ndd7n\" (UID: \"1bc3d212-b67c-4368-90a3-8e080a4f1452\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ndd7n" Oct 31 00:13:54.541650 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.541610 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7d0a920-91fe-4c4a-b9bc-67550c324e7f-tmp-dir\") pod \"node-resolver-gkm2c\" (UID: \"a7d0a920-91fe-4c4a-b9bc-67550c324e7f\") " pod="openshift-dns/node-resolver-gkm2c" Oct 31 00:13:54.541650 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.541606 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0eb4888f-40a8-4aba-85d8-3c845382bd02-etc-systemd\") pod \"tuned-vsfrc\" (UID: \"0eb4888f-40a8-4aba-85d8-3c845382bd02\") " pod="openshift-cluster-node-tuning-operator/tuned-vsfrc" Oct 31 00:13:54.542320 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.541670 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1bc3d212-b67c-4368-90a3-8e080a4f1452-socket-dir\") pod \"aws-ebs-csi-driver-node-ndd7n\" (UID: \"1bc3d212-b67c-4368-90a3-8e080a4f1452\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ndd7n" Oct 31 00:13:54.542320 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.541660 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0f8c3c03-a1fc-4376-98c6-feeeefa08068-node-log\") pod \"ovnkube-node-h9hsf\" (UID: \"0f8c3c03-a1fc-4376-98c6-feeeefa08068\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:13:54.542320 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.541729 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0f8c3c03-a1fc-4376-98c6-feeeefa08068-ovnkube-config\") pod \"ovnkube-node-h9hsf\" (UID: \"0f8c3c03-a1fc-4376-98c6-feeeefa08068\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:13:54.542320 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.541776 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0f8c3c03-a1fc-4376-98c6-feeeefa08068-node-log\") pod \"ovnkube-node-h9hsf\" (UID: \"0f8c3c03-a1fc-4376-98c6-feeeefa08068\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:13:54.542320 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.541828 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0f8c3c03-a1fc-4376-98c6-feeeefa08068-ovn-node-metrics-cert\") pod \"ovnkube-node-h9hsf\" (UID: \"0f8c3c03-a1fc-4376-98c6-feeeefa08068\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:13:54.542320 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.542000 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0f8c3c03-a1fc-4376-98c6-feeeefa08068-env-overrides\") pod \"ovnkube-node-h9hsf\" (UID: \"0f8c3c03-a1fc-4376-98c6-feeeefa08068\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:13:54.542320 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.542055 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0eb4888f-40a8-4aba-85d8-3c845382bd02-etc-sysconfig\") pod \"tuned-vsfrc\" (UID: \"0eb4888f-40a8-4aba-85d8-3c845382bd02\") " pod="openshift-cluster-node-tuning-operator/tuned-vsfrc" Oct 31 00:13:54.542320 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.542075 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0eb4888f-40a8-4aba-85d8-3c845382bd02-etc-tuned\") pod \"tuned-vsfrc\" (UID: \"0eb4888f-40a8-4aba-85d8-3c845382bd02\") " pod="openshift-cluster-node-tuning-operator/tuned-vsfrc" Oct 31 00:13:54.542320 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.542089 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9801ff00-9e9a-4141-8212-ca32bf0251b3-host-run-netns\") pod \"multus-b6vpd\" (UID: \"9801ff00-9e9a-4141-8212-ca32bf0251b3\") " pod="openshift-multus/multus-b6vpd" Oct 31 00:13:54.542320 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.542129 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0f8c3c03-a1fc-4376-98c6-feeeefa08068-ovnkube-script-lib\") pod \"ovnkube-node-h9hsf\" (UID: \"0f8c3c03-a1fc-4376-98c6-feeeefa08068\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:13:54.542320 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.542159 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0f8c3c03-a1fc-4376-98c6-feeeefa08068-ovnkube-config\") pod \"ovnkube-node-h9hsf\" (UID: \"0f8c3c03-a1fc-4376-98c6-feeeefa08068\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:13:54.542320 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.542162 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0eb4888f-40a8-4aba-85d8-3c845382bd02-etc-sysconfig\") pod \"tuned-vsfrc\" (UID: \"0eb4888f-40a8-4aba-85d8-3c845382bd02\") " pod="openshift-cluster-node-tuning-operator/tuned-vsfrc" Oct 31 00:13:54.542320 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.542220 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9801ff00-9e9a-4141-8212-ca32bf0251b3-host-var-lib-cni-bin\") pod \"multus-b6vpd\" (UID: \"9801ff00-9e9a-4141-8212-ca32bf0251b3\") " pod="openshift-multus/multus-b6vpd" Oct 31 00:13:54.542320 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.542236 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0f8c3c03-a1fc-4376-98c6-feeeefa08068-run-systemd\") pod \"ovnkube-node-h9hsf\" (UID: \"0f8c3c03-a1fc-4376-98c6-feeeefa08068\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:13:54.542320 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.542239 2581 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 31 00:13:54.542320 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.542251 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0eb4888f-40a8-4aba-85d8-3c845382bd02-etc-modprobe-d\") pod \"tuned-vsfrc\" (UID: \"0eb4888f-40a8-4aba-85d8-3c845382bd02\") " pod="openshift-cluster-node-tuning-operator/tuned-vsfrc" Oct 31 00:13:54.542320 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.542276 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9801ff00-9e9a-4141-8212-ca32bf0251b3-host-var-lib-cni-bin\") pod \"multus-b6vpd\" (UID: \"9801ff00-9e9a-4141-8212-ca32bf0251b3\") " pod="openshift-multus/multus-b6vpd" Oct 31 00:13:54.542320 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.542288 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0eb4888f-40a8-4aba-85d8-3c845382bd02-host\") pod \"tuned-vsfrc\" (UID: \"0eb4888f-40a8-4aba-85d8-3c845382bd02\") " pod="openshift-cluster-node-tuning-operator/tuned-vsfrc" Oct 31 00:13:54.543055 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.542246 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9801ff00-9e9a-4141-8212-ca32bf0251b3-host-run-netns\") pod \"multus-b6vpd\" (UID: \"9801ff00-9e9a-4141-8212-ca32bf0251b3\") " pod="openshift-multus/multus-b6vpd" Oct 31 00:13:54.543055 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.542292 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0f8c3c03-a1fc-4376-98c6-feeeefa08068-run-systemd\") pod \"ovnkube-node-h9hsf\" (UID: \"0f8c3c03-a1fc-4376-98c6-feeeefa08068\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:13:54.543055 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.542328 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbpb7\" (UniqueName: \"kubernetes.io/projected/978a9050-dddb-4869-85d8-84a001554c5b-kube-api-access-gbpb7\") pod \"iptables-alerter-4nwn7\" (UID: \"978a9050-dddb-4869-85d8-84a001554c5b\") " pod="openshift-network-operator/iptables-alerter-4nwn7" Oct 31 00:13:54.543055 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.542344 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0eb4888f-40a8-4aba-85d8-3c845382bd02-host\") pod \"tuned-vsfrc\" (UID: \"0eb4888f-40a8-4aba-85d8-3c845382bd02\") " pod="openshift-cluster-node-tuning-operator/tuned-vsfrc" Oct 31 00:13:54.543055 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.542363 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0eb4888f-40a8-4aba-85d8-3c845382bd02-etc-modprobe-d\") pod \"tuned-vsfrc\" (UID: \"0eb4888f-40a8-4aba-85d8-3c845382bd02\") " pod="openshift-cluster-node-tuning-operator/tuned-vsfrc" Oct 31 00:13:54.543055 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.542395 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nz8r2\" (UniqueName: \"kubernetes.io/projected/eadb6436-5e5b-4116-8ed7-26aa935fa749-kube-api-access-nz8r2\") pod \"node-ca-xlzv2\" (UID: \"eadb6436-5e5b-4116-8ed7-26aa935fa749\") " pod="openshift-image-registry/node-ca-xlzv2" Oct 31 00:13:54.543055 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.542426 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1bc3d212-b67c-4368-90a3-8e080a4f1452-device-dir\") pod \"aws-ebs-csi-driver-node-ndd7n\" (UID: \"1bc3d212-b67c-4368-90a3-8e080a4f1452\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ndd7n" Oct 31 00:13:54.543055 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.542457 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9801ff00-9e9a-4141-8212-ca32bf0251b3-cni-binary-copy\") pod \"multus-b6vpd\" (UID: \"9801ff00-9e9a-4141-8212-ca32bf0251b3\") " pod="openshift-multus/multus-b6vpd" Oct 31 00:13:54.543055 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.542484 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9801ff00-9e9a-4141-8212-ca32bf0251b3-host-var-lib-kubelet\") pod \"multus-b6vpd\" (UID: \"9801ff00-9e9a-4141-8212-ca32bf0251b3\") " pod="openshift-multus/multus-b6vpd" Oct 31 00:13:54.543055 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.542494 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1bc3d212-b67c-4368-90a3-8e080a4f1452-device-dir\") pod \"aws-ebs-csi-driver-node-ndd7n\" (UID: \"1bc3d212-b67c-4368-90a3-8e080a4f1452\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ndd7n" Oct 31 00:13:54.543055 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.542522 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzs4x\" (UniqueName: \"kubernetes.io/projected/9801ff00-9e9a-4141-8212-ca32bf0251b3-kube-api-access-xzs4x\") pod \"multus-b6vpd\" (UID: \"9801ff00-9e9a-4141-8212-ca32bf0251b3\") " pod="openshift-multus/multus-b6vpd" Oct 31 00:13:54.543055 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.542548 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8j6tz\" (UniqueName: \"kubernetes.io/projected/6a21f264-1e80-4eab-8c6e-27ca9ca19f0d-kube-api-access-8j6tz\") pod \"multus-additional-cni-plugins-jsctc\" (UID: \"6a21f264-1e80-4eab-8c6e-27ca9ca19f0d\") " pod="openshift-multus/multus-additional-cni-plugins-jsctc" Oct 31 00:13:54.543055 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.542547 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9801ff00-9e9a-4141-8212-ca32bf0251b3-host-var-lib-kubelet\") pod \"multus-b6vpd\" (UID: \"9801ff00-9e9a-4141-8212-ca32bf0251b3\") " pod="openshift-multus/multus-b6vpd" Oct 31 00:13:54.543055 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.542838 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0eb4888f-40a8-4aba-85d8-3c845382bd02-run\") pod \"tuned-vsfrc\" (UID: \"0eb4888f-40a8-4aba-85d8-3c845382bd02\") " pod="openshift-cluster-node-tuning-operator/tuned-vsfrc" Oct 31 00:13:54.543055 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.542945 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9801ff00-9e9a-4141-8212-ca32bf0251b3-cni-binary-copy\") pod \"multus-b6vpd\" (UID: \"9801ff00-9e9a-4141-8212-ca32bf0251b3\") " pod="openshift-multus/multus-b6vpd" Oct 31 00:13:54.543055 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.542575 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0eb4888f-40a8-4aba-85d8-3c845382bd02-run\") pod \"tuned-vsfrc\" (UID: \"0eb4888f-40a8-4aba-85d8-3c845382bd02\") " pod="openshift-cluster-node-tuning-operator/tuned-vsfrc" Oct 31 00:13:54.543055 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.543036 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9801ff00-9e9a-4141-8212-ca32bf0251b3-system-cni-dir\") pod \"multus-b6vpd\" (UID: \"9801ff00-9e9a-4141-8212-ca32bf0251b3\") " pod="openshift-multus/multus-b6vpd" Oct 31 00:13:54.543990 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.543072 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0f8c3c03-a1fc-4376-98c6-feeeefa08068-host-kubelet\") pod \"ovnkube-node-h9hsf\" (UID: \"0f8c3c03-a1fc-4376-98c6-feeeefa08068\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:13:54.543990 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.543095 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9801ff00-9e9a-4141-8212-ca32bf0251b3-system-cni-dir\") pod \"multus-b6vpd\" (UID: \"9801ff00-9e9a-4141-8212-ca32bf0251b3\") " pod="openshift-multus/multus-b6vpd" Oct 31 00:13:54.543990 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.543097 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0f8c3c03-a1fc-4376-98c6-feeeefa08068-host-run-netns\") pod \"ovnkube-node-h9hsf\" (UID: \"0f8c3c03-a1fc-4376-98c6-feeeefa08068\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:13:54.543990 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.543132 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f8c3c03-a1fc-4376-98c6-feeeefa08068-var-lib-openvswitch\") pod \"ovnkube-node-h9hsf\" (UID: \"0f8c3c03-a1fc-4376-98c6-feeeefa08068\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:13:54.543990 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.543140 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0f8c3c03-a1fc-4376-98c6-feeeefa08068-host-run-netns\") pod \"ovnkube-node-h9hsf\" (UID: \"0f8c3c03-a1fc-4376-98c6-feeeefa08068\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:13:54.543990 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.543158 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6a21f264-1e80-4eab-8c6e-27ca9ca19f0d-system-cni-dir\") pod \"multus-additional-cni-plugins-jsctc\" (UID: \"6a21f264-1e80-4eab-8c6e-27ca9ca19f0d\") " pod="openshift-multus/multus-additional-cni-plugins-jsctc" Oct 31 00:13:54.543990 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.543297 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0eb4888f-40a8-4aba-85d8-3c845382bd02-var-lib-kubelet\") pod \"tuned-vsfrc\" (UID: \"0eb4888f-40a8-4aba-85d8-3c845382bd02\") " pod="openshift-cluster-node-tuning-operator/tuned-vsfrc" Oct 31 00:13:54.543990 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.543298 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6a21f264-1e80-4eab-8c6e-27ca9ca19f0d-system-cni-dir\") pod \"multus-additional-cni-plugins-jsctc\" (UID: \"6a21f264-1e80-4eab-8c6e-27ca9ca19f0d\") " pod="openshift-multus/multus-additional-cni-plugins-jsctc" Oct 31 00:13:54.543990 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.543311 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0f8c3c03-a1fc-4376-98c6-feeeefa08068-host-kubelet\") pod \"ovnkube-node-h9hsf\" (UID: \"0f8c3c03-a1fc-4376-98c6-feeeefa08068\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:13:54.543990 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.543350 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f8c3c03-a1fc-4376-98c6-feeeefa08068-var-lib-openvswitch\") pod \"ovnkube-node-h9hsf\" (UID: \"0f8c3c03-a1fc-4376-98c6-feeeefa08068\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:13:54.543990 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.543326 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/978a9050-dddb-4869-85d8-84a001554c5b-iptables-alerter-script\") pod \"iptables-alerter-4nwn7\" (UID: \"978a9050-dddb-4869-85d8-84a001554c5b\") " pod="openshift-network-operator/iptables-alerter-4nwn7" Oct 31 00:13:54.543990 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.543401 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/978a9050-dddb-4869-85d8-84a001554c5b-host-slash\") pod \"iptables-alerter-4nwn7\" (UID: \"978a9050-dddb-4869-85d8-84a001554c5b\") " pod="openshift-network-operator/iptables-alerter-4nwn7" Oct 31 00:13:54.543990 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.543425 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wm7wn\" (UniqueName: \"kubernetes.io/projected/1bc3d212-b67c-4368-90a3-8e080a4f1452-kube-api-access-wm7wn\") pod \"aws-ebs-csi-driver-node-ndd7n\" (UID: \"1bc3d212-b67c-4368-90a3-8e080a4f1452\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ndd7n" Oct 31 00:13:54.543990 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.543450 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9801ff00-9e9a-4141-8212-ca32bf0251b3-multus-cni-dir\") pod \"multus-b6vpd\" (UID: \"9801ff00-9e9a-4141-8212-ca32bf0251b3\") " pod="openshift-multus/multus-b6vpd" Oct 31 00:13:54.543990 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.543474 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/978a9050-dddb-4869-85d8-84a001554c5b-host-slash\") pod \"iptables-alerter-4nwn7\" (UID: \"978a9050-dddb-4869-85d8-84a001554c5b\") " pod="openshift-network-operator/iptables-alerter-4nwn7" Oct 31 00:13:54.543990 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.543475 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9801ff00-9e9a-4141-8212-ca32bf0251b3-host-run-k8s-cni-cncf-io\") pod \"multus-b6vpd\" (UID: \"9801ff00-9e9a-4141-8212-ca32bf0251b3\") " pod="openshift-multus/multus-b6vpd" Oct 31 00:13:54.543990 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.543506 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9801ff00-9e9a-4141-8212-ca32bf0251b3-multus-daemon-config\") pod \"multus-b6vpd\" (UID: \"9801ff00-9e9a-4141-8212-ca32bf0251b3\") " pod="openshift-multus/multus-b6vpd" Oct 31 00:13:54.544762 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.543517 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9801ff00-9e9a-4141-8212-ca32bf0251b3-host-run-k8s-cni-cncf-io\") pod \"multus-b6vpd\" (UID: \"9801ff00-9e9a-4141-8212-ca32bf0251b3\") " pod="openshift-multus/multus-b6vpd" Oct 31 00:13:54.544762 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.543356 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0eb4888f-40a8-4aba-85d8-3c845382bd02-var-lib-kubelet\") pod \"tuned-vsfrc\" (UID: \"0eb4888f-40a8-4aba-85d8-3c845382bd02\") " pod="openshift-cluster-node-tuning-operator/tuned-vsfrc" Oct 31 00:13:54.544762 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.543536 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9801ff00-9e9a-4141-8212-ca32bf0251b3-host-run-multus-certs\") pod \"multus-b6vpd\" (UID: \"9801ff00-9e9a-4141-8212-ca32bf0251b3\") " pod="openshift-multus/multus-b6vpd" Oct 31 00:13:54.544762 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.543570 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ms4cs\" (UniqueName: \"kubernetes.io/projected/0cbf0042-931c-4135-9330-f9e798b7a567-kube-api-access-ms4cs\") pod \"network-check-target-ml8n5\" (UID: \"0cbf0042-931c-4135-9330-f9e798b7a567\") " pod="openshift-network-diagnostics/network-check-target-ml8n5" Oct 31 00:13:54.544762 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.543591 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6a21f264-1e80-4eab-8c6e-27ca9ca19f0d-cni-binary-copy\") pod \"multus-additional-cni-plugins-jsctc\" (UID: \"6a21f264-1e80-4eab-8c6e-27ca9ca19f0d\") " pod="openshift-multus/multus-additional-cni-plugins-jsctc" Oct 31 00:13:54.544762 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.543619 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6a21f264-1e80-4eab-8c6e-27ca9ca19f0d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jsctc\" (UID: \"6a21f264-1e80-4eab-8c6e-27ca9ca19f0d\") " pod="openshift-multus/multus-additional-cni-plugins-jsctc" Oct 31 00:13:54.544762 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.543632 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9801ff00-9e9a-4141-8212-ca32bf0251b3-multus-cni-dir\") pod \"multus-b6vpd\" (UID: \"9801ff00-9e9a-4141-8212-ca32bf0251b3\") " pod="openshift-multus/multus-b6vpd" Oct 31 00:13:54.544762 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.543649 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d02ae6b5-4860-45b3-8d16-4a49531b77d4-kubelet-config\") pod \"global-pull-secret-syncer-fwgtx\" (UID: \"d02ae6b5-4860-45b3-8d16-4a49531b77d4\") " pod="kube-system/global-pull-secret-syncer-fwgtx" Oct 31 00:13:54.544762 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.543678 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9801ff00-9e9a-4141-8212-ca32bf0251b3-cnibin\") pod \"multus-b6vpd\" (UID: \"9801ff00-9e9a-4141-8212-ca32bf0251b3\") " pod="openshift-multus/multus-b6vpd" Oct 31 00:13:54.544762 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.543700 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9801ff00-9e9a-4141-8212-ca32bf0251b3-multus-conf-dir\") pod \"multus-b6vpd\" (UID: \"9801ff00-9e9a-4141-8212-ca32bf0251b3\") " pod="openshift-multus/multus-b6vpd" Oct 31 00:13:54.544762 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.543728 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f8c3c03-a1fc-4376-98c6-feeeefa08068-run-openvswitch\") pod \"ovnkube-node-h9hsf\" (UID: \"0f8c3c03-a1fc-4376-98c6-feeeefa08068\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:13:54.544762 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.543753 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6a21f264-1e80-4eab-8c6e-27ca9ca19f0d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jsctc\" (UID: \"6a21f264-1e80-4eab-8c6e-27ca9ca19f0d\") " pod="openshift-multus/multus-additional-cni-plugins-jsctc" Oct 31 00:13:54.544762 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.543758 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9801ff00-9e9a-4141-8212-ca32bf0251b3-host-run-multus-certs\") pod \"multus-b6vpd\" (UID: \"9801ff00-9e9a-4141-8212-ca32bf0251b3\") " pod="openshift-multus/multus-b6vpd" Oct 31 00:13:54.544762 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.543774 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eadb6436-5e5b-4116-8ed7-26aa935fa749-host\") pod \"node-ca-xlzv2\" (UID: \"eadb6436-5e5b-4116-8ed7-26aa935fa749\") " pod="openshift-image-registry/node-ca-xlzv2" Oct 31 00:13:54.544762 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.543798 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9801ff00-9e9a-4141-8212-ca32bf0251b3-multus-socket-dir-parent\") pod \"multus-b6vpd\" (UID: \"9801ff00-9e9a-4141-8212-ca32bf0251b3\") " pod="openshift-multus/multus-b6vpd" Oct 31 00:13:54.544762 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.543823 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9801ff00-9e9a-4141-8212-ca32bf0251b3-cnibin\") pod \"multus-b6vpd\" (UID: \"9801ff00-9e9a-4141-8212-ca32bf0251b3\") " pod="openshift-multus/multus-b6vpd" Oct 31 00:13:54.544762 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.543820 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9801ff00-9e9a-4141-8212-ca32bf0251b3-hostroot\") pod \"multus-b6vpd\" (UID: \"9801ff00-9e9a-4141-8212-ca32bf0251b3\") " pod="openshift-multus/multus-b6vpd" Oct 31 00:13:54.544762 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.543865 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a077dea4-b5a3-4335-bb6a-7618e794eb8f-metrics-certs\") pod \"network-metrics-daemon-rjfn2\" (UID: \"a077dea4-b5a3-4335-bb6a-7618e794eb8f\") " pod="openshift-multus/network-metrics-daemon-rjfn2" Oct 31 00:13:54.545623 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.543887 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0f8c3c03-a1fc-4376-98c6-feeeefa08068-host-slash\") pod \"ovnkube-node-h9hsf\" (UID: \"0f8c3c03-a1fc-4376-98c6-feeeefa08068\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:13:54.545623 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.543911 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6a21f264-1e80-4eab-8c6e-27ca9ca19f0d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jsctc\" (UID: \"6a21f264-1e80-4eab-8c6e-27ca9ca19f0d\") " pod="openshift-multus/multus-additional-cni-plugins-jsctc" Oct 31 00:13:54.545623 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.543917 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/978a9050-dddb-4869-85d8-84a001554c5b-iptables-alerter-script\") pod \"iptables-alerter-4nwn7\" (UID: \"978a9050-dddb-4869-85d8-84a001554c5b\") " pod="openshift-network-operator/iptables-alerter-4nwn7" Oct 31 00:13:54.545623 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.543988 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6a21f264-1e80-4eab-8c6e-27ca9ca19f0d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jsctc\" (UID: \"6a21f264-1e80-4eab-8c6e-27ca9ca19f0d\") " pod="openshift-multus/multus-additional-cni-plugins-jsctc" Oct 31 00:13:54.547664 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.544242 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9801ff00-9e9a-4141-8212-ca32bf0251b3-multus-daemon-config\") pod \"multus-b6vpd\" (UID: \"9801ff00-9e9a-4141-8212-ca32bf0251b3\") " pod="openshift-multus/multus-b6vpd" Oct 31 00:13:54.547664 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.544300 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9801ff00-9e9a-4141-8212-ca32bf0251b3-multus-socket-dir-parent\") pod \"multus-b6vpd\" (UID: \"9801ff00-9e9a-4141-8212-ca32bf0251b3\") " pod="openshift-multus/multus-b6vpd" Oct 31 00:13:54.547836 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.543936 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0eb4888f-40a8-4aba-85d8-3c845382bd02-lib-modules\") pod \"tuned-vsfrc\" (UID: \"0eb4888f-40a8-4aba-85d8-3c845382bd02\") " pod="openshift-cluster-node-tuning-operator/tuned-vsfrc" Oct 31 00:13:54.547836 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.544309 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d02ae6b5-4860-45b3-8d16-4a49531b77d4-kubelet-config\") pod \"global-pull-secret-syncer-fwgtx\" (UID: \"d02ae6b5-4860-45b3-8d16-4a49531b77d4\") " pod="kube-system/global-pull-secret-syncer-fwgtx" Oct 31 00:13:54.547836 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.547748 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0eb4888f-40a8-4aba-85d8-3c845382bd02-etc-tuned\") pod \"tuned-vsfrc\" (UID: \"0eb4888f-40a8-4aba-85d8-3c845382bd02\") " pod="openshift-cluster-node-tuning-operator/tuned-vsfrc" Oct 31 00:13:54.547836 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.544385 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9801ff00-9e9a-4141-8212-ca32bf0251b3-multus-conf-dir\") pod \"multus-b6vpd\" (UID: \"9801ff00-9e9a-4141-8212-ca32bf0251b3\") " pod="openshift-multus/multus-b6vpd" Oct 31 00:13:54.547836 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.544482 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eadb6436-5e5b-4116-8ed7-26aa935fa749-host\") pod \"node-ca-xlzv2\" (UID: \"eadb6436-5e5b-4116-8ed7-26aa935fa749\") " pod="openshift-image-registry/node-ca-xlzv2" Oct 31 00:13:54.547836 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.544556 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9801ff00-9e9a-4141-8212-ca32bf0251b3-hostroot\") pod \"multus-b6vpd\" (UID: \"9801ff00-9e9a-4141-8212-ca32bf0251b3\") " pod="openshift-multus/multus-b6vpd" Oct 31 00:13:54.547836 ip-10-0-137-56 kubenswrapper[2581]: E1031 00:13:54.544650 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:13:54.547836 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.547830 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2ed1e615-cbca-482c-89cc-0fe7d8e7e597-konnectivity-ca\") pod \"konnectivity-agent-lvp2h\" (UID: \"2ed1e615-cbca-482c-89cc-0fe7d8e7e597\") " pod="kube-system/konnectivity-agent-lvp2h" Oct 31 00:13:54.548140 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.547868 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1bc3d212-b67c-4368-90a3-8e080a4f1452-registration-dir\") pod \"aws-ebs-csi-driver-node-ndd7n\" (UID: \"1bc3d212-b67c-4368-90a3-8e080a4f1452\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ndd7n" Oct 31 00:13:54.548140 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.545162 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6a21f264-1e80-4eab-8c6e-27ca9ca19f0d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jsctc\" (UID: \"6a21f264-1e80-4eab-8c6e-27ca9ca19f0d\") " pod="openshift-multus/multus-additional-cni-plugins-jsctc" Oct 31 00:13:54.548140 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.546086 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0f8c3c03-a1fc-4376-98c6-feeeefa08068-ovn-node-metrics-cert\") pod \"ovnkube-node-h9hsf\" (UID: \"0f8c3c03-a1fc-4376-98c6-feeeefa08068\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:13:54.548140 ip-10-0-137-56 kubenswrapper[2581]: E1031 00:13:54.547891 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a077dea4-b5a3-4335-bb6a-7618e794eb8f-metrics-certs podName:a077dea4-b5a3-4335-bb6a-7618e794eb8f nodeName:}" failed. No retries permitted until 2026-10-31 00:13:55.047870713 +0000 UTC m=+2.280406154 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a077dea4-b5a3-4335-bb6a-7618e794eb8f-metrics-certs") pod "network-metrics-daemon-rjfn2" (UID: "a077dea4-b5a3-4335-bb6a-7618e794eb8f") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:13:54.548140 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.544549 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f8c3c03-a1fc-4376-98c6-feeeefa08068-run-openvswitch\") pod \"ovnkube-node-h9hsf\" (UID: \"0f8c3c03-a1fc-4376-98c6-feeeefa08068\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:13:54.548140 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.546822 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6a21f264-1e80-4eab-8c6e-27ca9ca19f0d-cni-binary-copy\") pod \"multus-additional-cni-plugins-jsctc\" (UID: \"6a21f264-1e80-4eab-8c6e-27ca9ca19f0d\") " pod="openshift-multus/multus-additional-cni-plugins-jsctc" Oct 31 00:13:54.548140 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.544520 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0f8c3c03-a1fc-4376-98c6-feeeefa08068-host-slash\") pod \"ovnkube-node-h9hsf\" (UID: \"0f8c3c03-a1fc-4376-98c6-feeeefa08068\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:13:54.548140 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.544807 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0eb4888f-40a8-4aba-85d8-3c845382bd02-lib-modules\") pod \"tuned-vsfrc\" (UID: \"0eb4888f-40a8-4aba-85d8-3c845382bd02\") " pod="openshift-cluster-node-tuning-operator/tuned-vsfrc" Oct 31 00:13:54.548140 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.544927 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6a21f264-1e80-4eab-8c6e-27ca9ca19f0d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jsctc\" (UID: \"6a21f264-1e80-4eab-8c6e-27ca9ca19f0d\") " pod="openshift-multus/multus-additional-cni-plugins-jsctc" Oct 31 00:13:54.548140 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.548069 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0f8c3c03-a1fc-4376-98c6-feeeefa08068-host-cni-netd\") pod \"ovnkube-node-h9hsf\" (UID: \"0f8c3c03-a1fc-4376-98c6-feeeefa08068\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:13:54.548472 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.547992 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0f8c3c03-a1fc-4376-98c6-feeeefa08068-host-cni-netd\") pod \"ovnkube-node-h9hsf\" (UID: \"0f8c3c03-a1fc-4376-98c6-feeeefa08068\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:13:54.548472 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.548429 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1bc3d212-b67c-4368-90a3-8e080a4f1452-registration-dir\") pod \"aws-ebs-csi-driver-node-ndd7n\" (UID: \"1bc3d212-b67c-4368-90a3-8e080a4f1452\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ndd7n" Oct 31 00:13:54.548472 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.548451 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wrw4\" (UniqueName: \"kubernetes.io/projected/0f8c3c03-a1fc-4376-98c6-feeeefa08068-kube-api-access-8wrw4\") pod \"ovnkube-node-h9hsf\" (UID: \"0f8c3c03-a1fc-4376-98c6-feeeefa08068\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:13:54.548602 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.548479 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1bc3d212-b67c-4368-90a3-8e080a4f1452-sys-fs\") pod \"aws-ebs-csi-driver-node-ndd7n\" (UID: \"1bc3d212-b67c-4368-90a3-8e080a4f1452\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ndd7n" Oct 31 00:13:54.548602 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.548501 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9801ff00-9e9a-4141-8212-ca32bf0251b3-etc-kubernetes\") pod \"multus-b6vpd\" (UID: \"9801ff00-9e9a-4141-8212-ca32bf0251b3\") " pod="openshift-multus/multus-b6vpd" Oct 31 00:13:54.548602 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.548532 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0eb4888f-40a8-4aba-85d8-3c845382bd02-etc-sysctl-conf\") pod \"tuned-vsfrc\" (UID: \"0eb4888f-40a8-4aba-85d8-3c845382bd02\") " pod="openshift-cluster-node-tuning-operator/tuned-vsfrc" Oct 31 00:13:54.548602 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.548571 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1bc3d212-b67c-4368-90a3-8e080a4f1452-sys-fs\") pod \"aws-ebs-csi-driver-node-ndd7n\" (UID: \"1bc3d212-b67c-4368-90a3-8e080a4f1452\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ndd7n" Oct 31 00:13:54.548602 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.548578 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/eadb6436-5e5b-4116-8ed7-26aa935fa749-serviceca\") pod \"node-ca-xlzv2\" (UID: \"eadb6436-5e5b-4116-8ed7-26aa935fa749\") " pod="openshift-image-registry/node-ca-xlzv2" Oct 31 00:13:54.548782 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.548642 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0eb4888f-40a8-4aba-85d8-3c845382bd02-etc-sysctl-conf\") pod \"tuned-vsfrc\" (UID: \"0eb4888f-40a8-4aba-85d8-3c845382bd02\") " pod="openshift-cluster-node-tuning-operator/tuned-vsfrc" Oct 31 00:13:54.548782 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.548614 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0f8c3c03-a1fc-4376-98c6-feeeefa08068-systemd-units\") pod \"ovnkube-node-h9hsf\" (UID: \"0f8c3c03-a1fc-4376-98c6-feeeefa08068\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:13:54.548782 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.548679 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9801ff00-9e9a-4141-8212-ca32bf0251b3-etc-kubernetes\") pod \"multus-b6vpd\" (UID: \"9801ff00-9e9a-4141-8212-ca32bf0251b3\") " pod="openshift-multus/multus-b6vpd" Oct 31 00:13:54.548782 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.548679 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0f8c3c03-a1fc-4376-98c6-feeeefa08068-log-socket\") pod \"ovnkube-node-h9hsf\" (UID: \"0f8c3c03-a1fc-4376-98c6-feeeefa08068\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:13:54.548782 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.548708 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0f8c3c03-a1fc-4376-98c6-feeeefa08068-log-socket\") pod \"ovnkube-node-h9hsf\" (UID: \"0f8c3c03-a1fc-4376-98c6-feeeefa08068\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:13:54.548782 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.548709 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0f8c3c03-a1fc-4376-98c6-feeeefa08068-host-cni-bin\") pod \"ovnkube-node-h9hsf\" (UID: \"0f8c3c03-a1fc-4376-98c6-feeeefa08068\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:13:54.548782 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.548735 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0f8c3c03-a1fc-4376-98c6-feeeefa08068-host-cni-bin\") pod \"ovnkube-node-h9hsf\" (UID: \"0f8c3c03-a1fc-4376-98c6-feeeefa08068\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:13:54.548782 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.548737 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1bc3d212-b67c-4368-90a3-8e080a4f1452-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ndd7n\" (UID: \"1bc3d212-b67c-4368-90a3-8e080a4f1452\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ndd7n" Oct 31 00:13:54.548782 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.548756 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0f8c3c03-a1fc-4376-98c6-feeeefa08068-systemd-units\") pod \"ovnkube-node-h9hsf\" (UID: \"0f8c3c03-a1fc-4376-98c6-feeeefa08068\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:13:54.548782 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.548763 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1bc3d212-b67c-4368-90a3-8e080a4f1452-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ndd7n\" (UID: \"1bc3d212-b67c-4368-90a3-8e080a4f1452\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ndd7n" Oct 31 00:13:54.549044 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.548761 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0f8c3c03-a1fc-4376-98c6-feeeefa08068-host-run-ovn-kubernetes\") pod \"ovnkube-node-h9hsf\" (UID: \"0f8c3c03-a1fc-4376-98c6-feeeefa08068\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:13:54.549044 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.548806 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0eb4888f-40a8-4aba-85d8-3c845382bd02-etc-sysctl-d\") pod \"tuned-vsfrc\" (UID: \"0eb4888f-40a8-4aba-85d8-3c845382bd02\") " pod="openshift-cluster-node-tuning-operator/tuned-vsfrc" Oct 31 00:13:54.549044 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.548809 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0f8c3c03-a1fc-4376-98c6-feeeefa08068-host-run-ovn-kubernetes\") pod \"ovnkube-node-h9hsf\" (UID: \"0f8c3c03-a1fc-4376-98c6-feeeefa08068\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:13:54.549044 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.548823 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0eb4888f-40a8-4aba-85d8-3c845382bd02-tmp\") pod \"tuned-vsfrc\" (UID: \"0eb4888f-40a8-4aba-85d8-3c845382bd02\") " pod="openshift-cluster-node-tuning-operator/tuned-vsfrc" Oct 31 00:13:54.549044 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.548839 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrkcn\" (UniqueName: \"kubernetes.io/projected/0eb4888f-40a8-4aba-85d8-3c845382bd02-kube-api-access-nrkcn\") pod \"tuned-vsfrc\" (UID: \"0eb4888f-40a8-4aba-85d8-3c845382bd02\") " pod="openshift-cluster-node-tuning-operator/tuned-vsfrc" Oct 31 00:13:54.549044 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.548854 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2ed1e615-cbca-482c-89cc-0fe7d8e7e597-agent-certs\") pod \"konnectivity-agent-lvp2h\" (UID: \"2ed1e615-cbca-482c-89cc-0fe7d8e7e597\") " pod="kube-system/konnectivity-agent-lvp2h" Oct 31 00:13:54.549044 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.548868 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1bc3d212-b67c-4368-90a3-8e080a4f1452-etc-selinux\") pod \"aws-ebs-csi-driver-node-ndd7n\" (UID: \"1bc3d212-b67c-4368-90a3-8e080a4f1452\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ndd7n" Oct 31 00:13:54.549044 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.548881 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a7d0a920-91fe-4c4a-b9bc-67550c324e7f-hosts-file\") pod \"node-resolver-gkm2c\" (UID: \"a7d0a920-91fe-4c4a-b9bc-67550c324e7f\") " pod="openshift-dns/node-resolver-gkm2c" Oct 31 00:13:54.549044 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.548895 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0f8c3c03-a1fc-4376-98c6-feeeefa08068-run-ovn\") pod \"ovnkube-node-h9hsf\" (UID: \"0f8c3c03-a1fc-4376-98c6-feeeefa08068\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:13:54.549044 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.548897 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0eb4888f-40a8-4aba-85d8-3c845382bd02-etc-sysctl-d\") pod \"tuned-vsfrc\" (UID: \"0eb4888f-40a8-4aba-85d8-3c845382bd02\") " pod="openshift-cluster-node-tuning-operator/tuned-vsfrc" Oct 31 00:13:54.549044 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.548908 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0eb4888f-40a8-4aba-85d8-3c845382bd02-etc-kubernetes\") pod \"tuned-vsfrc\" (UID: \"0eb4888f-40a8-4aba-85d8-3c845382bd02\") " pod="openshift-cluster-node-tuning-operator/tuned-vsfrc" Oct 31 00:13:54.549044 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.548958 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0eb4888f-40a8-4aba-85d8-3c845382bd02-etc-kubernetes\") pod \"tuned-vsfrc\" (UID: \"0eb4888f-40a8-4aba-85d8-3c845382bd02\") " pod="openshift-cluster-node-tuning-operator/tuned-vsfrc" Oct 31 00:13:54.549044 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.548975 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1bc3d212-b67c-4368-90a3-8e080a4f1452-etc-selinux\") pod \"aws-ebs-csi-driver-node-ndd7n\" (UID: \"1bc3d212-b67c-4368-90a3-8e080a4f1452\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ndd7n" Oct 31 00:13:54.549044 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.548999 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0f8c3c03-a1fc-4376-98c6-feeeefa08068-run-ovn\") pod \"ovnkube-node-h9hsf\" (UID: \"0f8c3c03-a1fc-4376-98c6-feeeefa08068\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:13:54.549044 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.549028 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a7d0a920-91fe-4c4a-b9bc-67550c324e7f-hosts-file\") pod \"node-resolver-gkm2c\" (UID: \"a7d0a920-91fe-4c4a-b9bc-67550c324e7f\") " pod="openshift-dns/node-resolver-gkm2c" Oct 31 00:13:54.549773 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.549754 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2ed1e615-cbca-482c-89cc-0fe7d8e7e597-konnectivity-ca\") pod \"konnectivity-agent-lvp2h\" (UID: \"2ed1e615-cbca-482c-89cc-0fe7d8e7e597\") " pod="kube-system/konnectivity-agent-lvp2h" Oct 31 00:13:54.550012 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.549973 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/eadb6436-5e5b-4116-8ed7-26aa935fa749-serviceca\") pod \"node-ca-xlzv2\" (UID: \"eadb6436-5e5b-4116-8ed7-26aa935fa749\") " pod="openshift-image-registry/node-ca-xlzv2" Oct 31 00:13:54.551718 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.551702 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0eb4888f-40a8-4aba-85d8-3c845382bd02-tmp\") pod \"tuned-vsfrc\" (UID: \"0eb4888f-40a8-4aba-85d8-3c845382bd02\") " pod="openshift-cluster-node-tuning-operator/tuned-vsfrc" Oct 31 00:13:54.551850 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.551833 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2ed1e615-cbca-482c-89cc-0fe7d8e7e597-agent-certs\") pod \"konnectivity-agent-lvp2h\" (UID: \"2ed1e615-cbca-482c-89cc-0fe7d8e7e597\") " pod="kube-system/konnectivity-agent-lvp2h" Oct 31 00:13:54.601219 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.601150 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xc9bh\" (UniqueName: \"kubernetes.io/projected/a077dea4-b5a3-4335-bb6a-7618e794eb8f-kube-api-access-xc9bh\") pod \"network-metrics-daemon-rjfn2\" (UID: \"a077dea4-b5a3-4335-bb6a-7618e794eb8f\") " pod="openshift-multus/network-metrics-daemon-rjfn2" Oct 31 00:13:54.601335 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.601223 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djjgz\" (UniqueName: \"kubernetes.io/projected/a7d0a920-91fe-4c4a-b9bc-67550c324e7f-kube-api-access-djjgz\") pod \"node-resolver-gkm2c\" (UID: \"a7d0a920-91fe-4c4a-b9bc-67550c324e7f\") " pod="openshift-dns/node-resolver-gkm2c" Oct 31 00:13:54.611107 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.611064 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wm7wn\" (UniqueName: \"kubernetes.io/projected/1bc3d212-b67c-4368-90a3-8e080a4f1452-kube-api-access-wm7wn\") pod \"aws-ebs-csi-driver-node-ndd7n\" (UID: \"1bc3d212-b67c-4368-90a3-8e080a4f1452\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ndd7n" Oct 31 00:13:54.611618 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.611596 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8j6tz\" (UniqueName: \"kubernetes.io/projected/6a21f264-1e80-4eab-8c6e-27ca9ca19f0d-kube-api-access-8j6tz\") pod \"multus-additional-cni-plugins-jsctc\" (UID: \"6a21f264-1e80-4eab-8c6e-27ca9ca19f0d\") " pod="openshift-multus/multus-additional-cni-plugins-jsctc" Oct 31 00:13:54.611785 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.611762 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbpb7\" (UniqueName: \"kubernetes.io/projected/978a9050-dddb-4869-85d8-84a001554c5b-kube-api-access-gbpb7\") pod \"iptables-alerter-4nwn7\" (UID: \"978a9050-dddb-4869-85d8-84a001554c5b\") " pod="openshift-network-operator/iptables-alerter-4nwn7" Oct 31 00:13:54.611891 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.611876 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nz8r2\" (UniqueName: \"kubernetes.io/projected/eadb6436-5e5b-4116-8ed7-26aa935fa749-kube-api-access-nz8r2\") pod \"node-ca-xlzv2\" (UID: \"eadb6436-5e5b-4116-8ed7-26aa935fa749\") " pod="openshift-image-registry/node-ca-xlzv2" Oct 31 00:13:54.612279 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.612264 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzs4x\" (UniqueName: \"kubernetes.io/projected/9801ff00-9e9a-4141-8212-ca32bf0251b3-kube-api-access-xzs4x\") pod \"multus-b6vpd\" (UID: \"9801ff00-9e9a-4141-8212-ca32bf0251b3\") " pod="openshift-multus/multus-b6vpd" Oct 31 00:13:54.619815 ip-10-0-137-56 kubenswrapper[2581]: E1031 00:13:54.619798 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 00:13:54.619815 ip-10-0-137-56 kubenswrapper[2581]: E1031 00:13:54.619821 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 00:13:54.619937 ip-10-0-137-56 kubenswrapper[2581]: E1031 00:13:54.619832 2581 projected.go:196] Error preparing data for projected volume kube-api-access-ms4cs for pod openshift-network-diagnostics/network-check-target-ml8n5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:13:54.619937 ip-10-0-137-56 kubenswrapper[2581]: E1031 00:13:54.619904 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0cbf0042-931c-4135-9330-f9e798b7a567-kube-api-access-ms4cs podName:0cbf0042-931c-4135-9330-f9e798b7a567 nodeName:}" failed. No retries permitted until 2026-10-31 00:13:55.119889391 +0000 UTC m=+2.352424838 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-ms4cs" (UniqueName: "kubernetes.io/projected/0cbf0042-931c-4135-9330-f9e798b7a567-kube-api-access-ms4cs") pod "network-check-target-ml8n5" (UID: "0cbf0042-931c-4135-9330-f9e798b7a567") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:13:54.623814 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.623787 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wrw4\" (UniqueName: \"kubernetes.io/projected/0f8c3c03-a1fc-4376-98c6-feeeefa08068-kube-api-access-8wrw4\") pod \"ovnkube-node-h9hsf\" (UID: \"0f8c3c03-a1fc-4376-98c6-feeeefa08068\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:13:54.623924 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.623910 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrkcn\" (UniqueName: \"kubernetes.io/projected/0eb4888f-40a8-4aba-85d8-3c845382bd02-kube-api-access-nrkcn\") pod \"tuned-vsfrc\" (UID: \"0eb4888f-40a8-4aba-85d8-3c845382bd02\") " pod="openshift-cluster-node-tuning-operator/tuned-vsfrc" Oct 31 00:13:54.703036 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.703005 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:13:54.720691 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.720663 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-lvp2h" Oct 31 00:13:54.737691 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.737639 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ndd7n" Oct 31 00:13:54.746387 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.746352 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jsctc" Oct 31 00:13:54.767289 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.767258 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-b6vpd" Oct 31 00:13:54.790410 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.790380 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vsfrc" Oct 31 00:13:54.796453 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.796433 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4nwn7" Oct 31 00:13:54.808617 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.808587 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gkm2c" Oct 31 00:13:54.813311 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.813263 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-xlzv2" Oct 31 00:13:54.926141 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:54.926107 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode15cd53a0cf50b999ecad893fad25259.slice/crio-7eae82e9ea1bf2459313e7139b9f222797c511f08b4b8774861dcd838dd46ad9 WatchSource:0}: Error finding container 7eae82e9ea1bf2459313e7139b9f222797c511f08b4b8774861dcd838dd46ad9: Status 404 returned error can't find the container with id 7eae82e9ea1bf2459313e7139b9f222797c511f08b4b8774861dcd838dd46ad9 Oct 31 00:13:54.931343 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:54.931288 2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 00:13:54.970808 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:54.970773 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0db3b99b7954e19a6b4da51857943e0.slice/crio-2d6531cabcd80da4099819572fe53a24f4806363c480abd15f54c08134300f74 WatchSource:0}: Error finding container 2d6531cabcd80da4099819572fe53a24f4806363c480abd15f54c08134300f74: Status 404 returned error can't find the container with id 2d6531cabcd80da4099819572fe53a24f4806363c480abd15f54c08134300f74 Oct 31 00:13:55.053286 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:55.053247 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a077dea4-b5a3-4335-bb6a-7618e794eb8f-metrics-certs\") pod \"network-metrics-daemon-rjfn2\" (UID: \"a077dea4-b5a3-4335-bb6a-7618e794eb8f\") " pod="openshift-multus/network-metrics-daemon-rjfn2" Oct 31 00:13:55.053497 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:55.053342 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d02ae6b5-4860-45b3-8d16-4a49531b77d4-original-pull-secret\") pod \"global-pull-secret-syncer-fwgtx\" (UID: \"d02ae6b5-4860-45b3-8d16-4a49531b77d4\") " pod="kube-system/global-pull-secret-syncer-fwgtx" Oct 31 00:13:55.053497 ip-10-0-137-56 kubenswrapper[2581]: E1031 00:13:55.053256 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:13:55.053497 ip-10-0-137-56 kubenswrapper[2581]: E1031 00:13:55.053435 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a077dea4-b5a3-4335-bb6a-7618e794eb8f-metrics-certs podName:a077dea4-b5a3-4335-bb6a-7618e794eb8f nodeName:}" failed. No retries permitted until 2026-10-31 00:13:56.053414406 +0000 UTC m=+3.285949832 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a077dea4-b5a3-4335-bb6a-7618e794eb8f-metrics-certs") pod "network-metrics-daemon-rjfn2" (UID: "a077dea4-b5a3-4335-bb6a-7618e794eb8f") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:13:55.053497 ip-10-0-137-56 kubenswrapper[2581]: E1031 00:13:55.053458 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 00:13:55.053497 ip-10-0-137-56 kubenswrapper[2581]: E1031 00:13:55.053499 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d02ae6b5-4860-45b3-8d16-4a49531b77d4-original-pull-secret podName:d02ae6b5-4860-45b3-8d16-4a49531b77d4 nodeName:}" failed. No retries permitted until 2026-10-31 00:13:56.053488025 +0000 UTC m=+3.286023451 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d02ae6b5-4860-45b3-8d16-4a49531b77d4-original-pull-secret") pod "global-pull-secret-syncer-fwgtx" (UID: "d02ae6b5-4860-45b3-8d16-4a49531b77d4") : object "kube-system"/"original-pull-secret" not registered Oct 31 00:13:55.154072 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:55.153970 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ms4cs\" (UniqueName: \"kubernetes.io/projected/0cbf0042-931c-4135-9330-f9e798b7a567-kube-api-access-ms4cs\") pod \"network-check-target-ml8n5\" (UID: \"0cbf0042-931c-4135-9330-f9e798b7a567\") " pod="openshift-network-diagnostics/network-check-target-ml8n5" Oct 31 00:13:55.154719 ip-10-0-137-56 kubenswrapper[2581]: E1031 00:13:55.154134 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 00:13:55.154719 ip-10-0-137-56 kubenswrapper[2581]: E1031 00:13:55.154158 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 00:13:55.154719 ip-10-0-137-56 kubenswrapper[2581]: E1031 00:13:55.154168 2581 projected.go:196] Error preparing data for projected volume kube-api-access-ms4cs for pod openshift-network-diagnostics/network-check-target-ml8n5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:13:55.154719 ip-10-0-137-56 kubenswrapper[2581]: E1031 00:13:55.154239 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0cbf0042-931c-4135-9330-f9e798b7a567-kube-api-access-ms4cs podName:0cbf0042-931c-4135-9330-f9e798b7a567 nodeName:}" failed. No retries permitted until 2026-10-31 00:13:56.15421237 +0000 UTC m=+3.386747811 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-ms4cs" (UniqueName: "kubernetes.io/projected/0cbf0042-931c-4135-9330-f9e798b7a567-kube-api-access-ms4cs") pod "network-check-target-ml8n5" (UID: "0cbf0042-931c-4135-9330-f9e798b7a567") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:13:55.255559 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:55.255530 2581 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:13:55.278267 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:55.278218 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-56.us-west-2.compute.internal" event={"ID":"f0db3b99b7954e19a6b4da51857943e0","Type":"ContainerStarted","Data":"2d6531cabcd80da4099819572fe53a24f4806363c480abd15f54c08134300f74"} Oct 31 00:13:55.279353 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:55.279332 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-56.us-west-2.compute.internal" event={"ID":"e15cd53a0cf50b999ecad893fad25259","Type":"ContainerStarted","Data":"7eae82e9ea1bf2459313e7139b9f222797c511f08b4b8774861dcd838dd46ad9"} Oct 31 00:13:55.427680 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:55.427590 2581 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:13:55.463787 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:55.463742 2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 00:08:54 +0000 UTC" deadline="2028-08-03 18:00:59.249930608 +0000 UTC" Oct 31 00:13:55.463787 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:55.463779 2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15425h47m3.786155024s" Oct 31 00:13:55.624259 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:55.624225 2581 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:13:55.701878 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:55.701846 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeadb6436_5e5b_4116_8ed7_26aa935fa749.slice/crio-2f67477bcc4a7d7c671cdd426b7bd4aca4916983579ffc57621163ce85f1e77a WatchSource:0}: Error finding container 2f67477bcc4a7d7c671cdd426b7bd4aca4916983579ffc57621163ce85f1e77a: Status 404 returned error can't find the container with id 2f67477bcc4a7d7c671cdd426b7bd4aca4916983579ffc57621163ce85f1e77a Oct 31 00:13:55.725246 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:55.724595 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bc3d212_b67c_4368_90a3_8e080a4f1452.slice/crio-da2e414968db90f728e2dbdf6b02f29a507bfa7e488f8490650761508fefde8c WatchSource:0}: Error finding container da2e414968db90f728e2dbdf6b02f29a507bfa7e488f8490650761508fefde8c: Status 404 returned error can't find the container with id da2e414968db90f728e2dbdf6b02f29a507bfa7e488f8490650761508fefde8c Oct 31 00:13:55.728007 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:55.727975 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f8c3c03_a1fc_4376_98c6_feeeefa08068.slice/crio-2d7517a7af7b2bf180b122c5a436e86320f74f7a9b99a3c3a6e8af9bb71f5675 WatchSource:0}: Error finding container 2d7517a7af7b2bf180b122c5a436e86320f74f7a9b99a3c3a6e8af9bb71f5675: Status 404 returned error can't find the container with id 2d7517a7af7b2bf180b122c5a436e86320f74f7a9b99a3c3a6e8af9bb71f5675 Oct 31 00:13:55.728689 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:55.728661 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7d0a920_91fe_4c4a_b9bc_67550c324e7f.slice/crio-d84687d8838569571481ff1562bc7bac9a6445916f8efa90c7cc3a4d986ec18c WatchSource:0}: Error finding container d84687d8838569571481ff1562bc7bac9a6445916f8efa90c7cc3a4d986ec18c: Status 404 returned error can't find the container with id d84687d8838569571481ff1562bc7bac9a6445916f8efa90c7cc3a4d986ec18c Oct 31 00:13:55.767492 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:55.767460 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0eb4888f_40a8_4aba_85d8_3c845382bd02.slice/crio-f80d3dce6bdb9c8f5ec0436eac752dc4bb1222b8d384cc9cfee9046e2538a2ca WatchSource:0}: Error finding container f80d3dce6bdb9c8f5ec0436eac752dc4bb1222b8d384cc9cfee9046e2538a2ca: Status 404 returned error can't find the container with id f80d3dce6bdb9c8f5ec0436eac752dc4bb1222b8d384cc9cfee9046e2538a2ca Oct 31 00:13:55.768664 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:55.768637 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9801ff00_9e9a_4141_8212_ca32bf0251b3.slice/crio-e5c3e87bed66ffaf5ebc4ef908b7b5485a99b714652eb8b0b7f3ce5237e7acab WatchSource:0}: Error finding container e5c3e87bed66ffaf5ebc4ef908b7b5485a99b714652eb8b0b7f3ce5237e7acab: Status 404 returned error can't find the container with id e5c3e87bed66ffaf5ebc4ef908b7b5485a99b714652eb8b0b7f3ce5237e7acab Oct 31 00:13:55.797491 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:55.797454 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ed1e615_cbca_482c_89cc_0fe7d8e7e597.slice/crio-6810965ba74ad28fc39e610471d28dd74b2bd6d74c30ae3c36b47a9a6bd80630 WatchSource:0}: Error finding container 6810965ba74ad28fc39e610471d28dd74b2bd6d74c30ae3c36b47a9a6bd80630: Status 404 returned error can't find the container with id 6810965ba74ad28fc39e610471d28dd74b2bd6d74c30ae3c36b47a9a6bd80630 Oct 31 00:13:55.839167 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:55.839131 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a21f264_1e80_4eab_8c6e_27ca9ca19f0d.slice/crio-a41c4f5553eae427686ab6b45e27123b288ecab81d8c898fd14d6f2f24ab25d9 WatchSource:0}: Error finding container a41c4f5553eae427686ab6b45e27123b288ecab81d8c898fd14d6f2f24ab25d9: Status 404 returned error can't find the container with id a41c4f5553eae427686ab6b45e27123b288ecab81d8c898fd14d6f2f24ab25d9 Oct 31 00:13:55.839934 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:13:55.839906 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod978a9050_dddb_4869_85d8_84a001554c5b.slice/crio-6450b4ee33119ffd3082ede4ecbe8007a9275feabbf96dee54b65a703684250b WatchSource:0}: Error finding container 6450b4ee33119ffd3082ede4ecbe8007a9275feabbf96dee54b65a703684250b: Status 404 returned error can't find the container with id 6450b4ee33119ffd3082ede4ecbe8007a9275feabbf96dee54b65a703684250b Oct 31 00:13:56.059923 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:56.059837 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a077dea4-b5a3-4335-bb6a-7618e794eb8f-metrics-certs\") pod \"network-metrics-daemon-rjfn2\" (UID: \"a077dea4-b5a3-4335-bb6a-7618e794eb8f\") " pod="openshift-multus/network-metrics-daemon-rjfn2" Oct 31 00:13:56.059923 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:56.059887 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d02ae6b5-4860-45b3-8d16-4a49531b77d4-original-pull-secret\") pod \"global-pull-secret-syncer-fwgtx\" (UID: \"d02ae6b5-4860-45b3-8d16-4a49531b77d4\") " pod="kube-system/global-pull-secret-syncer-fwgtx" Oct 31 00:13:56.060174 ip-10-0-137-56 kubenswrapper[2581]: E1031 00:13:56.060003 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:13:56.060174 ip-10-0-137-56 kubenswrapper[2581]: E1031 00:13:56.060069 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a077dea4-b5a3-4335-bb6a-7618e794eb8f-metrics-certs podName:a077dea4-b5a3-4335-bb6a-7618e794eb8f nodeName:}" failed. No retries permitted until 2026-10-31 00:13:58.060053438 +0000 UTC m=+5.292588882 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a077dea4-b5a3-4335-bb6a-7618e794eb8f-metrics-certs") pod "network-metrics-daemon-rjfn2" (UID: "a077dea4-b5a3-4335-bb6a-7618e794eb8f") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:13:56.060174 ip-10-0-137-56 kubenswrapper[2581]: E1031 00:13:56.060003 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 00:13:56.060174 ip-10-0-137-56 kubenswrapper[2581]: E1031 00:13:56.060124 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d02ae6b5-4860-45b3-8d16-4a49531b77d4-original-pull-secret podName:d02ae6b5-4860-45b3-8d16-4a49531b77d4 nodeName:}" failed. No retries permitted until 2026-10-31 00:13:58.060112627 +0000 UTC m=+5.292648069 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d02ae6b5-4860-45b3-8d16-4a49531b77d4-original-pull-secret") pod "global-pull-secret-syncer-fwgtx" (UID: "d02ae6b5-4860-45b3-8d16-4a49531b77d4") : object "kube-system"/"original-pull-secret" not registered Oct 31 00:13:56.161117 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:56.161072 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ms4cs\" (UniqueName: \"kubernetes.io/projected/0cbf0042-931c-4135-9330-f9e798b7a567-kube-api-access-ms4cs\") pod \"network-check-target-ml8n5\" (UID: \"0cbf0042-931c-4135-9330-f9e798b7a567\") " pod="openshift-network-diagnostics/network-check-target-ml8n5" Oct 31 00:13:56.161833 ip-10-0-137-56 kubenswrapper[2581]: E1031 00:13:56.161256 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 00:13:56.161833 ip-10-0-137-56 kubenswrapper[2581]: E1031 00:13:56.161280 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 00:13:56.161833 ip-10-0-137-56 kubenswrapper[2581]: E1031 00:13:56.161290 2581 projected.go:196] Error preparing data for projected volume kube-api-access-ms4cs for pod openshift-network-diagnostics/network-check-target-ml8n5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:13:56.161833 ip-10-0-137-56 kubenswrapper[2581]: E1031 00:13:56.161344 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0cbf0042-931c-4135-9330-f9e798b7a567-kube-api-access-ms4cs podName:0cbf0042-931c-4135-9330-f9e798b7a567 nodeName:}" failed. No retries permitted until 2026-10-31 00:13:58.161329407 +0000 UTC m=+5.393864848 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-ms4cs" (UniqueName: "kubernetes.io/projected/0cbf0042-931c-4135-9330-f9e798b7a567-kube-api-access-ms4cs") pod "network-check-target-ml8n5" (UID: "0cbf0042-931c-4135-9330-f9e798b7a567") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:13:56.273394 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:56.273356 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rjfn2" Oct 31 00:13:56.273610 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:56.273404 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fwgtx" Oct 31 00:13:56.273610 ip-10-0-137-56 kubenswrapper[2581]: E1031 00:13:56.273574 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rjfn2" podUID="a077dea4-b5a3-4335-bb6a-7618e794eb8f" Oct 31 00:13:56.273749 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:56.273609 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ml8n5" Oct 31 00:13:56.273749 ip-10-0-137-56 kubenswrapper[2581]: E1031 00:13:56.273715 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fwgtx" podUID="d02ae6b5-4860-45b3-8d16-4a49531b77d4" Oct 31 00:13:56.273855 ip-10-0-137-56 kubenswrapper[2581]: E1031 00:13:56.273798 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ml8n5" podUID="0cbf0042-931c-4135-9330-f9e798b7a567" Oct 31 00:13:56.283808 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:56.283758 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4nwn7" event={"ID":"978a9050-dddb-4869-85d8-84a001554c5b","Type":"ContainerStarted","Data":"6450b4ee33119ffd3082ede4ecbe8007a9275feabbf96dee54b65a703684250b"} Oct 31 00:13:56.285452 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:56.285385 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jsctc" event={"ID":"6a21f264-1e80-4eab-8c6e-27ca9ca19f0d","Type":"ContainerStarted","Data":"a41c4f5553eae427686ab6b45e27123b288ecab81d8c898fd14d6f2f24ab25d9"} Oct 31 00:13:56.287049 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:56.287014 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-lvp2h" event={"ID":"2ed1e615-cbca-482c-89cc-0fe7d8e7e597","Type":"ContainerStarted","Data":"6810965ba74ad28fc39e610471d28dd74b2bd6d74c30ae3c36b47a9a6bd80630"} Oct 31 00:13:56.288503 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:56.288474 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vsfrc" event={"ID":"0eb4888f-40a8-4aba-85d8-3c845382bd02","Type":"ContainerStarted","Data":"f80d3dce6bdb9c8f5ec0436eac752dc4bb1222b8d384cc9cfee9046e2538a2ca"} Oct 31 00:13:56.290194 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:56.290164 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-b6vpd" event={"ID":"9801ff00-9e9a-4141-8212-ca32bf0251b3","Type":"ContainerStarted","Data":"e5c3e87bed66ffaf5ebc4ef908b7b5485a99b714652eb8b0b7f3ce5237e7acab"} Oct 31 00:13:56.292429 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:56.292379 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ndd7n" event={"ID":"1bc3d212-b67c-4368-90a3-8e080a4f1452","Type":"ContainerStarted","Data":"da2e414968db90f728e2dbdf6b02f29a507bfa7e488f8490650761508fefde8c"} Oct 31 00:13:56.294873 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:56.294827 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-xlzv2" event={"ID":"eadb6436-5e5b-4116-8ed7-26aa935fa749","Type":"ContainerStarted","Data":"2f67477bcc4a7d7c671cdd426b7bd4aca4916983579ffc57621163ce85f1e77a"} Oct 31 00:13:56.296938 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:56.296888 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gkm2c" event={"ID":"a7d0a920-91fe-4c4a-b9bc-67550c324e7f","Type":"ContainerStarted","Data":"d84687d8838569571481ff1562bc7bac9a6445916f8efa90c7cc3a4d986ec18c"} Oct 31 00:13:56.298897 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:56.298870 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" event={"ID":"0f8c3c03-a1fc-4376-98c6-feeeefa08068","Type":"ContainerStarted","Data":"2d7517a7af7b2bf180b122c5a436e86320f74f7a9b99a3c3a6e8af9bb71f5675"} Oct 31 00:13:56.464152 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:56.464092 2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 00:08:54 +0000 UTC" deadline="2028-04-26 12:39:39.823330973 +0000 UTC" Oct 31 00:13:56.464152 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:56.464151 2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13044h25m43.359184663s" Oct 31 00:13:58.077929 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:58.077894 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a077dea4-b5a3-4335-bb6a-7618e794eb8f-metrics-certs\") pod \"network-metrics-daemon-rjfn2\" (UID: \"a077dea4-b5a3-4335-bb6a-7618e794eb8f\") " pod="openshift-multus/network-metrics-daemon-rjfn2" Oct 31 00:13:58.078335 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:58.077969 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d02ae6b5-4860-45b3-8d16-4a49531b77d4-original-pull-secret\") pod \"global-pull-secret-syncer-fwgtx\" (UID: \"d02ae6b5-4860-45b3-8d16-4a49531b77d4\") " pod="kube-system/global-pull-secret-syncer-fwgtx" Oct 31 00:13:58.078335 ip-10-0-137-56 kubenswrapper[2581]: E1031 00:13:58.078155 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 00:13:58.078335 ip-10-0-137-56 kubenswrapper[2581]: E1031 00:13:58.078223 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d02ae6b5-4860-45b3-8d16-4a49531b77d4-original-pull-secret podName:d02ae6b5-4860-45b3-8d16-4a49531b77d4 nodeName:}" failed. No retries permitted until 2026-10-31 00:14:02.078202135 +0000 UTC m=+9.310737618 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d02ae6b5-4860-45b3-8d16-4a49531b77d4-original-pull-secret") pod "global-pull-secret-syncer-fwgtx" (UID: "d02ae6b5-4860-45b3-8d16-4a49531b77d4") : object "kube-system"/"original-pull-secret" not registered Oct 31 00:13:58.078335 ip-10-0-137-56 kubenswrapper[2581]: E1031 00:13:58.078292 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:13:58.078335 ip-10-0-137-56 kubenswrapper[2581]: E1031 00:13:58.078326 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a077dea4-b5a3-4335-bb6a-7618e794eb8f-metrics-certs podName:a077dea4-b5a3-4335-bb6a-7618e794eb8f nodeName:}" failed. No retries permitted until 2026-10-31 00:14:02.078313858 +0000 UTC m=+9.310849301 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a077dea4-b5a3-4335-bb6a-7618e794eb8f-metrics-certs") pod "network-metrics-daemon-rjfn2" (UID: "a077dea4-b5a3-4335-bb6a-7618e794eb8f") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:13:58.179334 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:58.179264 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ms4cs\" (UniqueName: \"kubernetes.io/projected/0cbf0042-931c-4135-9330-f9e798b7a567-kube-api-access-ms4cs\") pod \"network-check-target-ml8n5\" (UID: \"0cbf0042-931c-4135-9330-f9e798b7a567\") " pod="openshift-network-diagnostics/network-check-target-ml8n5" Oct 31 00:13:58.179543 ip-10-0-137-56 kubenswrapper[2581]: E1031 00:13:58.179500 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 00:13:58.179543 ip-10-0-137-56 kubenswrapper[2581]: E1031 00:13:58.179519 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 00:13:58.179543 ip-10-0-137-56 kubenswrapper[2581]: E1031 00:13:58.179533 2581 projected.go:196] Error preparing data for projected volume kube-api-access-ms4cs for pod openshift-network-diagnostics/network-check-target-ml8n5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:13:58.179721 ip-10-0-137-56 kubenswrapper[2581]: E1031 00:13:58.179601 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0cbf0042-931c-4135-9330-f9e798b7a567-kube-api-access-ms4cs podName:0cbf0042-931c-4135-9330-f9e798b7a567 nodeName:}" failed. No retries permitted until 2026-10-31 00:14:02.179580718 +0000 UTC m=+9.412116161 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-ms4cs" (UniqueName: "kubernetes.io/projected/0cbf0042-931c-4135-9330-f9e798b7a567-kube-api-access-ms4cs") pod "network-check-target-ml8n5" (UID: "0cbf0042-931c-4135-9330-f9e798b7a567") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:13:58.274135 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:58.274070 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fwgtx" Oct 31 00:13:58.274330 ip-10-0-137-56 kubenswrapper[2581]: E1031 00:13:58.274263 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fwgtx" podUID="d02ae6b5-4860-45b3-8d16-4a49531b77d4" Oct 31 00:13:58.274409 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:58.274365 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ml8n5" Oct 31 00:13:58.274491 ip-10-0-137-56 kubenswrapper[2581]: E1031 00:13:58.274477 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ml8n5" podUID="0cbf0042-931c-4135-9330-f9e798b7a567" Oct 31 00:13:58.274553 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:58.274544 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rjfn2" Oct 31 00:13:58.274657 ip-10-0-137-56 kubenswrapper[2581]: E1031 00:13:58.274635 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rjfn2" podUID="a077dea4-b5a3-4335-bb6a-7618e794eb8f" Oct 31 00:13:59.338193 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:59.337073 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-56.us-west-2.compute.internal" event={"ID":"e15cd53a0cf50b999ecad893fad25259","Type":"ContainerStarted","Data":"f3b77481bf449c6e6b4bdb7a5cea467c340f76f475aa912f461f3c205b5789bb"} Oct 31 00:13:59.347060 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:59.347016 2581 generic.go:356] "Generic (PLEG): container finished" podID="f0db3b99b7954e19a6b4da51857943e0" containerID="2c03b055a0212025b10c111f4d14a1de5905eb9e16e87dffd9dd16492c9cde1b" exitCode=0 Oct 31 00:13:59.347254 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:59.347080 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-56.us-west-2.compute.internal" event={"ID":"f0db3b99b7954e19a6b4da51857943e0","Type":"ContainerDied","Data":"2c03b055a0212025b10c111f4d14a1de5905eb9e16e87dffd9dd16492c9cde1b"} Oct 31 00:13:59.578748 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:13:59.578674 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-56.us-west-2.compute.internal" podStartSLOduration=5.5786502 podStartE2EDuration="5.5786502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:13:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:13:59.459163122 +0000 UTC m=+6.691698574" watchObservedRunningTime="2026-10-31 00:13:59.5786502 +0000 UTC m=+6.811185653" Oct 31 00:14:00.274731 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:00.274265 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rjfn2" Oct 31 00:14:00.274731 ip-10-0-137-56 kubenswrapper[2581]: E1031 00:14:00.274428 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rjfn2" podUID="a077dea4-b5a3-4335-bb6a-7618e794eb8f" Oct 31 00:14:00.274731 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:00.274506 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fwgtx" Oct 31 00:14:00.274731 ip-10-0-137-56 kubenswrapper[2581]: E1031 00:14:00.274578 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fwgtx" podUID="d02ae6b5-4860-45b3-8d16-4a49531b77d4" Oct 31 00:14:00.274731 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:00.274627 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ml8n5" Oct 31 00:14:00.274731 ip-10-0-137-56 kubenswrapper[2581]: E1031 00:14:00.274691 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ml8n5" podUID="0cbf0042-931c-4135-9330-f9e798b7a567" Oct 31 00:14:00.356176 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:00.356136 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-56.us-west-2.compute.internal" event={"ID":"f0db3b99b7954e19a6b4da51857943e0","Type":"ContainerStarted","Data":"5dffef95bb6a69122bd4e904e0470bd2df4e0f09c15d0bf8f6008912321e57f1"} Oct 31 00:14:00.360398 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:00.360269 2581 generic.go:356] "Generic (PLEG): container finished" podID="6a21f264-1e80-4eab-8c6e-27ca9ca19f0d" containerID="fcdf3d9574e31e0b3ce80efda5296f44c4b2d23a51348c64a8650eed1c376191" exitCode=0 Oct 31 00:14:00.360715 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:00.360639 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jsctc" event={"ID":"6a21f264-1e80-4eab-8c6e-27ca9ca19f0d","Type":"ContainerDied","Data":"fcdf3d9574e31e0b3ce80efda5296f44c4b2d23a51348c64a8650eed1c376191"} Oct 31 00:14:00.588472 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:00.588303 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-56.us-west-2.compute.internal" podStartSLOduration=7.5882795640000005 podStartE2EDuration="7.588279564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:13:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:14:00.470183846 +0000 UTC m=+7.702719295" watchObservedRunningTime="2026-10-31 00:14:00.588279564 +0000 UTC m=+7.820815012" Oct 31 00:14:02.113042 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:02.112950 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a077dea4-b5a3-4335-bb6a-7618e794eb8f-metrics-certs\") pod \"network-metrics-daemon-rjfn2\" (UID: \"a077dea4-b5a3-4335-bb6a-7618e794eb8f\") " pod="openshift-multus/network-metrics-daemon-rjfn2" Oct 31 00:14:02.113042 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:02.113023 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d02ae6b5-4860-45b3-8d16-4a49531b77d4-original-pull-secret\") pod \"global-pull-secret-syncer-fwgtx\" (UID: \"d02ae6b5-4860-45b3-8d16-4a49531b77d4\") " pod="kube-system/global-pull-secret-syncer-fwgtx" Oct 31 00:14:02.113575 ip-10-0-137-56 kubenswrapper[2581]: E1031 00:14:02.113156 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 00:14:02.113575 ip-10-0-137-56 kubenswrapper[2581]: E1031 00:14:02.113219 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d02ae6b5-4860-45b3-8d16-4a49531b77d4-original-pull-secret podName:d02ae6b5-4860-45b3-8d16-4a49531b77d4 nodeName:}" failed. No retries permitted until 2026-10-31 00:14:10.113201047 +0000 UTC m=+17.345736487 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d02ae6b5-4860-45b3-8d16-4a49531b77d4-original-pull-secret") pod "global-pull-secret-syncer-fwgtx" (UID: "d02ae6b5-4860-45b3-8d16-4a49531b77d4") : object "kube-system"/"original-pull-secret" not registered Oct 31 00:14:02.113575 ip-10-0-137-56 kubenswrapper[2581]: E1031 00:14:02.113303 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:14:02.113575 ip-10-0-137-56 kubenswrapper[2581]: E1031 00:14:02.113336 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a077dea4-b5a3-4335-bb6a-7618e794eb8f-metrics-certs podName:a077dea4-b5a3-4335-bb6a-7618e794eb8f nodeName:}" failed. No retries permitted until 2026-10-31 00:14:10.113325228 +0000 UTC m=+17.345860658 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a077dea4-b5a3-4335-bb6a-7618e794eb8f-metrics-certs") pod "network-metrics-daemon-rjfn2" (UID: "a077dea4-b5a3-4335-bb6a-7618e794eb8f") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:14:02.213699 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:02.213404 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ms4cs\" (UniqueName: \"kubernetes.io/projected/0cbf0042-931c-4135-9330-f9e798b7a567-kube-api-access-ms4cs\") pod \"network-check-target-ml8n5\" (UID: \"0cbf0042-931c-4135-9330-f9e798b7a567\") " pod="openshift-network-diagnostics/network-check-target-ml8n5" Oct 31 00:14:02.213699 ip-10-0-137-56 kubenswrapper[2581]: E1031 00:14:02.213593 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 00:14:02.213699 ip-10-0-137-56 kubenswrapper[2581]: E1031 00:14:02.213610 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 00:14:02.213699 ip-10-0-137-56 kubenswrapper[2581]: E1031 00:14:02.213620 2581 projected.go:196] Error preparing data for projected volume kube-api-access-ms4cs for pod openshift-network-diagnostics/network-check-target-ml8n5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:14:02.213699 ip-10-0-137-56 kubenswrapper[2581]: E1031 00:14:02.213674 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0cbf0042-931c-4135-9330-f9e798b7a567-kube-api-access-ms4cs podName:0cbf0042-931c-4135-9330-f9e798b7a567 nodeName:}" failed. No retries permitted until 2026-10-31 00:14:10.2136608 +0000 UTC m=+17.446196225 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-ms4cs" (UniqueName: "kubernetes.io/projected/0cbf0042-931c-4135-9330-f9e798b7a567-kube-api-access-ms4cs") pod "network-check-target-ml8n5" (UID: "0cbf0042-931c-4135-9330-f9e798b7a567") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:14:02.273273 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:02.273242 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rjfn2" Oct 31 00:14:02.273472 ip-10-0-137-56 kubenswrapper[2581]: E1031 00:14:02.273351 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rjfn2" podUID="a077dea4-b5a3-4335-bb6a-7618e794eb8f" Oct 31 00:14:02.273472 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:02.273427 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fwgtx" Oct 31 00:14:02.273589 ip-10-0-137-56 kubenswrapper[2581]: E1031 00:14:02.273566 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fwgtx" podUID="d02ae6b5-4860-45b3-8d16-4a49531b77d4" Oct 31 00:14:02.273645 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:02.273434 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ml8n5" Oct 31 00:14:02.273697 ip-10-0-137-56 kubenswrapper[2581]: E1031 00:14:02.273664 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ml8n5" podUID="0cbf0042-931c-4135-9330-f9e798b7a567" Oct 31 00:14:02.370432 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:02.370336 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-lvp2h" event={"ID":"2ed1e615-cbca-482c-89cc-0fe7d8e7e597","Type":"ContainerStarted","Data":"c95a19e2417b067353821add163dda71e530e83eb49b782a648201fa7417f045"} Oct 31 00:14:02.380576 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:02.380538 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-xlzv2" event={"ID":"eadb6436-5e5b-4116-8ed7-26aa935fa749","Type":"ContainerStarted","Data":"aee45b19c9fb4e1471d8d56d5bde7b5e47e2487ca613437d3de20183233e31eb"} Oct 31 00:14:02.591721 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:02.591663 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-lvp2h" podStartSLOduration=4.102524659 podStartE2EDuration="9.591628903s" totalImagesPullingTime="5.489104244s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:13:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:14:02.481157649 +0000 UTC m=+9.713693097" watchObservedRunningTime="2026-10-31 00:14:02.591628903 +0000 UTC m=+9.824164327" Oct 31 00:14:03.386334 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:03.385734 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ndd7n" event={"ID":"1bc3d212-b67c-4368-90a3-8e080a4f1452","Type":"ContainerStarted","Data":"7fedc2d6d088b1f4230a4c223b86a7196878dd04a33d02c61c9c4be0e1f139f8"} Oct 31 00:14:04.275205 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:04.274187 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ml8n5" Oct 31 00:14:04.275205 ip-10-0-137-56 kubenswrapper[2581]: E1031 00:14:04.274321 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ml8n5" podUID="0cbf0042-931c-4135-9330-f9e798b7a567" Oct 31 00:14:04.275205 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:04.274832 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rjfn2" Oct 31 00:14:04.275205 ip-10-0-137-56 kubenswrapper[2581]: E1031 00:14:04.274952 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rjfn2" podUID="a077dea4-b5a3-4335-bb6a-7618e794eb8f" Oct 31 00:14:04.275205 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:04.275017 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fwgtx" Oct 31 00:14:04.275205 ip-10-0-137-56 kubenswrapper[2581]: E1031 00:14:04.275108 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fwgtx" podUID="d02ae6b5-4860-45b3-8d16-4a49531b77d4" Oct 31 00:14:04.811075 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:04.811050 2581 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 31 00:14:05.204217 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:05.204044 2581 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T00:14:04.811074864Z","UUID":"e5b4b03a-aa17-4200-869e-74c40f277fc5","Handler":null,"Name":"","Endpoint":""} Oct 31 00:14:05.206044 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:05.205961 2581 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 31 00:14:05.206044 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:05.205991 2581 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 31 00:14:05.393337 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:05.393297 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-b6vpd" event={"ID":"9801ff00-9e9a-4141-8212-ca32bf0251b3","Type":"ContainerStarted","Data":"f6b30659e9037accf0cae594bb8270fff97ba91042d759645d6311fc09bb65bf"} Oct 31 00:14:05.396093 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:05.396061 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ndd7n" event={"ID":"1bc3d212-b67c-4368-90a3-8e080a4f1452","Type":"ContainerStarted","Data":"d02d8bfa8b823ebadaae59acc77c6305e16f7053e6ef0a10071570817d937905"} Oct 31 00:14:05.399270 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:05.399238 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gkm2c" event={"ID":"a7d0a920-91fe-4c4a-b9bc-67550c324e7f","Type":"ContainerStarted","Data":"65f8761ad686d99cc9b4c2d76e578a24d1ce8b5267bdc4b0d576752878a6ae3b"} Oct 31 00:14:05.506431 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:05.506311 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-b6vpd" podStartSLOduration=3.44584701 podStartE2EDuration="12.506281933s" totalImagesPullingTime="9.060434923s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:13:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:14:05.505776382 +0000 UTC m=+12.738311831" watchObservedRunningTime="2026-10-31 00:14:05.506281933 +0000 UTC m=+12.738817380" Oct 31 00:14:05.506750 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:05.506722 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-xlzv2" podStartSLOduration=6.442987423 podStartE2EDuration="12.506711822s" totalImagesPullingTime="6.063724399s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:13:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:14:02.592039353 +0000 UTC m=+9.824574800" watchObservedRunningTime="2026-10-31 00:14:05.506711822 +0000 UTC m=+12.739247270" Oct 31 00:14:05.616186 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:05.616126 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-gkm2c" podStartSLOduration=3.756496039 podStartE2EDuration="12.616107244s" totalImagesPullingTime="8.859611205s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:13:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:14:05.61543095 +0000 UTC m=+12.847966396" watchObservedRunningTime="2026-10-31 00:14:05.616107244 +0000 UTC m=+12.848642694" Oct 31 00:14:06.273279 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:06.273241 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fwgtx" Oct 31 00:14:06.273279 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:06.273266 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rjfn2" Oct 31 00:14:06.273806 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:06.273256 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ml8n5" Oct 31 00:14:06.273806 ip-10-0-137-56 kubenswrapper[2581]: E1031 00:14:06.273400 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fwgtx" podUID="d02ae6b5-4860-45b3-8d16-4a49531b77d4" Oct 31 00:14:06.273806 ip-10-0-137-56 kubenswrapper[2581]: E1031 00:14:06.273474 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rjfn2" podUID="a077dea4-b5a3-4335-bb6a-7618e794eb8f" Oct 31 00:14:06.273806 ip-10-0-137-56 kubenswrapper[2581]: E1031 00:14:06.273534 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ml8n5" podUID="0cbf0042-931c-4135-9330-f9e798b7a567" Oct 31 00:14:06.405404 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:06.405353 2581 generic.go:356] "Generic (PLEG): container finished" podID="6a21f264-1e80-4eab-8c6e-27ca9ca19f0d" containerID="52415b55a41cf87669b44191c47b1cde246de5060df99a6510cc3064d3d014e3" exitCode=0 Oct 31 00:14:06.405553 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:06.405473 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jsctc" event={"ID":"6a21f264-1e80-4eab-8c6e-27ca9ca19f0d","Type":"ContainerDied","Data":"52415b55a41cf87669b44191c47b1cde246de5060df99a6510cc3064d3d014e3"} Oct 31 00:14:06.408715 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:06.408689 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4nwn7" event={"ID":"978a9050-dddb-4869-85d8-84a001554c5b","Type":"ContainerStarted","Data":"fdb489d159a75ece4bd279f88a3f06f50cfb8f8a8016fe6c99293ee5d1883d2f"} Oct 31 00:14:06.656642 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:06.656557 2581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-lvp2h" Oct 31 00:14:06.657235 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:06.657213 2581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-lvp2h" Oct 31 00:14:06.790901 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:06.790839 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-4nwn7" podStartSLOduration=4.779125787 podStartE2EDuration="13.79082289s" totalImagesPullingTime="9.011697103s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:13:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:14:06.680442558 +0000 UTC m=+13.912978005" watchObservedRunningTime="2026-10-31 00:14:06.79082289 +0000 UTC m=+14.023358337" Oct 31 00:14:07.412380 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:07.412166 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vsfrc" event={"ID":"0eb4888f-40a8-4aba-85d8-3c845382bd02","Type":"ContainerStarted","Data":"9fb08154881cbd9a8e719e00def09f7f67ce5cf13d7440a7c243fb64d02a9c5a"} Oct 31 00:14:07.414668 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:07.414626 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ndd7n" event={"ID":"1bc3d212-b67c-4368-90a3-8e080a4f1452","Type":"ContainerStarted","Data":"bc67a3849b48310a83c27781bc992b65ed04d8d309598ae8baab6a4f8ff7d665"} Oct 31 00:14:07.525003 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:07.524946 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-vsfrc" podStartSLOduration=3.366431 podStartE2EDuration="14.524933432s" totalImagesPullingTime="11.158502432s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:13:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:14:07.524473139 +0000 UTC m=+14.757008588" watchObservedRunningTime="2026-10-31 00:14:07.524933432 +0000 UTC m=+14.757468878" Oct 31 00:14:07.638304 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:07.638249 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ndd7n" podStartSLOduration=4.45226888 podStartE2EDuration="14.638236543s" totalImagesPullingTime="10.185967663s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:13:53 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:14:07.637994815 +0000 UTC m=+14.870530263" watchObservedRunningTime="2026-10-31 00:14:07.638236543 +0000 UTC m=+14.870771989" Oct 31 00:14:08.274016 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:08.273984 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rjfn2" Oct 31 00:14:08.274176 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:08.274062 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ml8n5" Oct 31 00:14:08.274176 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:08.274090 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fwgtx" Oct 31 00:14:08.274300 ip-10-0-137-56 kubenswrapper[2581]: E1031 00:14:08.274207 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ml8n5" podUID="0cbf0042-931c-4135-9330-f9e798b7a567" Oct 31 00:14:08.274300 ip-10-0-137-56 kubenswrapper[2581]: E1031 00:14:08.274272 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fwgtx" podUID="d02ae6b5-4860-45b3-8d16-4a49531b77d4" Oct 31 00:14:08.274415 ip-10-0-137-56 kubenswrapper[2581]: E1031 00:14:08.274387 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rjfn2" podUID="a077dea4-b5a3-4335-bb6a-7618e794eb8f" Oct 31 00:14:08.418848 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:08.418817 2581 generic.go:356] "Generic (PLEG): container finished" podID="6a21f264-1e80-4eab-8c6e-27ca9ca19f0d" containerID="930d8f45dd17e51267539e5fa1767003a4800c189d8076391923826c00d17121" exitCode=0 Oct 31 00:14:08.419556 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:08.418875 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jsctc" event={"ID":"6a21f264-1e80-4eab-8c6e-27ca9ca19f0d","Type":"ContainerDied","Data":"930d8f45dd17e51267539e5fa1767003a4800c189d8076391923826c00d17121"} Oct 31 00:14:10.178002 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:10.177962 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a077dea4-b5a3-4335-bb6a-7618e794eb8f-metrics-certs\") pod \"network-metrics-daemon-rjfn2\" (UID: \"a077dea4-b5a3-4335-bb6a-7618e794eb8f\") " pod="openshift-multus/network-metrics-daemon-rjfn2" Oct 31 00:14:10.178783 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:10.178024 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d02ae6b5-4860-45b3-8d16-4a49531b77d4-original-pull-secret\") pod \"global-pull-secret-syncer-fwgtx\" (UID: \"d02ae6b5-4860-45b3-8d16-4a49531b77d4\") " pod="kube-system/global-pull-secret-syncer-fwgtx" Oct 31 00:14:10.178783 ip-10-0-137-56 kubenswrapper[2581]: E1031 00:14:10.178132 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 00:14:10.178783 ip-10-0-137-56 kubenswrapper[2581]: E1031 00:14:10.178133 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:14:10.178783 ip-10-0-137-56 kubenswrapper[2581]: E1031 00:14:10.178199 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a077dea4-b5a3-4335-bb6a-7618e794eb8f-metrics-certs podName:a077dea4-b5a3-4335-bb6a-7618e794eb8f nodeName:}" failed. No retries permitted until 2026-10-31 00:14:26.178180295 +0000 UTC m=+33.410715720 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a077dea4-b5a3-4335-bb6a-7618e794eb8f-metrics-certs") pod "network-metrics-daemon-rjfn2" (UID: "a077dea4-b5a3-4335-bb6a-7618e794eb8f") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:14:10.178783 ip-10-0-137-56 kubenswrapper[2581]: E1031 00:14:10.178217 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d02ae6b5-4860-45b3-8d16-4a49531b77d4-original-pull-secret podName:d02ae6b5-4860-45b3-8d16-4a49531b77d4 nodeName:}" failed. No retries permitted until 2026-10-31 00:14:26.178210101 +0000 UTC m=+33.410745526 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d02ae6b5-4860-45b3-8d16-4a49531b77d4-original-pull-secret") pod "global-pull-secret-syncer-fwgtx" (UID: "d02ae6b5-4860-45b3-8d16-4a49531b77d4") : object "kube-system"/"original-pull-secret" not registered Oct 31 00:14:10.273580 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:10.273539 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ml8n5" Oct 31 00:14:10.273767 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:10.273593 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rjfn2" Oct 31 00:14:10.273767 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:10.273538 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fwgtx" Oct 31 00:14:10.273767 ip-10-0-137-56 kubenswrapper[2581]: E1031 00:14:10.273670 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ml8n5" podUID="0cbf0042-931c-4135-9330-f9e798b7a567" Oct 31 00:14:10.273767 ip-10-0-137-56 kubenswrapper[2581]: E1031 00:14:10.273711 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fwgtx" podUID="d02ae6b5-4860-45b3-8d16-4a49531b77d4" Oct 31 00:14:10.273947 ip-10-0-137-56 kubenswrapper[2581]: E1031 00:14:10.273769 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rjfn2" podUID="a077dea4-b5a3-4335-bb6a-7618e794eb8f" Oct 31 00:14:10.278689 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:10.278661 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ms4cs\" (UniqueName: \"kubernetes.io/projected/0cbf0042-931c-4135-9330-f9e798b7a567-kube-api-access-ms4cs\") pod \"network-check-target-ml8n5\" (UID: \"0cbf0042-931c-4135-9330-f9e798b7a567\") " pod="openshift-network-diagnostics/network-check-target-ml8n5" Oct 31 00:14:10.278854 ip-10-0-137-56 kubenswrapper[2581]: E1031 00:14:10.278834 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 00:14:10.278895 ip-10-0-137-56 kubenswrapper[2581]: E1031 00:14:10.278866 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 00:14:10.278895 ip-10-0-137-56 kubenswrapper[2581]: E1031 00:14:10.278881 2581 projected.go:196] Error preparing data for projected volume kube-api-access-ms4cs for pod openshift-network-diagnostics/network-check-target-ml8n5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:14:10.278964 ip-10-0-137-56 kubenswrapper[2581]: E1031 00:14:10.278933 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0cbf0042-931c-4135-9330-f9e798b7a567-kube-api-access-ms4cs podName:0cbf0042-931c-4135-9330-f9e798b7a567 nodeName:}" failed. No retries permitted until 2026-10-31 00:14:26.278919139 +0000 UTC m=+33.511454568 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-ms4cs" (UniqueName: "kubernetes.io/projected/0cbf0042-931c-4135-9330-f9e798b7a567-kube-api-access-ms4cs") pod "network-check-target-ml8n5" (UID: "0cbf0042-931c-4135-9330-f9e798b7a567") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:14:10.428077 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:10.427056 2581 generic.go:356] "Generic (PLEG): container finished" podID="6a21f264-1e80-4eab-8c6e-27ca9ca19f0d" containerID="c795df5e107ca603cfcc0faff37467699bbf3e7b6bb9b8a695e9b270f5deeaa2" exitCode=0 Oct 31 00:14:10.428077 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:10.427128 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jsctc" event={"ID":"6a21f264-1e80-4eab-8c6e-27ca9ca19f0d","Type":"ContainerDied","Data":"c795df5e107ca603cfcc0faff37467699bbf3e7b6bb9b8a695e9b270f5deeaa2"} Oct 31 00:14:11.430780 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:11.430514 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" event={"ID":"0f8c3c03-a1fc-4376-98c6-feeeefa08068","Type":"ContainerStarted","Data":"9cbdd729f4c95049e7df839e58b6aa1bee83c5b1b599c09fe20873fa1872eba1"} Oct 31 00:14:12.273582 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:12.273338 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rjfn2" Oct 31 00:14:12.273758 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:12.273343 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fwgtx" Oct 31 00:14:12.273758 ip-10-0-137-56 kubenswrapper[2581]: E1031 00:14:12.273694 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rjfn2" podUID="a077dea4-b5a3-4335-bb6a-7618e794eb8f" Oct 31 00:14:12.273877 ip-10-0-137-56 kubenswrapper[2581]: E1031 00:14:12.273779 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fwgtx" podUID="d02ae6b5-4860-45b3-8d16-4a49531b77d4" Oct 31 00:14:12.273877 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:12.273402 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ml8n5" Oct 31 00:14:12.273955 ip-10-0-137-56 kubenswrapper[2581]: E1031 00:14:12.273875 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ml8n5" podUID="0cbf0042-931c-4135-9330-f9e798b7a567" Oct 31 00:14:12.436582 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:12.436558 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h9hsf_0f8c3c03-a1fc-4376-98c6-feeeefa08068/ovn-acl-logging/0.log" Oct 31 00:14:12.437120 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:12.436963 2581 generic.go:356] "Generic (PLEG): container finished" podID="0f8c3c03-a1fc-4376-98c6-feeeefa08068" containerID="e809bac0da7017764d3d8602eb7c48a7076f0a38548a4d2eb49344ce7cff62c6" exitCode=1 Oct 31 00:14:12.437120 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:12.437004 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" event={"ID":"0f8c3c03-a1fc-4376-98c6-feeeefa08068","Type":"ContainerStarted","Data":"024aac419d55956b632d29f0755e823bd6fa4d98b1b8e9b19e717eb3b123ad7e"} Oct 31 00:14:12.437120 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:12.437080 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" event={"ID":"0f8c3c03-a1fc-4376-98c6-feeeefa08068","Type":"ContainerStarted","Data":"54847071ca826e97d1201aed2c0d0a8bc38e97487e23799d95e3078d1b84154b"} Oct 31 00:14:12.437283 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:12.437122 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" event={"ID":"0f8c3c03-a1fc-4376-98c6-feeeefa08068","Type":"ContainerStarted","Data":"c45c283b8986a58c810273b79b2c574409b3977abdfaf009bc71e5bde70138b8"} Oct 31 00:14:12.437283 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:12.437169 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" event={"ID":"0f8c3c03-a1fc-4376-98c6-feeeefa08068","Type":"ContainerStarted","Data":"f131dea1828b4e83f12d0eb7034ef192fd57881e3d94fab343ec79a39ac7d6ae"} Oct 31 00:14:12.437283 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:12.437216 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" event={"ID":"0f8c3c03-a1fc-4376-98c6-feeeefa08068","Type":"ContainerDied","Data":"e809bac0da7017764d3d8602eb7c48a7076f0a38548a4d2eb49344ce7cff62c6"} Oct 31 00:14:13.441884 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:13.441851 2581 generic.go:356] "Generic (PLEG): container finished" podID="6a21f264-1e80-4eab-8c6e-27ca9ca19f0d" containerID="9c9530986ffadecb481dc14c4227502352d72f18ed883588fc22c6909388577e" exitCode=0 Oct 31 00:14:13.442408 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:13.441907 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jsctc" event={"ID":"6a21f264-1e80-4eab-8c6e-27ca9ca19f0d","Type":"ContainerDied","Data":"9c9530986ffadecb481dc14c4227502352d72f18ed883588fc22c6909388577e"} Oct 31 00:14:14.273335 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:14.273308 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fwgtx" Oct 31 00:14:14.273488 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:14.273335 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ml8n5" Oct 31 00:14:14.273488 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:14.273311 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rjfn2" Oct 31 00:14:14.273488 ip-10-0-137-56 kubenswrapper[2581]: E1031 00:14:14.273438 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fwgtx" podUID="d02ae6b5-4860-45b3-8d16-4a49531b77d4" Oct 31 00:14:14.273619 ip-10-0-137-56 kubenswrapper[2581]: E1031 00:14:14.273502 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rjfn2" podUID="a077dea4-b5a3-4335-bb6a-7618e794eb8f" Oct 31 00:14:14.273619 ip-10-0-137-56 kubenswrapper[2581]: E1031 00:14:14.273550 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ml8n5" podUID="0cbf0042-931c-4135-9330-f9e798b7a567" Oct 31 00:14:14.449233 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:14.449197 2581 generic.go:356] "Generic (PLEG): container finished" podID="6a21f264-1e80-4eab-8c6e-27ca9ca19f0d" containerID="440774ebb1e3724120772862c008f21d5fd77338c168c6fc0def1b9c97cfc0e5" exitCode=0 Oct 31 00:14:14.449717 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:14.449275 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jsctc" event={"ID":"6a21f264-1e80-4eab-8c6e-27ca9ca19f0d","Type":"ContainerDied","Data":"440774ebb1e3724120772862c008f21d5fd77338c168c6fc0def1b9c97cfc0e5"} Oct 31 00:14:14.454755 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:14.454733 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h9hsf_0f8c3c03-a1fc-4376-98c6-feeeefa08068/ovn-acl-logging/0.log" Oct 31 00:14:14.455158 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:14.455138 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" event={"ID":"0f8c3c03-a1fc-4376-98c6-feeeefa08068","Type":"ContainerStarted","Data":"fc0d87c6c61405472e1293a34cd6d2c0a4eb30971d059b4fe7a0c53028d757a2"} Oct 31 00:14:14.760880 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:14.760844 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-lvp2h" Oct 31 00:14:14.761080 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:14.760991 2581 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 00:14:14.761557 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:14.761543 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-lvp2h" Oct 31 00:14:15.460671 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:15.460632 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jsctc" event={"ID":"6a21f264-1e80-4eab-8c6e-27ca9ca19f0d","Type":"ContainerStarted","Data":"b50d6716ae46e4428c51434995a4a66ece08ac03e1ce232c47a550e34cce4d44"} Oct 31 00:14:15.598650 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:15.598601 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-jsctc" podStartSLOduration=9.048738704 podStartE2EDuration="22.598585793s" totalImagesPullingTime="13.549847089s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:13:53 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:14:15.598505273 +0000 UTC m=+22.831040720" watchObservedRunningTime="2026-10-31 00:14:15.598585793 +0000 UTC m=+22.831121240" Oct 31 00:14:16.273814 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:16.273776 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ml8n5" Oct 31 00:14:16.273979 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:16.273836 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rjfn2" Oct 31 00:14:16.273979 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:16.273840 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fwgtx" Oct 31 00:14:16.273979 ip-10-0-137-56 kubenswrapper[2581]: E1031 00:14:16.273929 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rjfn2" podUID="a077dea4-b5a3-4335-bb6a-7618e794eb8f" Oct 31 00:14:16.274086 ip-10-0-137-56 kubenswrapper[2581]: E1031 00:14:16.273991 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ml8n5" podUID="0cbf0042-931c-4135-9330-f9e798b7a567" Oct 31 00:14:16.274086 ip-10-0-137-56 kubenswrapper[2581]: E1031 00:14:16.273985 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fwgtx" podUID="d02ae6b5-4860-45b3-8d16-4a49531b77d4" Oct 31 00:14:16.466593 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:16.466515 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h9hsf_0f8c3c03-a1fc-4376-98c6-feeeefa08068/ovn-acl-logging/0.log" Oct 31 00:14:16.467210 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:16.466925 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" event={"ID":"0f8c3c03-a1fc-4376-98c6-feeeefa08068","Type":"ContainerStarted","Data":"8cfdb6c29105521ee0f4b6379d64e462f5b8b732fa2a3e0081b7e8471b4c3d18"} Oct 31 00:14:16.467516 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:16.467500 2581 scope.go:122] "RemoveContainer" containerID="e809bac0da7017764d3d8602eb7c48a7076f0a38548a4d2eb49344ce7cff62c6" Oct 31 00:14:17.473763 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:17.473591 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h9hsf_0f8c3c03-a1fc-4376-98c6-feeeefa08068/ovn-acl-logging/0.log" Oct 31 00:14:17.474192 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:17.474146 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" event={"ID":"0f8c3c03-a1fc-4376-98c6-feeeefa08068","Type":"ContainerStarted","Data":"9ed6fe42cc2064b9d50ffe9233436572fa1e809c511b48636dad336f1453f7bc"} Oct 31 00:14:17.474402 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:17.474386 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:14:17.474489 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:17.474455 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:14:17.474542 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:17.474505 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:14:17.490908 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:17.490882 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:14:17.491057 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:17.490986 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:14:17.739555 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:17.739454 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" podStartSLOduration=9.12865269 podStartE2EDuration="24.739436697s" totalImagesPullingTime="15.610784007s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:13:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:14:17.691868998 +0000 UTC m=+24.924404444" watchObservedRunningTime="2026-10-31 00:14:17.739436697 +0000 UTC m=+24.971972144" Oct 31 00:14:18.273617 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:18.273585 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rjfn2" Oct 31 00:14:18.273617 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:18.273608 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fwgtx" Oct 31 00:14:18.273802 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:18.273586 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ml8n5" Oct 31 00:14:18.273802 ip-10-0-137-56 kubenswrapper[2581]: E1031 00:14:18.273708 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rjfn2" podUID="a077dea4-b5a3-4335-bb6a-7618e794eb8f" Oct 31 00:14:18.273802 ip-10-0-137-56 kubenswrapper[2581]: E1031 00:14:18.273762 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ml8n5" podUID="0cbf0042-931c-4135-9330-f9e798b7a567" Oct 31 00:14:18.273894 ip-10-0-137-56 kubenswrapper[2581]: E1031 00:14:18.273865 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fwgtx" podUID="d02ae6b5-4860-45b3-8d16-4a49531b77d4" Oct 31 00:14:18.798333 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:18.798305 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-fwgtx"] Oct 31 00:14:18.798708 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:18.798425 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fwgtx" Oct 31 00:14:18.798708 ip-10-0-137-56 kubenswrapper[2581]: E1031 00:14:18.798509 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fwgtx" podUID="d02ae6b5-4860-45b3-8d16-4a49531b77d4" Oct 31 00:14:18.801506 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:18.801478 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ml8n5"] Oct 31 00:14:18.801654 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:18.801589 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ml8n5" Oct 31 00:14:18.801726 ip-10-0-137-56 kubenswrapper[2581]: E1031 00:14:18.801685 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ml8n5" podUID="0cbf0042-931c-4135-9330-f9e798b7a567" Oct 31 00:14:18.802161 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:18.802143 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-rjfn2"] Oct 31 00:14:18.802258 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:18.802220 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rjfn2" Oct 31 00:14:18.802320 ip-10-0-137-56 kubenswrapper[2581]: E1031 00:14:18.802302 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rjfn2" podUID="a077dea4-b5a3-4335-bb6a-7618e794eb8f" Oct 31 00:14:20.273506 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:20.273473 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ml8n5" Oct 31 00:14:20.273506 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:20.273503 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fwgtx" Oct 31 00:14:20.273917 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:20.273515 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rjfn2" Oct 31 00:14:20.273917 ip-10-0-137-56 kubenswrapper[2581]: E1031 00:14:20.273582 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ml8n5" podUID="0cbf0042-931c-4135-9330-f9e798b7a567" Oct 31 00:14:20.273917 ip-10-0-137-56 kubenswrapper[2581]: E1031 00:14:20.273645 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rjfn2" podUID="a077dea4-b5a3-4335-bb6a-7618e794eb8f" Oct 31 00:14:20.273917 ip-10-0-137-56 kubenswrapper[2581]: E1031 00:14:20.273696 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fwgtx" podUID="d02ae6b5-4860-45b3-8d16-4a49531b77d4" Oct 31 00:14:22.273550 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:22.273522 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ml8n5" Oct 31 00:14:22.273886 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:22.273522 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rjfn2" Oct 31 00:14:22.273886 ip-10-0-137-56 kubenswrapper[2581]: E1031 00:14:22.273635 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ml8n5" podUID="0cbf0042-931c-4135-9330-f9e798b7a567" Oct 31 00:14:22.273886 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:22.273522 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fwgtx" Oct 31 00:14:22.273886 ip-10-0-137-56 kubenswrapper[2581]: E1031 00:14:22.273716 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rjfn2" podUID="a077dea4-b5a3-4335-bb6a-7618e794eb8f" Oct 31 00:14:22.273886 ip-10-0-137-56 kubenswrapper[2581]: E1031 00:14:22.273775 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fwgtx" podUID="d02ae6b5-4860-45b3-8d16-4a49531b77d4" Oct 31 00:14:24.273913 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:24.273877 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rjfn2" Oct 31 00:14:24.273913 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:24.273904 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fwgtx" Oct 31 00:14:24.274302 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:24.273909 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ml8n5" Oct 31 00:14:24.274302 ip-10-0-137-56 kubenswrapper[2581]: E1031 00:14:24.273995 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rjfn2" podUID="a077dea4-b5a3-4335-bb6a-7618e794eb8f" Oct 31 00:14:24.274302 ip-10-0-137-56 kubenswrapper[2581]: E1031 00:14:24.274052 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fwgtx" podUID="d02ae6b5-4860-45b3-8d16-4a49531b77d4" Oct 31 00:14:24.274302 ip-10-0-137-56 kubenswrapper[2581]: E1031 00:14:24.274155 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ml8n5" podUID="0cbf0042-931c-4135-9330-f9e798b7a567" Oct 31 00:14:24.524606 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:24.524573 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-56.us-west-2.compute.internal" event="NodeReady" Oct 31 00:14:24.524762 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:24.524721 2581 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 31 00:14:26.189311 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:26.189108 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d02ae6b5-4860-45b3-8d16-4a49531b77d4-original-pull-secret\") pod \"global-pull-secret-syncer-fwgtx\" (UID: \"d02ae6b5-4860-45b3-8d16-4a49531b77d4\") " pod="kube-system/global-pull-secret-syncer-fwgtx" Oct 31 00:14:26.189723 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:26.189356 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a077dea4-b5a3-4335-bb6a-7618e794eb8f-metrics-certs\") pod \"network-metrics-daemon-rjfn2\" (UID: \"a077dea4-b5a3-4335-bb6a-7618e794eb8f\") " pod="openshift-multus/network-metrics-daemon-rjfn2" Oct 31 00:14:26.189723 ip-10-0-137-56 kubenswrapper[2581]: E1031 00:14:26.189276 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 00:14:26.189723 ip-10-0-137-56 kubenswrapper[2581]: E1031 00:14:26.189479 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:14:26.189723 ip-10-0-137-56 kubenswrapper[2581]: E1031 00:14:26.189524 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d02ae6b5-4860-45b3-8d16-4a49531b77d4-original-pull-secret podName:d02ae6b5-4860-45b3-8d16-4a49531b77d4 nodeName:}" failed. No retries permitted until 2026-10-31 00:14:58.189499759 +0000 UTC m=+65.422035209 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d02ae6b5-4860-45b3-8d16-4a49531b77d4-original-pull-secret") pod "global-pull-secret-syncer-fwgtx" (UID: "d02ae6b5-4860-45b3-8d16-4a49531b77d4") : object "kube-system"/"original-pull-secret" not registered Oct 31 00:14:26.189723 ip-10-0-137-56 kubenswrapper[2581]: E1031 00:14:26.189541 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a077dea4-b5a3-4335-bb6a-7618e794eb8f-metrics-certs podName:a077dea4-b5a3-4335-bb6a-7618e794eb8f nodeName:}" failed. No retries permitted until 2026-10-31 00:14:58.189533654 +0000 UTC m=+65.422069079 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a077dea4-b5a3-4335-bb6a-7618e794eb8f-metrics-certs") pod "network-metrics-daemon-rjfn2" (UID: "a077dea4-b5a3-4335-bb6a-7618e794eb8f") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:14:26.273533 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:26.273504 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fwgtx" Oct 31 00:14:26.273639 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:26.273528 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rjfn2" Oct 31 00:14:26.273639 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:26.273627 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ml8n5" Oct 31 00:14:26.289619 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:26.289592 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ms4cs\" (UniqueName: \"kubernetes.io/projected/0cbf0042-931c-4135-9330-f9e798b7a567-kube-api-access-ms4cs\") pod \"network-check-target-ml8n5\" (UID: \"0cbf0042-931c-4135-9330-f9e798b7a567\") " pod="openshift-network-diagnostics/network-check-target-ml8n5" Oct 31 00:14:26.323565 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:26.323540 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 31 00:14:26.345098 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:26.345065 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 31 00:14:26.345098 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:26.345068 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 31 00:14:26.354364 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:26.354333 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ms4cs\" (UniqueName: \"kubernetes.io/projected/0cbf0042-931c-4135-9330-f9e798b7a567-kube-api-access-ms4cs\") pod \"network-check-target-ml8n5\" (UID: \"0cbf0042-931c-4135-9330-f9e798b7a567\") " pod="openshift-network-diagnostics/network-check-target-ml8n5" Oct 31 00:14:26.371037 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:26.371008 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-plmbm\"" Oct 31 00:14:26.371037 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:26.371007 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-lg2hb\"" Oct 31 00:14:26.371250 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:26.371108 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zk5sj\"" Oct 31 00:14:26.371250 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:26.371161 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 31 00:14:26.620083 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:26.620049 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ml8n5" Oct 31 00:14:27.047023 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:27.046997 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gkm2c_a7d0a920-91fe-4c4a-b9bc-67550c324e7f/dns-node-resolver/0.log" Oct 31 00:14:27.049004 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:27.048981 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ml8n5"] Oct 31 00:14:27.052991 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:14:27.052960 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cbf0042_931c_4135_9330_f9e798b7a567.slice/crio-13aeedbbd5795e01fadf0288886faebf6854062a3d743cbf2896b364a908f55d WatchSource:0}: Error finding container 13aeedbbd5795e01fadf0288886faebf6854062a3d743cbf2896b364a908f55d: Status 404 returned error can't find the container with id 13aeedbbd5795e01fadf0288886faebf6854062a3d743cbf2896b364a908f55d Oct 31 00:14:27.496466 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:27.496340 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ml8n5" event={"ID":"0cbf0042-931c-4135-9330-f9e798b7a567","Type":"ContainerStarted","Data":"13aeedbbd5795e01fadf0288886faebf6854062a3d743cbf2896b364a908f55d"} Oct 31 00:14:27.863467 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:27.863438 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-xlzv2_eadb6436-5e5b-4116-8ed7-26aa935fa749/node-ca/0.log" Oct 31 00:14:31.507573 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:31.507535 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ml8n5" event={"ID":"0cbf0042-931c-4135-9330-f9e798b7a567","Type":"ContainerStarted","Data":"9a1ce846bf02c531afded9f5a4bc33bedceb47f4cce5007fc2637a83734f8fa1"} Oct 31 00:14:31.507966 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:31.507673 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-ml8n5" Oct 31 00:14:31.620685 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:31.620578 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-ml8n5" podStartSLOduration=34.304828757 podStartE2EDuration="38.620564245s" totalImagesPullingTime="4.315735488s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:13:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:14:31.620247259 +0000 UTC m=+38.852782699" watchObservedRunningTime="2026-10-31 00:14:31.620564245 +0000 UTC m=+38.853099692" Oct 31 00:14:33.626598 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:33.626550 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvp2h_2ed1e615-cbca-482c-89cc-0fe7d8e7e597/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:33.836146 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:33.836115 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvp2h_2ed1e615-cbca-482c-89cc-0fe7d8e7e597/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:34.032624 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:34.032598 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvp2h_2ed1e615-cbca-482c-89cc-0fe7d8e7e597/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:34.230600 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:34.230567 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lvp2h_2ed1e615-cbca-482c-89cc-0fe7d8e7e597/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:44.479622 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:44.479594 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-lrh4r"] Oct 31 00:14:44.481180 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:44.481164 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lrh4r" Oct 31 00:14:44.490063 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:44.490036 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-x5s75"] Oct 31 00:14:44.491768 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:44.491753 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-x5s75" Oct 31 00:14:44.511885 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:44.511857 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-nq56p"] Oct 31 00:14:44.513709 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:44.513692 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-nq56p" Oct 31 00:14:44.531452 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:44.531411 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 31 00:14:44.536800 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:44.536781 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-74dxc\"" Oct 31 00:14:44.536933 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:44.536873 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 31 00:14:44.559351 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:44.559322 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-lrh4r"] Oct 31 00:14:44.563431 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:44.563413 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 31 00:14:44.580608 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:44.580582 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 31 00:14:44.581300 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:44.581270 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-x5s75"] Oct 31 00:14:44.589719 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:44.589698 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 31 00:14:44.591612 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:44.591592 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 31 00:14:44.595723 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:44.595315 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6644876fb8-9bszd"] Oct 31 00:14:44.596602 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:44.596575 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 31 00:14:44.601052 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:44.601029 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-nq56p"] Oct 31 00:14:44.601157 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:44.601142 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6644876fb8-9bszd" Oct 31 00:14:44.615205 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:44.615181 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d22409bc-1bcf-46b1-9789-6f7298566a7f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nq56p\" (UID: \"d22409bc-1bcf-46b1-9789-6f7298566a7f\") " pod="openshift-insights/insights-runtime-extractor-nq56p" Oct 31 00:14:44.615321 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:44.615212 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d22409bc-1bcf-46b1-9789-6f7298566a7f-crio-socket\") pod \"insights-runtime-extractor-nq56p\" (UID: \"d22409bc-1bcf-46b1-9789-6f7298566a7f\") " pod="openshift-insights/insights-runtime-extractor-nq56p" Oct 31 00:14:44.615321 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:44.615229 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d22409bc-1bcf-46b1-9789-6f7298566a7f-data-volume\") pod \"insights-runtime-extractor-nq56p\" (UID: \"d22409bc-1bcf-46b1-9789-6f7298566a7f\") " pod="openshift-insights/insights-runtime-extractor-nq56p" Oct 31 00:14:44.615321 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:44.615247 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pc97\" (UniqueName: \"kubernetes.io/projected/97d11160-2114-4fbf-95c2-fa7b4c9d2260-kube-api-access-6pc97\") pod \"ingress-canary-x5s75\" (UID: \"97d11160-2114-4fbf-95c2-fa7b4c9d2260\") " pod="openshift-ingress-canary/ingress-canary-x5s75" Oct 31 00:14:44.615452 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:44.615348 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lsrt\" (UniqueName: \"kubernetes.io/projected/83d351d1-c6e7-43a6-a0fb-eaecefa8c5b2-kube-api-access-5lsrt\") pod \"dns-default-lrh4r\" (UID: \"83d351d1-c6e7-43a6-a0fb-eaecefa8c5b2\") " pod="openshift-dns/dns-default-lrh4r" Oct 31 00:14:44.615452 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:44.615391 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d22409bc-1bcf-46b1-9789-6f7298566a7f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nq56p\" (UID: \"d22409bc-1bcf-46b1-9789-6f7298566a7f\") " pod="openshift-insights/insights-runtime-extractor-nq56p" Oct 31 00:14:44.615452 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:44.615410 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srgbh\" (UniqueName: \"kubernetes.io/projected/d22409bc-1bcf-46b1-9789-6f7298566a7f-kube-api-access-srgbh\") pod \"insights-runtime-extractor-nq56p\" (UID: \"d22409bc-1bcf-46b1-9789-6f7298566a7f\") " pod="openshift-insights/insights-runtime-extractor-nq56p" Oct 31 00:14:44.615542 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:44.615460 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83d351d1-c6e7-43a6-a0fb-eaecefa8c5b2-tmp-dir\") pod \"dns-default-lrh4r\" (UID: \"83d351d1-c6e7-43a6-a0fb-eaecefa8c5b2\") " pod="openshift-dns/dns-default-lrh4r" Oct 31 00:14:44.615542 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:44.615489 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/83d351d1-c6e7-43a6-a0fb-eaecefa8c5b2-config-volume\") pod \"dns-default-lrh4r\" (UID: \"83d351d1-c6e7-43a6-a0fb-eaecefa8c5b2\") " pod="openshift-dns/dns-default-lrh4r" Oct 31 00:14:44.615605 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:44.615552 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/83d351d1-c6e7-43a6-a0fb-eaecefa8c5b2-metrics-tls\") pod \"dns-default-lrh4r\" (UID: \"83d351d1-c6e7-43a6-a0fb-eaecefa8c5b2\") " pod="openshift-dns/dns-default-lrh4r" Oct 31 00:14:44.615605 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:44.615588 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/97d11160-2114-4fbf-95c2-fa7b4c9d2260-cert\") pod \"ingress-canary-x5s75\" (UID: \"97d11160-2114-4fbf-95c2-fa7b4c9d2260\") " pod="openshift-ingress-canary/ingress-canary-x5s75" Oct 31 00:14:44.632149 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:44.632123 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 31 00:14:44.632276 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:44.632174 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-hcz6l\"" Oct 31 00:14:44.632328 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:44.632286 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-qszwm\"" Oct 31 00:14:44.632328 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:44.632290 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 31 00:14:44.701978 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:44.701946 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 31 00:14:44.713279 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:44.713239 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6644876fb8-9bszd"] Oct 31 00:14:44.715902 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:44.715873 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83d351d1-c6e7-43a6-a0fb-eaecefa8c5b2-tmp-dir\") pod \"dns-default-lrh4r\" (UID: \"83d351d1-c6e7-43a6-a0fb-eaecefa8c5b2\") " pod="openshift-dns/dns-default-lrh4r" Oct 31 00:14:44.716020 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:44.715914 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5c58f866-ef80-42d5-aab0-0cdefbc7a158-bound-sa-token\") pod \"image-registry-6644876fb8-9bszd\" (UID: \"5c58f866-ef80-42d5-aab0-0cdefbc7a158\") " pod="openshift-image-registry/image-registry-6644876fb8-9bszd" Oct 31 00:14:44.716020 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:44.715935 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/83d351d1-c6e7-43a6-a0fb-eaecefa8c5b2-config-volume\") pod \"dns-default-lrh4r\" (UID: \"83d351d1-c6e7-43a6-a0fb-eaecefa8c5b2\") " pod="openshift-dns/dns-default-lrh4r" Oct 31 00:14:44.716139 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:44.716038 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5c58f866-ef80-42d5-aab0-0cdefbc7a158-image-registry-private-configuration\") pod \"image-registry-6644876fb8-9bszd\" (UID: \"5c58f866-ef80-42d5-aab0-0cdefbc7a158\") " pod="openshift-image-registry/image-registry-6644876fb8-9bszd" Oct 31 00:14:44.716139 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:44.716121 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5c58f866-ef80-42d5-aab0-0cdefbc7a158-registry-tls\") pod \"image-registry-6644876fb8-9bszd\" (UID: \"5c58f866-ef80-42d5-aab0-0cdefbc7a158\") " pod="openshift-image-registry/image-registry-6644876fb8-9bszd" Oct 31 00:14:44.716406 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:44.716358 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83d351d1-c6e7-43a6-a0fb-eaecefa8c5b2-tmp-dir\") pod \"dns-default-lrh4r\" (UID: \"83d351d1-c6e7-43a6-a0fb-eaecefa8c5b2\") " pod="openshift-dns/dns-default-lrh4r" Oct 31 00:14:44.716493 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:44.716480 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5c58f866-ef80-42d5-aab0-0cdefbc7a158-ca-trust-extracted\") pod \"image-registry-6644876fb8-9bszd\" (UID: \"5c58f866-ef80-42d5-aab0-0cdefbc7a158\") " pod="openshift-image-registry/image-registry-6644876fb8-9bszd" Oct 31 00:14:44.716537 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:44.716502 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5c58f866-ef80-42d5-aab0-0cdefbc7a158-trusted-ca\") pod \"image-registry-6644876fb8-9bszd\" (UID: \"5c58f866-ef80-42d5-aab0-0cdefbc7a158\") " pod="openshift-image-registry/image-registry-6644876fb8-9bszd" Oct 31 00:14:44.716572 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:44.716536 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/83d351d1-c6e7-43a6-a0fb-eaecefa8c5b2-metrics-tls\") pod \"dns-default-lrh4r\" (UID: \"83d351d1-c6e7-43a6-a0fb-eaecefa8c5b2\") " pod="openshift-dns/dns-default-lrh4r" Oct 31 00:14:44.716572 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:44.716567 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/97d11160-2114-4fbf-95c2-fa7b4c9d2260-cert\") pod \"ingress-canary-x5s75\" (UID: \"97d11160-2114-4fbf-95c2-fa7b4c9d2260\") " pod="openshift-ingress-canary/ingress-canary-x5s75" Oct 31 00:14:44.716631 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:44.716582 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5hld\" (UniqueName: \"kubernetes.io/projected/5c58f866-ef80-42d5-aab0-0cdefbc7a158-kube-api-access-n5hld\") pod \"image-registry-6644876fb8-9bszd\" (UID: \"5c58f866-ef80-42d5-aab0-0cdefbc7a158\") " pod="openshift-image-registry/image-registry-6644876fb8-9bszd" Oct 31 00:14:44.716631 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:44.716600 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d22409bc-1bcf-46b1-9789-6f7298566a7f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nq56p\" (UID: \"d22409bc-1bcf-46b1-9789-6f7298566a7f\") " pod="openshift-insights/insights-runtime-extractor-nq56p" Oct 31 00:14:44.716631 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:44.716620 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d22409bc-1bcf-46b1-9789-6f7298566a7f-crio-socket\") pod \"insights-runtime-extractor-nq56p\" (UID: \"d22409bc-1bcf-46b1-9789-6f7298566a7f\") " pod="openshift-insights/insights-runtime-extractor-nq56p" Oct 31 00:14:44.716712 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:44.716636 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d22409bc-1bcf-46b1-9789-6f7298566a7f-data-volume\") pod \"insights-runtime-extractor-nq56p\" (UID: \"d22409bc-1bcf-46b1-9789-6f7298566a7f\") " pod="openshift-insights/insights-runtime-extractor-nq56p" Oct 31 00:14:44.716712 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:44.716652 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5c58f866-ef80-42d5-aab0-0cdefbc7a158-installation-pull-secrets\") pod \"image-registry-6644876fb8-9bszd\" (UID: \"5c58f866-ef80-42d5-aab0-0cdefbc7a158\") " pod="openshift-image-registry/image-registry-6644876fb8-9bszd" Oct 31 00:14:44.716712 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:44.716673 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pc97\" (UniqueName: \"kubernetes.io/projected/97d11160-2114-4fbf-95c2-fa7b4c9d2260-kube-api-access-6pc97\") pod \"ingress-canary-x5s75\" (UID: \"97d11160-2114-4fbf-95c2-fa7b4c9d2260\") " pod="openshift-ingress-canary/ingress-canary-x5s75" Oct 31 00:14:44.716712 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:44.716701 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lsrt\" (UniqueName: \"kubernetes.io/projected/83d351d1-c6e7-43a6-a0fb-eaecefa8c5b2-kube-api-access-5lsrt\") pod \"dns-default-lrh4r\" (UID: \"83d351d1-c6e7-43a6-a0fb-eaecefa8c5b2\") " pod="openshift-dns/dns-default-lrh4r" Oct 31 00:14:44.716822 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:44.716715 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d22409bc-1bcf-46b1-9789-6f7298566a7f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nq56p\" (UID: \"d22409bc-1bcf-46b1-9789-6f7298566a7f\") " pod="openshift-insights/insights-runtime-extractor-nq56p" Oct 31 00:14:44.716822 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:44.716732 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srgbh\" (UniqueName: \"kubernetes.io/projected/d22409bc-1bcf-46b1-9789-6f7298566a7f-kube-api-access-srgbh\") pod \"insights-runtime-extractor-nq56p\" (UID: \"d22409bc-1bcf-46b1-9789-6f7298566a7f\") " pod="openshift-insights/insights-runtime-extractor-nq56p" Oct 31 00:14:44.716822 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:44.716746 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5c58f866-ef80-42d5-aab0-0cdefbc7a158-registry-certificates\") pod \"image-registry-6644876fb8-9bszd\" (UID: \"5c58f866-ef80-42d5-aab0-0cdefbc7a158\") " pod="openshift-image-registry/image-registry-6644876fb8-9bszd" Oct 31 00:14:44.716956 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:44.716816 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/83d351d1-c6e7-43a6-a0fb-eaecefa8c5b2-config-volume\") pod \"dns-default-lrh4r\" (UID: \"83d351d1-c6e7-43a6-a0fb-eaecefa8c5b2\") " pod="openshift-dns/dns-default-lrh4r" Oct 31 00:14:44.717117 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:44.717093 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d22409bc-1bcf-46b1-9789-6f7298566a7f-data-volume\") pod \"insights-runtime-extractor-nq56p\" (UID: \"d22409bc-1bcf-46b1-9789-6f7298566a7f\") " pod="openshift-insights/insights-runtime-extractor-nq56p" Oct 31 00:14:44.717215 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:44.717172 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d22409bc-1bcf-46b1-9789-6f7298566a7f-crio-socket\") pod \"insights-runtime-extractor-nq56p\" (UID: \"d22409bc-1bcf-46b1-9789-6f7298566a7f\") " pod="openshift-insights/insights-runtime-extractor-nq56p" Oct 31 00:14:44.717527 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:44.717499 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d22409bc-1bcf-46b1-9789-6f7298566a7f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nq56p\" (UID: \"d22409bc-1bcf-46b1-9789-6f7298566a7f\") " pod="openshift-insights/insights-runtime-extractor-nq56p" Oct 31 00:14:44.719176 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:44.719156 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/83d351d1-c6e7-43a6-a0fb-eaecefa8c5b2-metrics-tls\") pod \"dns-default-lrh4r\" (UID: \"83d351d1-c6e7-43a6-a0fb-eaecefa8c5b2\") " pod="openshift-dns/dns-default-lrh4r" Oct 31 00:14:44.719425 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:44.719410 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/97d11160-2114-4fbf-95c2-fa7b4c9d2260-cert\") pod \"ingress-canary-x5s75\" (UID: \"97d11160-2114-4fbf-95c2-fa7b4c9d2260\") " pod="openshift-ingress-canary/ingress-canary-x5s75" Oct 31 00:14:44.719669 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:44.719648 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d22409bc-1bcf-46b1-9789-6f7298566a7f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nq56p\" (UID: \"d22409bc-1bcf-46b1-9789-6f7298566a7f\") " pod="openshift-insights/insights-runtime-extractor-nq56p" Oct 31 00:14:44.736494 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:44.736427 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 31 00:14:44.736693 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:44.736677 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 31 00:14:44.736760 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:44.736706 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 31 00:14:44.736760 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:44.736732 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-pk6r6\"" Oct 31 00:14:44.775321 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:44.775281 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lsrt\" (UniqueName: \"kubernetes.io/projected/83d351d1-c6e7-43a6-a0fb-eaecefa8c5b2-kube-api-access-5lsrt\") pod \"dns-default-lrh4r\" (UID: \"83d351d1-c6e7-43a6-a0fb-eaecefa8c5b2\") " pod="openshift-dns/dns-default-lrh4r" Oct 31 00:14:44.775687 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:44.775669 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pc97\" (UniqueName: \"kubernetes.io/projected/97d11160-2114-4fbf-95c2-fa7b4c9d2260-kube-api-access-6pc97\") pod \"ingress-canary-x5s75\" (UID: \"97d11160-2114-4fbf-95c2-fa7b4c9d2260\") " pod="openshift-ingress-canary/ingress-canary-x5s75" Oct 31 00:14:44.775739 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:44.775712 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srgbh\" (UniqueName: \"kubernetes.io/projected/d22409bc-1bcf-46b1-9789-6f7298566a7f-kube-api-access-srgbh\") pod \"insights-runtime-extractor-nq56p\" (UID: \"d22409bc-1bcf-46b1-9789-6f7298566a7f\") " pod="openshift-insights/insights-runtime-extractor-nq56p" Oct 31 00:14:44.792674 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:44.792642 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lrh4r" Oct 31 00:14:44.800786 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:44.800760 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-x5s75" Oct 31 00:14:44.817053 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:44.817022 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5c58f866-ef80-42d5-aab0-0cdefbc7a158-ca-trust-extracted\") pod \"image-registry-6644876fb8-9bszd\" (UID: \"5c58f866-ef80-42d5-aab0-0cdefbc7a158\") " pod="openshift-image-registry/image-registry-6644876fb8-9bszd" Oct 31 00:14:44.817053 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:44.817056 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5c58f866-ef80-42d5-aab0-0cdefbc7a158-trusted-ca\") pod \"image-registry-6644876fb8-9bszd\" (UID: \"5c58f866-ef80-42d5-aab0-0cdefbc7a158\") " pod="openshift-image-registry/image-registry-6644876fb8-9bszd" Oct 31 00:14:44.817248 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:44.817091 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5hld\" (UniqueName: \"kubernetes.io/projected/5c58f866-ef80-42d5-aab0-0cdefbc7a158-kube-api-access-n5hld\") pod \"image-registry-6644876fb8-9bszd\" (UID: \"5c58f866-ef80-42d5-aab0-0cdefbc7a158\") " pod="openshift-image-registry/image-registry-6644876fb8-9bszd" Oct 31 00:14:44.817248 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:44.817118 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5c58f866-ef80-42d5-aab0-0cdefbc7a158-installation-pull-secrets\") pod \"image-registry-6644876fb8-9bszd\" (UID: \"5c58f866-ef80-42d5-aab0-0cdefbc7a158\") " pod="openshift-image-registry/image-registry-6644876fb8-9bszd" Oct 31 00:14:44.817248 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:44.817167 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5c58f866-ef80-42d5-aab0-0cdefbc7a158-registry-certificates\") pod \"image-registry-6644876fb8-9bszd\" (UID: \"5c58f866-ef80-42d5-aab0-0cdefbc7a158\") " pod="openshift-image-registry/image-registry-6644876fb8-9bszd" Oct 31 00:14:44.817248 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:44.817195 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5c58f866-ef80-42d5-aab0-0cdefbc7a158-bound-sa-token\") pod \"image-registry-6644876fb8-9bszd\" (UID: \"5c58f866-ef80-42d5-aab0-0cdefbc7a158\") " pod="openshift-image-registry/image-registry-6644876fb8-9bszd" Oct 31 00:14:44.817248 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:44.817220 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5c58f866-ef80-42d5-aab0-0cdefbc7a158-image-registry-private-configuration\") pod \"image-registry-6644876fb8-9bszd\" (UID: \"5c58f866-ef80-42d5-aab0-0cdefbc7a158\") " pod="openshift-image-registry/image-registry-6644876fb8-9bszd" Oct 31 00:14:44.817248 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:44.817242 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5c58f866-ef80-42d5-aab0-0cdefbc7a158-registry-tls\") pod \"image-registry-6644876fb8-9bszd\" (UID: \"5c58f866-ef80-42d5-aab0-0cdefbc7a158\") " pod="openshift-image-registry/image-registry-6644876fb8-9bszd" Oct 31 00:14:44.817546 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:44.817516 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5c58f866-ef80-42d5-aab0-0cdefbc7a158-ca-trust-extracted\") pod \"image-registry-6644876fb8-9bszd\" (UID: \"5c58f866-ef80-42d5-aab0-0cdefbc7a158\") " pod="openshift-image-registry/image-registry-6644876fb8-9bszd" Oct 31 00:14:44.818174 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:44.818071 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5c58f866-ef80-42d5-aab0-0cdefbc7a158-registry-certificates\") pod \"image-registry-6644876fb8-9bszd\" (UID: \"5c58f866-ef80-42d5-aab0-0cdefbc7a158\") " pod="openshift-image-registry/image-registry-6644876fb8-9bszd" Oct 31 00:14:44.818347 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:44.818324 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5c58f866-ef80-42d5-aab0-0cdefbc7a158-trusted-ca\") pod \"image-registry-6644876fb8-9bszd\" (UID: \"5c58f866-ef80-42d5-aab0-0cdefbc7a158\") " pod="openshift-image-registry/image-registry-6644876fb8-9bszd" Oct 31 00:14:44.819751 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:44.819724 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5c58f866-ef80-42d5-aab0-0cdefbc7a158-registry-tls\") pod \"image-registry-6644876fb8-9bszd\" (UID: \"5c58f866-ef80-42d5-aab0-0cdefbc7a158\") " pod="openshift-image-registry/image-registry-6644876fb8-9bszd" Oct 31 00:14:44.819839 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:44.819766 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5c58f866-ef80-42d5-aab0-0cdefbc7a158-installation-pull-secrets\") pod \"image-registry-6644876fb8-9bszd\" (UID: \"5c58f866-ef80-42d5-aab0-0cdefbc7a158\") " pod="openshift-image-registry/image-registry-6644876fb8-9bszd" Oct 31 00:14:44.819880 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:44.819853 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5c58f866-ef80-42d5-aab0-0cdefbc7a158-image-registry-private-configuration\") pod \"image-registry-6644876fb8-9bszd\" (UID: \"5c58f866-ef80-42d5-aab0-0cdefbc7a158\") " pod="openshift-image-registry/image-registry-6644876fb8-9bszd" Oct 31 00:14:44.825712 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:44.825695 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-nq56p" Oct 31 00:14:44.884665 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:44.884600 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5hld\" (UniqueName: \"kubernetes.io/projected/5c58f866-ef80-42d5-aab0-0cdefbc7a158-kube-api-access-n5hld\") pod \"image-registry-6644876fb8-9bszd\" (UID: \"5c58f866-ef80-42d5-aab0-0cdefbc7a158\") " pod="openshift-image-registry/image-registry-6644876fb8-9bszd" Oct 31 00:14:44.889782 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:44.889736 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5c58f866-ef80-42d5-aab0-0cdefbc7a158-bound-sa-token\") pod \"image-registry-6644876fb8-9bszd\" (UID: \"5c58f866-ef80-42d5-aab0-0cdefbc7a158\") " pod="openshift-image-registry/image-registry-6644876fb8-9bszd" Oct 31 00:14:44.912126 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:44.912087 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6644876fb8-9bszd" Oct 31 00:14:45.187247 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:45.187217 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-lrh4r"] Oct 31 00:14:45.191236 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:14:45.191210 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83d351d1_c6e7_43a6_a0fb_eaecefa8c5b2.slice/crio-a3f9f3fe924dfb4d4b266048dfe16c8b0550c3570ad9df858260e0e531a7a485 WatchSource:0}: Error finding container a3f9f3fe924dfb4d4b266048dfe16c8b0550c3570ad9df858260e0e531a7a485: Status 404 returned error can't find the container with id a3f9f3fe924dfb4d4b266048dfe16c8b0550c3570ad9df858260e0e531a7a485 Oct 31 00:14:45.238465 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:45.238425 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-x5s75"] Oct 31 00:14:45.242095 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:14:45.242059 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97d11160_2114_4fbf_95c2_fa7b4c9d2260.slice/crio-85cdbe46bdd5e24c6593111a991360dc0159e1e871dafb1bd333367618c00f41 WatchSource:0}: Error finding container 85cdbe46bdd5e24c6593111a991360dc0159e1e871dafb1bd333367618c00f41: Status 404 returned error can't find the container with id 85cdbe46bdd5e24c6593111a991360dc0159e1e871dafb1bd333367618c00f41 Oct 31 00:14:45.247980 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:45.247955 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-nq56p"] Oct 31 00:14:45.251193 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:14:45.251169 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd22409bc_1bcf_46b1_9789_6f7298566a7f.slice/crio-b9d29c14a25ecc7102c32fec2fe49a6da0882079c0b09b5e376492bf0b7ce5d6 WatchSource:0}: Error finding container b9d29c14a25ecc7102c32fec2fe49a6da0882079c0b09b5e376492bf0b7ce5d6: Status 404 returned error can't find the container with id b9d29c14a25ecc7102c32fec2fe49a6da0882079c0b09b5e376492bf0b7ce5d6 Oct 31 00:14:45.306466 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:45.306438 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6644876fb8-9bszd"] Oct 31 00:14:45.309832 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:14:45.309803 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c58f866_ef80_42d5_aab0_0cdefbc7a158.slice/crio-7e1bf6c5db5b115aaee51fc3686ab440559202deb4f763f9c005dbdf951ef51d WatchSource:0}: Error finding container 7e1bf6c5db5b115aaee51fc3686ab440559202deb4f763f9c005dbdf951ef51d: Status 404 returned error can't find the container with id 7e1bf6c5db5b115aaee51fc3686ab440559202deb4f763f9c005dbdf951ef51d Oct 31 00:14:45.541780 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:45.541741 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nq56p" event={"ID":"d22409bc-1bcf-46b1-9789-6f7298566a7f","Type":"ContainerStarted","Data":"206dfda6ba61bea0ae8c2e9df3e9e7fcff6fdff5a597201ab1bc5e465bf78742"} Oct 31 00:14:45.542215 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:45.541807 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nq56p" event={"ID":"d22409bc-1bcf-46b1-9789-6f7298566a7f","Type":"ContainerStarted","Data":"b9d29c14a25ecc7102c32fec2fe49a6da0882079c0b09b5e376492bf0b7ce5d6"} Oct 31 00:14:45.542998 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:45.542973 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-x5s75" event={"ID":"97d11160-2114-4fbf-95c2-fa7b4c9d2260","Type":"ContainerStarted","Data":"85cdbe46bdd5e24c6593111a991360dc0159e1e871dafb1bd333367618c00f41"} Oct 31 00:14:45.544689 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:45.544664 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6644876fb8-9bszd" event={"ID":"5c58f866-ef80-42d5-aab0-0cdefbc7a158","Type":"ContainerStarted","Data":"42b16fcd937e7927196e5ead0504827314311de66fcacc86823ccb671ad9d728"} Oct 31 00:14:45.544782 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:45.544707 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6644876fb8-9bszd" event={"ID":"5c58f866-ef80-42d5-aab0-0cdefbc7a158","Type":"ContainerStarted","Data":"7e1bf6c5db5b115aaee51fc3686ab440559202deb4f763f9c005dbdf951ef51d"} Oct 31 00:14:45.544782 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:45.544751 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6644876fb8-9bszd" Oct 31 00:14:45.545871 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:45.545851 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lrh4r" event={"ID":"83d351d1-c6e7-43a6-a0fb-eaecefa8c5b2","Type":"ContainerStarted","Data":"a3f9f3fe924dfb4d4b266048dfe16c8b0550c3570ad9df858260e0e531a7a485"} Oct 31 00:14:45.680731 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:45.680676 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6644876fb8-9bszd" podStartSLOduration=1.680663632 podStartE2EDuration="1.680663632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:14:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:14:45.678712328 +0000 UTC m=+52.911247774" watchObservedRunningTime="2026-10-31 00:14:45.680663632 +0000 UTC m=+52.913199078" Oct 31 00:14:47.554960 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:47.554918 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nq56p" event={"ID":"d22409bc-1bcf-46b1-9789-6f7298566a7f","Type":"ContainerStarted","Data":"1179d4a423ffddad4533645e760a17b6455870eb9a3d4b472df72ce8e81c4622"} Oct 31 00:14:48.560855 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:48.560819 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lrh4r" event={"ID":"83d351d1-c6e7-43a6-a0fb-eaecefa8c5b2","Type":"ContainerStarted","Data":"9a76140a15b2592553beea09d624628985c71ea1ae1dafa10e4aa1a9585cdf4d"} Oct 31 00:14:48.561341 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:48.560876 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lrh4r" event={"ID":"83d351d1-c6e7-43a6-a0fb-eaecefa8c5b2","Type":"ContainerStarted","Data":"4dc235fb2f8f5c89d668e8e16620b72a0d23546b76836da94050af872f82165a"} Oct 31 00:14:48.561341 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:48.560975 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-lrh4r" Oct 31 00:14:48.676930 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:48.676862 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-lrh4r" podStartSLOduration=2.228373828 podStartE2EDuration="4.676840145s" totalImagesPullingTime="2.448466317s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:14:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:14:48.676450355 +0000 UTC m=+55.908985804" watchObservedRunningTime="2026-10-31 00:14:48.676840145 +0000 UTC m=+55.909375592" Oct 31 00:14:49.490230 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:49.490204 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:14:49.566426 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:49.566387 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nq56p" event={"ID":"d22409bc-1bcf-46b1-9789-6f7298566a7f","Type":"ContainerStarted","Data":"014ed59c08def4e6647a48658b4055289fb74f1d56a1994fc256ce2c6767cdd7"} Oct 31 00:14:49.568127 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:49.568096 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-x5s75" event={"ID":"97d11160-2114-4fbf-95c2-fa7b4c9d2260","Type":"ContainerStarted","Data":"fbf6cabf18ea18e4fd8c2a29a4a6edb2829f2ce962347779e10d4c5780f70f32"} Oct 31 00:14:49.821926 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:49.821869 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-x5s75" podStartSLOduration=1.562456711 podStartE2EDuration="5.82185603s" totalImagesPullingTime="4.259399319s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:14:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:14:49.821731954 +0000 UTC m=+57.054267402" watchObservedRunningTime="2026-10-31 00:14:49.82185603 +0000 UTC m=+57.054391476" Oct 31 00:14:49.935009 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:49.934956 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-nq56p" podStartSLOduration=2.311102512 podStartE2EDuration="5.934943455s" totalImagesPullingTime="3.623840943s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:14:44 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:14:49.934156949 +0000 UTC m=+57.166692396" watchObservedRunningTime="2026-10-31 00:14:49.934943455 +0000 UTC m=+57.167478902" Oct 31 00:14:58.219178 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:58.219138 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d02ae6b5-4860-45b3-8d16-4a49531b77d4-original-pull-secret\") pod \"global-pull-secret-syncer-fwgtx\" (UID: \"d02ae6b5-4860-45b3-8d16-4a49531b77d4\") " pod="kube-system/global-pull-secret-syncer-fwgtx" Oct 31 00:14:58.219605 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:58.219196 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a077dea4-b5a3-4335-bb6a-7618e794eb8f-metrics-certs\") pod \"network-metrics-daemon-rjfn2\" (UID: \"a077dea4-b5a3-4335-bb6a-7618e794eb8f\") " pod="openshift-multus/network-metrics-daemon-rjfn2" Oct 31 00:14:58.315750 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:58.315714 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 31 00:14:58.315932 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:58.315757 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 31 00:14:58.322260 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:58.322233 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d02ae6b5-4860-45b3-8d16-4a49531b77d4-original-pull-secret\") pod \"global-pull-secret-syncer-fwgtx\" (UID: \"d02ae6b5-4860-45b3-8d16-4a49531b77d4\") " pod="kube-system/global-pull-secret-syncer-fwgtx" Oct 31 00:14:58.322401 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:58.322269 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a077dea4-b5a3-4335-bb6a-7618e794eb8f-metrics-certs\") pod \"network-metrics-daemon-rjfn2\" (UID: \"a077dea4-b5a3-4335-bb6a-7618e794eb8f\") " pod="openshift-multus/network-metrics-daemon-rjfn2" Oct 31 00:14:58.451524 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:58.451491 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-lg2hb\"" Oct 31 00:14:58.453547 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:58.453525 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fwgtx" Oct 31 00:14:58.495721 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:58.495645 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zk5sj\"" Oct 31 00:14:58.496647 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:58.496626 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rjfn2" Oct 31 00:14:58.570760 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:58.570735 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-lrh4r" Oct 31 00:14:58.578337 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:58.578310 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-nbn2h"] Oct 31 00:14:58.582430 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:58.582414 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-nbn2h" Oct 31 00:14:58.632205 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:58.632173 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 31 00:14:58.632425 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:58.632284 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 31 00:14:58.632474 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:58.632424 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-xx7tk\"" Oct 31 00:14:58.637305 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:58.637281 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-nbn2h"] Oct 31 00:14:58.724087 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:58.724046 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c61956a7-ba51-4063-bc25-aaa418168bc2-tmp\") pod \"downloads-6dd789c8b7-nbn2h\" (UID: \"c61956a7-ba51-4063-bc25-aaa418168bc2\") " pod="openshift-console/downloads-6dd789c8b7-nbn2h" Oct 31 00:14:58.724274 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:58.724119 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cspld\" (UniqueName: \"kubernetes.io/projected/c61956a7-ba51-4063-bc25-aaa418168bc2-kube-api-access-cspld\") pod \"downloads-6dd789c8b7-nbn2h\" (UID: \"c61956a7-ba51-4063-bc25-aaa418168bc2\") " pod="openshift-console/downloads-6dd789c8b7-nbn2h" Oct 31 00:14:58.818691 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:58.818659 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-fwgtx"] Oct 31 00:14:58.821720 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:14:58.821690 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd02ae6b5_4860_45b3_8d16_4a49531b77d4.slice/crio-c879df49638ffe7b92434481fc181e47422074d35dc456a77ac6926fc51595a1 WatchSource:0}: Error finding container c879df49638ffe7b92434481fc181e47422074d35dc456a77ac6926fc51595a1: Status 404 returned error can't find the container with id c879df49638ffe7b92434481fc181e47422074d35dc456a77ac6926fc51595a1 Oct 31 00:14:58.825134 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:58.825114 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c61956a7-ba51-4063-bc25-aaa418168bc2-tmp\") pod \"downloads-6dd789c8b7-nbn2h\" (UID: \"c61956a7-ba51-4063-bc25-aaa418168bc2\") " pod="openshift-console/downloads-6dd789c8b7-nbn2h" Oct 31 00:14:58.825254 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:58.825155 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cspld\" (UniqueName: \"kubernetes.io/projected/c61956a7-ba51-4063-bc25-aaa418168bc2-kube-api-access-cspld\") pod \"downloads-6dd789c8b7-nbn2h\" (UID: \"c61956a7-ba51-4063-bc25-aaa418168bc2\") " pod="openshift-console/downloads-6dd789c8b7-nbn2h" Oct 31 00:14:58.825474 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:58.825455 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c61956a7-ba51-4063-bc25-aaa418168bc2-tmp\") pod \"downloads-6dd789c8b7-nbn2h\" (UID: \"c61956a7-ba51-4063-bc25-aaa418168bc2\") " pod="openshift-console/downloads-6dd789c8b7-nbn2h" Oct 31 00:14:58.867353 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:58.867324 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-rjfn2"] Oct 31 00:14:58.870262 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:14:58.870238 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda077dea4_b5a3_4335_bb6a_7618e794eb8f.slice/crio-a4c6c3886ed113a59ea0d1ca28218a244394c08b02880bb9e841d2c55dc730ef WatchSource:0}: Error finding container a4c6c3886ed113a59ea0d1ca28218a244394c08b02880bb9e841d2c55dc730ef: Status 404 returned error can't find the container with id a4c6c3886ed113a59ea0d1ca28218a244394c08b02880bb9e841d2c55dc730ef Oct 31 00:14:58.885788 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:58.885754 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cspld\" (UniqueName: \"kubernetes.io/projected/c61956a7-ba51-4063-bc25-aaa418168bc2-kube-api-access-cspld\") pod \"downloads-6dd789c8b7-nbn2h\" (UID: \"c61956a7-ba51-4063-bc25-aaa418168bc2\") " pod="openshift-console/downloads-6dd789c8b7-nbn2h" Oct 31 00:14:58.893512 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:58.893492 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-nbn2h" Oct 31 00:14:59.257323 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:59.257240 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-nbn2h"] Oct 31 00:14:59.260935 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:14:59.260906 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc61956a7_ba51_4063_bc25_aaa418168bc2.slice/crio-b1592c8b3c26564beed25f9e4abca5a54557d95546cab7063ac25aa7da5d0251 WatchSource:0}: Error finding container b1592c8b3c26564beed25f9e4abca5a54557d95546cab7063ac25aa7da5d0251: Status 404 returned error can't find the container with id b1592c8b3c26564beed25f9e4abca5a54557d95546cab7063ac25aa7da5d0251 Oct 31 00:14:59.598992 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:59.598956 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-nbn2h" event={"ID":"c61956a7-ba51-4063-bc25-aaa418168bc2","Type":"ContainerStarted","Data":"b1592c8b3c26564beed25f9e4abca5a54557d95546cab7063ac25aa7da5d0251"} Oct 31 00:14:59.600121 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:59.600086 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rjfn2" event={"ID":"a077dea4-b5a3-4335-bb6a-7618e794eb8f","Type":"ContainerStarted","Data":"a4c6c3886ed113a59ea0d1ca28218a244394c08b02880bb9e841d2c55dc730ef"} Oct 31 00:14:59.601123 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:14:59.601105 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-fwgtx" event={"ID":"d02ae6b5-4860-45b3-8d16-4a49531b77d4","Type":"ContainerStarted","Data":"c879df49638ffe7b92434481fc181e47422074d35dc456a77ac6926fc51595a1"} Oct 31 00:15:01.612839 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:01.612798 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rjfn2" event={"ID":"a077dea4-b5a3-4335-bb6a-7618e794eb8f","Type":"ContainerStarted","Data":"4122121183de614047eaa02ff018b7a6bacfbc58b71dd63ff0970aaafcb68e46"} Oct 31 00:15:01.613315 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:01.612857 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rjfn2" event={"ID":"a077dea4-b5a3-4335-bb6a-7618e794eb8f","Type":"ContainerStarted","Data":"066676c1bc7842be071c34ad40fe1640de8e40b8e5ea11e9db1bad0d4059e592"} Oct 31 00:15:01.732845 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:01.732787 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-rjfn2" podStartSLOduration=66.948811485 podStartE2EDuration="1m8.732771039s" totalImagesPullingTime="1.783959554s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:13:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:15:01.731799878 +0000 UTC m=+68.964335326" watchObservedRunningTime="2026-10-31 00:15:01.732771039 +0000 UTC m=+68.965306490" Oct 31 00:15:02.513047 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:02.513016 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-ml8n5" Oct 31 00:15:04.087766 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:04.087730 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-xd6zl"] Oct 31 00:15:04.090643 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:04.090619 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xd6zl" Oct 31 00:15:04.190968 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:04.190933 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 31 00:15:04.191156 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:04.190953 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 00:15:04.191156 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:04.190997 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 31 00:15:04.191156 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:04.191045 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-mkn2t\"" Oct 31 00:15:04.191325 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:04.191312 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 31 00:15:04.191612 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:04.191589 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 31 00:15:04.240268 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:04.240235 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-xd6zl"] Oct 31 00:15:04.268043 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:04.268003 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwf5l\" (UniqueName: \"kubernetes.io/projected/652defc0-301f-4679-8e49-6729381259dd-kube-api-access-jwf5l\") pod \"openshift-state-metrics-7bd7489957-xd6zl\" (UID: \"652defc0-301f-4679-8e49-6729381259dd\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xd6zl" Oct 31 00:15:04.268294 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:04.268064 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/652defc0-301f-4679-8e49-6729381259dd-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-xd6zl\" (UID: \"652defc0-301f-4679-8e49-6729381259dd\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xd6zl" Oct 31 00:15:04.268294 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:04.268187 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/652defc0-301f-4679-8e49-6729381259dd-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-xd6zl\" (UID: \"652defc0-301f-4679-8e49-6729381259dd\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xd6zl" Oct 31 00:15:04.268294 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:04.268237 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/652defc0-301f-4679-8e49-6729381259dd-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-xd6zl\" (UID: \"652defc0-301f-4679-8e49-6729381259dd\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xd6zl" Oct 31 00:15:04.274003 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:04.273966 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-dbcsc"] Oct 31 00:15:04.276310 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:04.276287 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-dbcsc" Oct 31 00:15:04.328929 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:04.328896 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-rsf6d"] Oct 31 00:15:04.330927 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:04.330907 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rsf6d" Oct 31 00:15:04.336056 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:04.336035 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 31 00:15:04.345043 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:04.344983 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-ks82c\"" Oct 31 00:15:04.345043 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:04.345010 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 31 00:15:04.345203 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:04.345058 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 31 00:15:04.369383 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:04.369331 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/652defc0-301f-4679-8e49-6729381259dd-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-xd6zl\" (UID: \"652defc0-301f-4679-8e49-6729381259dd\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xd6zl" Oct 31 00:15:04.369568 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:04.369408 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/652defc0-301f-4679-8e49-6729381259dd-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-xd6zl\" (UID: \"652defc0-301f-4679-8e49-6729381259dd\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xd6zl" Oct 31 00:15:04.369568 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:04.369444 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwf5l\" (UniqueName: \"kubernetes.io/projected/652defc0-301f-4679-8e49-6729381259dd-kube-api-access-jwf5l\") pod \"openshift-state-metrics-7bd7489957-xd6zl\" (UID: \"652defc0-301f-4679-8e49-6729381259dd\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xd6zl" Oct 31 00:15:04.369568 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:04.369474 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/652defc0-301f-4679-8e49-6729381259dd-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-xd6zl\" (UID: \"652defc0-301f-4679-8e49-6729381259dd\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xd6zl" Oct 31 00:15:04.370244 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:04.370214 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/652defc0-301f-4679-8e49-6729381259dd-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-xd6zl\" (UID: \"652defc0-301f-4679-8e49-6729381259dd\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xd6zl" Oct 31 00:15:04.372510 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:04.372483 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/652defc0-301f-4679-8e49-6729381259dd-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-xd6zl\" (UID: \"652defc0-301f-4679-8e49-6729381259dd\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xd6zl" Oct 31 00:15:04.372592 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:04.372548 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/652defc0-301f-4679-8e49-6729381259dd-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-xd6zl\" (UID: \"652defc0-301f-4679-8e49-6729381259dd\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xd6zl" Oct 31 00:15:04.394096 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:04.394061 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-rsf6d"] Oct 31 00:15:04.435181 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:04.435155 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 31 00:15:04.437733 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:04.437704 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwf5l\" (UniqueName: \"kubernetes.io/projected/652defc0-301f-4679-8e49-6729381259dd-kube-api-access-jwf5l\") pod \"openshift-state-metrics-7bd7489957-xd6zl\" (UID: \"652defc0-301f-4679-8e49-6729381259dd\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xd6zl" Oct 31 00:15:04.445087 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:04.445058 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 31 00:15:04.445489 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:04.445457 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 31 00:15:04.445489 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:04.445461 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-gf7wd\"" Oct 31 00:15:04.470837 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:04.470707 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ea9a5160-4696-4b75-8a10-0fbc54b2ad04-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-rsf6d\" (UID: \"ea9a5160-4696-4b75-8a10-0fbc54b2ad04\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rsf6d" Oct 31 00:15:04.470837 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:04.470764 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqw94\" (UniqueName: \"kubernetes.io/projected/1b88d3b9-589f-4f4e-b8a6-872f4fb5536b-kube-api-access-dqw94\") pod \"node-exporter-dbcsc\" (UID: \"1b88d3b9-589f-4f4e-b8a6-872f4fb5536b\") " pod="openshift-monitoring/node-exporter-dbcsc" Oct 31 00:15:04.470837 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:04.470802 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1b88d3b9-589f-4f4e-b8a6-872f4fb5536b-node-exporter-tls\") pod \"node-exporter-dbcsc\" (UID: \"1b88d3b9-589f-4f4e-b8a6-872f4fb5536b\") " pod="openshift-monitoring/node-exporter-dbcsc" Oct 31 00:15:04.471141 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:04.470871 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/ea9a5160-4696-4b75-8a10-0fbc54b2ad04-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-rsf6d\" (UID: \"ea9a5160-4696-4b75-8a10-0fbc54b2ad04\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rsf6d" Oct 31 00:15:04.471141 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:04.470965 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1b88d3b9-589f-4f4e-b8a6-872f4fb5536b-node-exporter-textfile\") pod \"node-exporter-dbcsc\" (UID: \"1b88d3b9-589f-4f4e-b8a6-872f4fb5536b\") " pod="openshift-monitoring/node-exporter-dbcsc" Oct 31 00:15:04.471141 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:04.471044 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1b88d3b9-589f-4f4e-b8a6-872f4fb5536b-node-exporter-wtmp\") pod \"node-exporter-dbcsc\" (UID: \"1b88d3b9-589f-4f4e-b8a6-872f4fb5536b\") " pod="openshift-monitoring/node-exporter-dbcsc" Oct 31 00:15:04.471141 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:04.471081 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1b88d3b9-589f-4f4e-b8a6-872f4fb5536b-node-exporter-accelerators-collector-config\") pod \"node-exporter-dbcsc\" (UID: \"1b88d3b9-589f-4f4e-b8a6-872f4fb5536b\") " pod="openshift-monitoring/node-exporter-dbcsc" Oct 31 00:15:04.471141 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:04.471114 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1b88d3b9-589f-4f4e-b8a6-872f4fb5536b-sys\") pod \"node-exporter-dbcsc\" (UID: \"1b88d3b9-589f-4f4e-b8a6-872f4fb5536b\") " pod="openshift-monitoring/node-exporter-dbcsc" Oct 31 00:15:04.471414 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:04.471150 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1b88d3b9-589f-4f4e-b8a6-872f4fb5536b-root\") pod \"node-exporter-dbcsc\" (UID: \"1b88d3b9-589f-4f4e-b8a6-872f4fb5536b\") " pod="openshift-monitoring/node-exporter-dbcsc" Oct 31 00:15:04.471414 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:04.471180 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ea9a5160-4696-4b75-8a10-0fbc54b2ad04-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-rsf6d\" (UID: \"ea9a5160-4696-4b75-8a10-0fbc54b2ad04\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rsf6d" Oct 31 00:15:04.471414 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:04.471206 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ea9a5160-4696-4b75-8a10-0fbc54b2ad04-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-rsf6d\" (UID: \"ea9a5160-4696-4b75-8a10-0fbc54b2ad04\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rsf6d" Oct 31 00:15:04.471414 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:04.471224 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1b88d3b9-589f-4f4e-b8a6-872f4fb5536b-metrics-client-ca\") pod \"node-exporter-dbcsc\" (UID: \"1b88d3b9-589f-4f4e-b8a6-872f4fb5536b\") " pod="openshift-monitoring/node-exporter-dbcsc" Oct 31 00:15:04.471414 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:04.471245 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1b88d3b9-589f-4f4e-b8a6-872f4fb5536b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dbcsc\" (UID: \"1b88d3b9-589f-4f4e-b8a6-872f4fb5536b\") " pod="openshift-monitoring/node-exporter-dbcsc" Oct 31 00:15:04.471414 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:04.471268 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/ea9a5160-4696-4b75-8a10-0fbc54b2ad04-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-rsf6d\" (UID: \"ea9a5160-4696-4b75-8a10-0fbc54b2ad04\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rsf6d" Oct 31 00:15:04.471414 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:04.471290 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7hn4\" (UniqueName: \"kubernetes.io/projected/ea9a5160-4696-4b75-8a10-0fbc54b2ad04-kube-api-access-v7hn4\") pod \"kube-state-metrics-5dc8d7d89b-rsf6d\" (UID: \"ea9a5160-4696-4b75-8a10-0fbc54b2ad04\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rsf6d" Oct 31 00:15:04.571990 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:04.571946 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1b88d3b9-589f-4f4e-b8a6-872f4fb5536b-root\") pod \"node-exporter-dbcsc\" (UID: \"1b88d3b9-589f-4f4e-b8a6-872f4fb5536b\") " pod="openshift-monitoring/node-exporter-dbcsc" Oct 31 00:15:04.571990 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:04.571991 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ea9a5160-4696-4b75-8a10-0fbc54b2ad04-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-rsf6d\" (UID: \"ea9a5160-4696-4b75-8a10-0fbc54b2ad04\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rsf6d" Oct 31 00:15:04.572212 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:04.572016 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ea9a5160-4696-4b75-8a10-0fbc54b2ad04-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-rsf6d\" (UID: \"ea9a5160-4696-4b75-8a10-0fbc54b2ad04\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rsf6d" Oct 31 00:15:04.572212 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:04.572051 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1b88d3b9-589f-4f4e-b8a6-872f4fb5536b-metrics-client-ca\") pod \"node-exporter-dbcsc\" (UID: \"1b88d3b9-589f-4f4e-b8a6-872f4fb5536b\") " pod="openshift-monitoring/node-exporter-dbcsc" Oct 31 00:15:04.572212 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:04.572072 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1b88d3b9-589f-4f4e-b8a6-872f4fb5536b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dbcsc\" (UID: \"1b88d3b9-589f-4f4e-b8a6-872f4fb5536b\") " pod="openshift-monitoring/node-exporter-dbcsc" Oct 31 00:15:04.572212 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:04.572093 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/ea9a5160-4696-4b75-8a10-0fbc54b2ad04-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-rsf6d\" (UID: \"ea9a5160-4696-4b75-8a10-0fbc54b2ad04\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rsf6d" Oct 31 00:15:04.572212 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:04.572119 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7hn4\" (UniqueName: \"kubernetes.io/projected/ea9a5160-4696-4b75-8a10-0fbc54b2ad04-kube-api-access-v7hn4\") pod \"kube-state-metrics-5dc8d7d89b-rsf6d\" (UID: \"ea9a5160-4696-4b75-8a10-0fbc54b2ad04\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rsf6d" Oct 31 00:15:04.572212 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:04.572152 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ea9a5160-4696-4b75-8a10-0fbc54b2ad04-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-rsf6d\" (UID: \"ea9a5160-4696-4b75-8a10-0fbc54b2ad04\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rsf6d" Oct 31 00:15:04.572212 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:04.572174 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqw94\" (UniqueName: \"kubernetes.io/projected/1b88d3b9-589f-4f4e-b8a6-872f4fb5536b-kube-api-access-dqw94\") pod \"node-exporter-dbcsc\" (UID: \"1b88d3b9-589f-4f4e-b8a6-872f4fb5536b\") " pod="openshift-monitoring/node-exporter-dbcsc" Oct 31 00:15:04.572212 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:04.572193 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1b88d3b9-589f-4f4e-b8a6-872f4fb5536b-node-exporter-tls\") pod \"node-exporter-dbcsc\" (UID: \"1b88d3b9-589f-4f4e-b8a6-872f4fb5536b\") " pod="openshift-monitoring/node-exporter-dbcsc" Oct 31 00:15:04.572212 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:04.572208 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/ea9a5160-4696-4b75-8a10-0fbc54b2ad04-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-rsf6d\" (UID: \"ea9a5160-4696-4b75-8a10-0fbc54b2ad04\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rsf6d" Oct 31 00:15:04.572622 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:04.572225 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1b88d3b9-589f-4f4e-b8a6-872f4fb5536b-node-exporter-textfile\") pod \"node-exporter-dbcsc\" (UID: \"1b88d3b9-589f-4f4e-b8a6-872f4fb5536b\") " pod="openshift-monitoring/node-exporter-dbcsc" Oct 31 00:15:04.572622 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:04.572243 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1b88d3b9-589f-4f4e-b8a6-872f4fb5536b-node-exporter-wtmp\") pod \"node-exporter-dbcsc\" (UID: \"1b88d3b9-589f-4f4e-b8a6-872f4fb5536b\") " pod="openshift-monitoring/node-exporter-dbcsc" Oct 31 00:15:04.572622 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:04.572268 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1b88d3b9-589f-4f4e-b8a6-872f4fb5536b-node-exporter-accelerators-collector-config\") pod \"node-exporter-dbcsc\" (UID: \"1b88d3b9-589f-4f4e-b8a6-872f4fb5536b\") " pod="openshift-monitoring/node-exporter-dbcsc" Oct 31 00:15:04.572622 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:04.572287 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1b88d3b9-589f-4f4e-b8a6-872f4fb5536b-sys\") pod \"node-exporter-dbcsc\" (UID: \"1b88d3b9-589f-4f4e-b8a6-872f4fb5536b\") " pod="openshift-monitoring/node-exporter-dbcsc" Oct 31 00:15:04.572622 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:04.572341 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1b88d3b9-589f-4f4e-b8a6-872f4fb5536b-sys\") pod \"node-exporter-dbcsc\" (UID: \"1b88d3b9-589f-4f4e-b8a6-872f4fb5536b\") " pod="openshift-monitoring/node-exporter-dbcsc" Oct 31 00:15:04.575798 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:04.572905 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1b88d3b9-589f-4f4e-b8a6-872f4fb5536b-metrics-client-ca\") pod \"node-exporter-dbcsc\" (UID: \"1b88d3b9-589f-4f4e-b8a6-872f4fb5536b\") " pod="openshift-monitoring/node-exporter-dbcsc" Oct 31 00:15:04.575798 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:04.573141 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ea9a5160-4696-4b75-8a10-0fbc54b2ad04-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-rsf6d\" (UID: \"ea9a5160-4696-4b75-8a10-0fbc54b2ad04\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rsf6d" Oct 31 00:15:04.575798 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:04.572089 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1b88d3b9-589f-4f4e-b8a6-872f4fb5536b-root\") pod \"node-exporter-dbcsc\" (UID: \"1b88d3b9-589f-4f4e-b8a6-872f4fb5536b\") " pod="openshift-monitoring/node-exporter-dbcsc" Oct 31 00:15:04.575798 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:04.574055 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/ea9a5160-4696-4b75-8a10-0fbc54b2ad04-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-rsf6d\" (UID: \"ea9a5160-4696-4b75-8a10-0fbc54b2ad04\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rsf6d" Oct 31 00:15:04.575798 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:04.574202 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1b88d3b9-589f-4f4e-b8a6-872f4fb5536b-node-exporter-wtmp\") pod \"node-exporter-dbcsc\" (UID: \"1b88d3b9-589f-4f4e-b8a6-872f4fb5536b\") " pod="openshift-monitoring/node-exporter-dbcsc" Oct 31 00:15:04.575798 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:04.574341 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1b88d3b9-589f-4f4e-b8a6-872f4fb5536b-node-exporter-textfile\") pod \"node-exporter-dbcsc\" (UID: \"1b88d3b9-589f-4f4e-b8a6-872f4fb5536b\") " pod="openshift-monitoring/node-exporter-dbcsc" Oct 31 00:15:04.575798 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:04.574677 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1b88d3b9-589f-4f4e-b8a6-872f4fb5536b-node-exporter-accelerators-collector-config\") pod \"node-exporter-dbcsc\" (UID: \"1b88d3b9-589f-4f4e-b8a6-872f4fb5536b\") " pod="openshift-monitoring/node-exporter-dbcsc" Oct 31 00:15:04.576329 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:04.576081 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/ea9a5160-4696-4b75-8a10-0fbc54b2ad04-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-rsf6d\" (UID: \"ea9a5160-4696-4b75-8a10-0fbc54b2ad04\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rsf6d" Oct 31 00:15:04.576421 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:04.575116 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ea9a5160-4696-4b75-8a10-0fbc54b2ad04-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-rsf6d\" (UID: \"ea9a5160-4696-4b75-8a10-0fbc54b2ad04\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rsf6d" Oct 31 00:15:04.577546 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:04.577517 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1b88d3b9-589f-4f4e-b8a6-872f4fb5536b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dbcsc\" (UID: \"1b88d3b9-589f-4f4e-b8a6-872f4fb5536b\") " pod="openshift-monitoring/node-exporter-dbcsc" Oct 31 00:15:04.577655 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:04.577570 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ea9a5160-4696-4b75-8a10-0fbc54b2ad04-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-rsf6d\" (UID: \"ea9a5160-4696-4b75-8a10-0fbc54b2ad04\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rsf6d" Oct 31 00:15:04.579390 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:04.579346 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1b88d3b9-589f-4f4e-b8a6-872f4fb5536b-node-exporter-tls\") pod \"node-exporter-dbcsc\" (UID: \"1b88d3b9-589f-4f4e-b8a6-872f4fb5536b\") " pod="openshift-monitoring/node-exporter-dbcsc" Oct 31 00:15:04.632465 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:04.632389 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqw94\" (UniqueName: \"kubernetes.io/projected/1b88d3b9-589f-4f4e-b8a6-872f4fb5536b-kube-api-access-dqw94\") pod \"node-exporter-dbcsc\" (UID: \"1b88d3b9-589f-4f4e-b8a6-872f4fb5536b\") " pod="openshift-monitoring/node-exporter-dbcsc" Oct 31 00:15:04.632778 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:04.632745 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7hn4\" (UniqueName: \"kubernetes.io/projected/ea9a5160-4696-4b75-8a10-0fbc54b2ad04-kube-api-access-v7hn4\") pod \"kube-state-metrics-5dc8d7d89b-rsf6d\" (UID: \"ea9a5160-4696-4b75-8a10-0fbc54b2ad04\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rsf6d" Oct 31 00:15:04.643519 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:04.643475 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rsf6d" Oct 31 00:15:04.707117 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:04.706917 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xd6zl" Oct 31 00:15:04.890067 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:04.889988 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-dbcsc" Oct 31 00:15:04.902601 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:15:04.902557 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b88d3b9_589f_4f4e_b8a6_872f4fb5536b.slice/crio-526acb8408e46da20b2ba7f24e9ac667d7b7e021118d73e3d193ec380f36efb7 WatchSource:0}: Error finding container 526acb8408e46da20b2ba7f24e9ac667d7b7e021118d73e3d193ec380f36efb7: Status 404 returned error can't find the container with id 526acb8408e46da20b2ba7f24e9ac667d7b7e021118d73e3d193ec380f36efb7 Oct 31 00:15:05.106580 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:05.106540 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-rsf6d"] Oct 31 00:15:05.111425 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:15:05.111389 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea9a5160_4696_4b75_8a10_0fbc54b2ad04.slice/crio-3c171721a06bad3a5280a8be2538f726a9af19dbd55856c3aaa9873dd468bd8f WatchSource:0}: Error finding container 3c171721a06bad3a5280a8be2538f726a9af19dbd55856c3aaa9873dd468bd8f: Status 404 returned error can't find the container with id 3c171721a06bad3a5280a8be2538f726a9af19dbd55856c3aaa9873dd468bd8f Oct 31 00:15:05.141564 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:05.141402 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-xd6zl"] Oct 31 00:15:05.144740 ip-10-0-137-56 kubenswrapper[2581]: W1031 00:15:05.144691 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod652defc0_301f_4679_8e49_6729381259dd.slice/crio-d7f607419a7dbf03da1d4459c5376eb54185fa2270d3616edd508177ed7540eb WatchSource:0}: Error finding container d7f607419a7dbf03da1d4459c5376eb54185fa2270d3616edd508177ed7540eb: Status 404 returned error can't find the container with id d7f607419a7dbf03da1d4459c5376eb54185fa2270d3616edd508177ed7540eb Oct 31 00:15:05.630541 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:05.630500 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dbcsc" event={"ID":"1b88d3b9-589f-4f4e-b8a6-872f4fb5536b","Type":"ContainerStarted","Data":"526acb8408e46da20b2ba7f24e9ac667d7b7e021118d73e3d193ec380f36efb7"} Oct 31 00:15:05.632958 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:05.632921 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xd6zl" event={"ID":"652defc0-301f-4679-8e49-6729381259dd","Type":"ContainerStarted","Data":"f7d1a7da447240063158dc3f590b53ce1a95d1e1a5acaa0173402212a1e2aac8"} Oct 31 00:15:05.633112 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:05.632988 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xd6zl" event={"ID":"652defc0-301f-4679-8e49-6729381259dd","Type":"ContainerStarted","Data":"1c8e1aaeb393a504af2698764f044908dee8d246b3e0d504cafd8448cba92dc5"} Oct 31 00:15:05.633112 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:05.633018 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xd6zl" event={"ID":"652defc0-301f-4679-8e49-6729381259dd","Type":"ContainerStarted","Data":"d7f607419a7dbf03da1d4459c5376eb54185fa2270d3616edd508177ed7540eb"} Oct 31 00:15:05.634278 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:05.634252 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rsf6d" event={"ID":"ea9a5160-4696-4b75-8a10-0fbc54b2ad04","Type":"ContainerStarted","Data":"3c171721a06bad3a5280a8be2538f726a9af19dbd55856c3aaa9873dd468bd8f"} Oct 31 00:15:06.555874 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:06.555843 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6644876fb8-9bszd" Oct 31 00:15:06.561984 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:06.560087 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 00:15:06.565444 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:06.564160 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:15:06.584807 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:06.584775 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/26aba8fc-e2da-430f-b0d8-6ee45468441d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"26aba8fc-e2da-430f-b0d8-6ee45468441d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:15:06.584951 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:06.584812 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/26aba8fc-e2da-430f-b0d8-6ee45468441d-web-config\") pod \"alertmanager-main-0\" (UID: \"26aba8fc-e2da-430f-b0d8-6ee45468441d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:15:06.584951 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:06.584836 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/26aba8fc-e2da-430f-b0d8-6ee45468441d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"26aba8fc-e2da-430f-b0d8-6ee45468441d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:15:06.584951 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:06.584854 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/26aba8fc-e2da-430f-b0d8-6ee45468441d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"26aba8fc-e2da-430f-b0d8-6ee45468441d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:15:06.584951 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:06.584870 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26aba8fc-e2da-430f-b0d8-6ee45468441d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"26aba8fc-e2da-430f-b0d8-6ee45468441d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:15:06.584951 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:06.584888 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/26aba8fc-e2da-430f-b0d8-6ee45468441d-config-volume\") pod \"alertmanager-main-0\" (UID: \"26aba8fc-e2da-430f-b0d8-6ee45468441d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:15:06.584951 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:06.584916 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/26aba8fc-e2da-430f-b0d8-6ee45468441d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"26aba8fc-e2da-430f-b0d8-6ee45468441d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:15:06.584951 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:06.584932 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/26aba8fc-e2da-430f-b0d8-6ee45468441d-config-out\") pod \"alertmanager-main-0\" (UID: \"26aba8fc-e2da-430f-b0d8-6ee45468441d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:15:06.584951 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:06.584951 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/26aba8fc-e2da-430f-b0d8-6ee45468441d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"26aba8fc-e2da-430f-b0d8-6ee45468441d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:15:06.585306 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:06.584995 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4v246\" (UniqueName: \"kubernetes.io/projected/26aba8fc-e2da-430f-b0d8-6ee45468441d-kube-api-access-4v246\") pod \"alertmanager-main-0\" (UID: \"26aba8fc-e2da-430f-b0d8-6ee45468441d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:15:06.585306 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:06.585028 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/26aba8fc-e2da-430f-b0d8-6ee45468441d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"26aba8fc-e2da-430f-b0d8-6ee45468441d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:15:06.585306 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:06.585046 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/26aba8fc-e2da-430f-b0d8-6ee45468441d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"26aba8fc-e2da-430f-b0d8-6ee45468441d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:15:06.585306 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:06.585069 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/26aba8fc-e2da-430f-b0d8-6ee45468441d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"26aba8fc-e2da-430f-b0d8-6ee45468441d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:15:06.608091 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:06.608038 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 00:15:06.655621 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:06.653950 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-ltd79\"" Oct 31 00:15:06.655621 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:06.655041 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 31 00:15:06.655621 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:06.655122 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 31 00:15:06.655621 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:06.655232 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 31 00:15:06.655621 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:06.655300 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 31 00:15:06.655621 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:06.655361 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 31 00:15:06.655621 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:06.655484 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 31 00:15:06.655621 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:06.655550 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 31 00:15:06.686631 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:06.685977 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/26aba8fc-e2da-430f-b0d8-6ee45468441d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"26aba8fc-e2da-430f-b0d8-6ee45468441d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:15:06.686631 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:06.686022 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/26aba8fc-e2da-430f-b0d8-6ee45468441d-web-config\") pod \"alertmanager-main-0\" (UID: \"26aba8fc-e2da-430f-b0d8-6ee45468441d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:15:06.686631 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:06.686052 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/26aba8fc-e2da-430f-b0d8-6ee45468441d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"26aba8fc-e2da-430f-b0d8-6ee45468441d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:15:06.686631 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:06.686081 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/26aba8fc-e2da-430f-b0d8-6ee45468441d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"26aba8fc-e2da-430f-b0d8-6ee45468441d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:15:06.686631 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:06.686106 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26aba8fc-e2da-430f-b0d8-6ee45468441d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"26aba8fc-e2da-430f-b0d8-6ee45468441d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:15:06.686631 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:06.686129 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/26aba8fc-e2da-430f-b0d8-6ee45468441d-config-volume\") pod \"alertmanager-main-0\" (UID: \"26aba8fc-e2da-430f-b0d8-6ee45468441d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:15:06.686631 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:06.686158 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/26aba8fc-e2da-430f-b0d8-6ee45468441d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"26aba8fc-e2da-430f-b0d8-6ee45468441d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:15:06.686631 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:06.686186 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/26aba8fc-e2da-430f-b0d8-6ee45468441d-config-out\") pod \"alertmanager-main-0\" (UID: \"26aba8fc-e2da-430f-b0d8-6ee45468441d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:15:06.686631 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:06.686215 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/26aba8fc-e2da-430f-b0d8-6ee45468441d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"26aba8fc-e2da-430f-b0d8-6ee45468441d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:15:06.686631 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:06.686272 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4v246\" (UniqueName: \"kubernetes.io/projected/26aba8fc-e2da-430f-b0d8-6ee45468441d-kube-api-access-4v246\") pod \"alertmanager-main-0\" (UID: \"26aba8fc-e2da-430f-b0d8-6ee45468441d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:15:06.686631 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:06.686307 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/26aba8fc-e2da-430f-b0d8-6ee45468441d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"26aba8fc-e2da-430f-b0d8-6ee45468441d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:15:06.686631 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:06.686341 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/26aba8fc-e2da-430f-b0d8-6ee45468441d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"26aba8fc-e2da-430f-b0d8-6ee45468441d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:15:06.686631 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:06.686396 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/26aba8fc-e2da-430f-b0d8-6ee45468441d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"26aba8fc-e2da-430f-b0d8-6ee45468441d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:15:06.689450 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:06.689286 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/26aba8fc-e2da-430f-b0d8-6ee45468441d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"26aba8fc-e2da-430f-b0d8-6ee45468441d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:15:06.690072 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:06.690045 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/26aba8fc-e2da-430f-b0d8-6ee45468441d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"26aba8fc-e2da-430f-b0d8-6ee45468441d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:15:06.693248 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:06.693192 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/26aba8fc-e2da-430f-b0d8-6ee45468441d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"26aba8fc-e2da-430f-b0d8-6ee45468441d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:15:06.694140 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:06.693602 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/26aba8fc-e2da-430f-b0d8-6ee45468441d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"26aba8fc-e2da-430f-b0d8-6ee45468441d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:15:06.694140 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:06.693930 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/26aba8fc-e2da-430f-b0d8-6ee45468441d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"26aba8fc-e2da-430f-b0d8-6ee45468441d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:15:06.694668 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:06.694643 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/26aba8fc-e2da-430f-b0d8-6ee45468441d-config-volume\") pod \"alertmanager-main-0\" (UID: \"26aba8fc-e2da-430f-b0d8-6ee45468441d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:15:06.694891 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:06.694847 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/26aba8fc-e2da-430f-b0d8-6ee45468441d-web-config\") pod \"alertmanager-main-0\" (UID: \"26aba8fc-e2da-430f-b0d8-6ee45468441d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:15:06.696454 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:06.696323 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/26aba8fc-e2da-430f-b0d8-6ee45468441d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"26aba8fc-e2da-430f-b0d8-6ee45468441d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:15:06.697836 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:06.697810 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/26aba8fc-e2da-430f-b0d8-6ee45468441d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"26aba8fc-e2da-430f-b0d8-6ee45468441d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:15:06.697934 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:06.697819 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/26aba8fc-e2da-430f-b0d8-6ee45468441d-config-out\") pod \"alertmanager-main-0\" (UID: \"26aba8fc-e2da-430f-b0d8-6ee45468441d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:15:06.749461 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:06.749301 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 31 00:15:06.749461 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:06.749451 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 31 00:15:06.765574 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:06.765509 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26aba8fc-e2da-430f-b0d8-6ee45468441d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"26aba8fc-e2da-430f-b0d8-6ee45468441d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:15:06.771013 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:06.770978 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4v246\" (UniqueName: \"kubernetes.io/projected/26aba8fc-e2da-430f-b0d8-6ee45468441d-kube-api-access-4v246\") pod \"alertmanager-main-0\" (UID: \"26aba8fc-e2da-430f-b0d8-6ee45468441d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:15:06.772767 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:06.772695 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/26aba8fc-e2da-430f-b0d8-6ee45468441d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"26aba8fc-e2da-430f-b0d8-6ee45468441d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:15:06.892267 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:06.892190 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:15:07.361339 ip-10-0-137-56 kubenswrapper[2581]: I1031 00:15:07.361307 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 00:15:07.455603 ip-10-0-137-56 systemd[1]: Stopping Kubernetes Kubelet... Oct 31 00:15:07.469406 ip-10-0-137-56 systemd[1]: kubelet.service: Deactivated successfully. Oct 31 00:15:07.469585 ip-10-0-137-56 systemd[1]: Stopped Kubernetes Kubelet. Oct 31 00:15:07.469636 ip-10-0-137-56 systemd[1]: kubelet.service: Consumed 3.712s CPU time. Oct 31 00:15:07.482034 ip-10-0-137-56 systemd[1]: Starting Kubernetes Kubelet... Oct 31 00:15:07.561800 ip-10-0-137-56 kubenswrapper[5222]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 00:15:07.561800 ip-10-0-137-56 kubenswrapper[5222]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 31 00:15:07.561800 ip-10-0-137-56 kubenswrapper[5222]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 00:15:07.561800 ip-10-0-137-56 kubenswrapper[5222]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 00:15:07.563993 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.563904 5222 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:15:07.563993 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.563992 5222 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:15:07.563993 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.563997 5222 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:15:07.564083 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.564001 5222 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:15:07.564083 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.564005 5222 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:15:07.564083 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.564007 5222 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:15:07.564083 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.564010 5222 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:15:07.564083 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.564013 5222 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:15:07.564083 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.564015 5222 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:15:07.564083 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.564018 5222 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:15:07.564083 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.564020 5222 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:15:07.564083 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.564023 5222 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:15:07.564083 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.564026 5222 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:15:07.564083 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.564029 5222 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:15:07.564083 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.564031 5222 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:15:07.564083 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.564034 5222 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:15:07.564083 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.564036 5222 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:15:07.564083 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.564039 5222 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:15:07.564083 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.564042 5222 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:15:07.564083 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.564044 5222 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:15:07.564083 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.564047 5222 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:15:07.564083 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.564066 5222 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:15:07.564083 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.564069 5222 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:15:07.564619 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.564072 5222 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:15:07.564619 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.564075 5222 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:15:07.564619 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.564079 5222 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:15:07.564619 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.564084 5222 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:15:07.564619 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.564087 5222 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:15:07.564619 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.564090 5222 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:15:07.564619 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.564093 5222 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:15:07.564619 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.564096 5222 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:15:07.564619 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.564098 5222 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:15:07.564619 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.564101 5222 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:15:07.564619 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.564104 5222 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:15:07.564619 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.564107 5222 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:15:07.564619 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.564110 5222 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:15:07.564619 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.564112 5222 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:15:07.564619 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.564115 5222 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:15:07.564619 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.564118 5222 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:15:07.564619 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.564122 5222 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:15:07.564619 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.564126 5222 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:15:07.564619 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.564130 5222 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:15:07.565076 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.564132 5222 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:15:07.565076 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.564135 5222 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:15:07.565076 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.564138 5222 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:15:07.565076 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.564141 5222 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:15:07.565076 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.564144 5222 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:15:07.565076 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.564147 5222 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:15:07.565076 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.564149 5222 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:15:07.565076 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.564152 5222 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:15:07.565076 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.564156 5222 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:15:07.565076 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.564159 5222 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:15:07.565076 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.564162 5222 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:15:07.565076 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.564164 5222 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:15:07.565076 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.564167 5222 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:15:07.565076 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.564170 5222 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:15:07.565076 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.564173 5222 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:15:07.565076 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.564176 5222 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:15:07.565076 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.564178 5222 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:15:07.565076 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.564181 5222 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:15:07.565076 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.564183 5222 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:15:07.565076 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.564186 5222 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:15:07.565581 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.564189 5222 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:15:07.565581 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.564192 5222 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:15:07.565581 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.564194 5222 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:15:07.565581 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.564197 5222 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:15:07.565581 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.564199 5222 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:15:07.565581 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.564202 5222 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:15:07.565581 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.564204 5222 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:15:07.565581 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.564207 5222 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:15:07.565581 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.564210 5222 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:15:07.565581 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.564212 5222 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:15:07.565581 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.564215 5222 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:15:07.565581 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.564218 5222 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:15:07.565581 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.564220 5222 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:15:07.565581 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.564223 5222 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:15:07.565581 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.564225 5222 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:15:07.565581 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.564228 5222 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:15:07.565581 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.564230 5222 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:15:07.565581 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.564233 5222 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:15:07.565581 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.564236 5222 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:15:07.565581 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.564238 5222 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:15:07.566095 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.564241 5222 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:15:07.566095 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.564243 5222 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:15:07.566095 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.564246 5222 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:15:07.566095 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.564248 5222 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:15:07.566095 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.564251 5222 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:15:07.566095 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.564253 5222 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:15:07.566095 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.564257 5222 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:15:07.566095 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.564260 5222 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:15:07.566095 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.564262 5222 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:15:07.566095 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.564265 5222 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:15:07.566095 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.564267 5222 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:15:07.566095 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.564269 5222 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:15:07.566095 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.564272 5222 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:15:07.566095 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.564274 5222 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:15:07.566095 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.564277 5222 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:15:07.566095 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.564280 5222 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:15:07.566095 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.564282 5222 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:15:07.566095 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.564285 5222 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:15:07.566095 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.564287 5222 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:15:07.566095 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.564290 5222 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:15:07.566642 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.564292 5222 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:15:07.566642 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.564295 5222 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:15:07.566642 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.564297 5222 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:15:07.566642 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.564302 5222 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:15:07.566642 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.565126 5222 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:15:07.566642 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.565133 5222 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:15:07.566642 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.565136 5222 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:15:07.566642 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.565140 5222 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:15:07.566642 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.565142 5222 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:15:07.566642 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.565146 5222 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:15:07.566642 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.565149 5222 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:15:07.566642 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.565152 5222 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:15:07.566642 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.565155 5222 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:15:07.566642 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.565157 5222 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:15:07.566642 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.565160 5222 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:15:07.566642 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.565162 5222 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:15:07.566642 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.565166 5222 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:15:07.566642 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.565168 5222 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:15:07.566642 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.565171 5222 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:15:07.566642 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.565175 5222 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:15:07.567139 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.565179 5222 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:15:07.567139 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.565182 5222 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:15:07.567139 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.565185 5222 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:15:07.567139 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.565188 5222 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:15:07.567139 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.565191 5222 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:15:07.567139 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.565193 5222 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:15:07.567139 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.565196 5222 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:15:07.567139 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.565198 5222 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:15:07.567139 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.565201 5222 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:15:07.567139 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.565203 5222 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:15:07.567139 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.565206 5222 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:15:07.567139 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.565209 5222 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:15:07.567139 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.565211 5222 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:15:07.567139 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.565214 5222 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:15:07.567139 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.565216 5222 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:15:07.567139 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.565219 5222 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:15:07.567139 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.565223 5222 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:15:07.567139 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.565226 5222 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:15:07.567139 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.565228 5222 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:15:07.567702 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.565231 5222 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:15:07.567702 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.565233 5222 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:15:07.567702 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.565236 5222 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:15:07.567702 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.565238 5222 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:15:07.567702 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.565241 5222 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:15:07.567702 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.565244 5222 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:15:07.567702 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.565246 5222 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:15:07.567702 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.565249 5222 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:15:07.567702 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.565251 5222 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:15:07.567702 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.565254 5222 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:15:07.567702 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.565256 5222 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:15:07.567702 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.565259 5222 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:15:07.567702 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.565262 5222 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:15:07.567702 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.565264 5222 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:15:07.567702 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.565267 5222 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:15:07.567702 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.565269 5222 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:15:07.567702 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.565272 5222 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:15:07.567702 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.565275 5222 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:15:07.567702 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.565277 5222 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:15:07.567702 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.565279 5222 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:15:07.568189 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.565282 5222 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:15:07.568189 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.565284 5222 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:15:07.568189 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.565287 5222 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:15:07.568189 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.565290 5222 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:15:07.568189 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.565292 5222 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:15:07.568189 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.565295 5222 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:15:07.568189 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.565297 5222 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:15:07.568189 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.565300 5222 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:15:07.568189 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.565303 5222 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:15:07.568189 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.565305 5222 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:15:07.568189 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.565309 5222 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:15:07.568189 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.565311 5222 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:15:07.568189 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.565314 5222 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:15:07.568189 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.565318 5222 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:15:07.568189 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.565322 5222 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:15:07.568189 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.565325 5222 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:15:07.568189 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.565328 5222 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:15:07.568189 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.565331 5222 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:15:07.568189 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.565334 5222 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:15:07.568673 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.565336 5222 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:15:07.568673 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.565339 5222 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:15:07.568673 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.565341 5222 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:15:07.568673 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.565344 5222 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:15:07.568673 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.565347 5222 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:15:07.568673 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.565350 5222 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:15:07.568673 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.565352 5222 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:15:07.568673 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.565355 5222 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:15:07.568673 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.565357 5222 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:15:07.568673 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.565360 5222 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:15:07.568673 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.565362 5222 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:15:07.568673 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.565380 5222 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:15:07.568673 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.565383 5222 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:15:07.568673 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.565385 5222 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:15:07.568673 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.565388 5222 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:15:07.568673 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.565390 5222 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:15:07.568673 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.565393 5222 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:15:07.568673 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.565396 5222 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:15:07.568673 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.565398 5222 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:15:07.568673 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.565401 5222 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:15:07.569188 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.565404 5222 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:15:07.569188 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.565407 5222 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:15:07.569188 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.565409 5222 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:15:07.569188 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.565412 5222 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:15:07.569188 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.565415 5222 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:15:07.569188 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.565418 5222 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:15:07.569188 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.565421 5222 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:15:07.569188 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.565424 5222 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:15:07.569188 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.565427 5222 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:15:07.569188 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.565430 5222 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:15:07.569188 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.565432 5222 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:15:07.569188 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.565435 5222 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:15:07.569188 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.565958 5222 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:15:07.569188 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.565964 5222 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:15:07.569188 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.565967 5222 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:15:07.569188 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.565970 5222 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:15:07.569188 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.565974 5222 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:15:07.569188 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.565976 5222 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:15:07.569188 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.565979 5222 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:15:07.569188 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.565981 5222 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:15:07.569685 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.565984 5222 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:15:07.569685 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.565987 5222 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:15:07.569685 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.565989 5222 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:15:07.569685 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.565992 5222 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:15:07.569685 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.565994 5222 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:15:07.569685 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.565997 5222 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:15:07.569685 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.565999 5222 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:15:07.569685 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.566002 5222 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:15:07.569685 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.566005 5222 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:15:07.569685 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.566008 5222 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:15:07.569685 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.566013 5222 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:15:07.569685 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.566018 5222 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:15:07.569685 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.566022 5222 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:15:07.569685 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.566027 5222 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:15:07.569685 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.566031 5222 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:15:07.569685 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.566034 5222 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:15:07.569685 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.566041 5222 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:15:07.569685 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.566045 5222 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:15:07.569685 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.566048 5222 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:15:07.569685 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.566050 5222 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:15:07.570218 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.566052 5222 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:15:07.570218 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.566055 5222 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:15:07.570218 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.566057 5222 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:15:07.570218 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.566060 5222 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:15:07.570218 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.566063 5222 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:15:07.570218 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.566065 5222 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:15:07.570218 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.566068 5222 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:15:07.570218 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.566070 5222 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:15:07.570218 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.566073 5222 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:15:07.570218 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.566076 5222 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:15:07.570218 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.566079 5222 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:15:07.570218 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.566081 5222 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:15:07.570218 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.566084 5222 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:15:07.570218 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.566086 5222 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:15:07.570218 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.566089 5222 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:15:07.570218 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.566097 5222 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:15:07.570218 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.566102 5222 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:15:07.570218 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.566106 5222 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:15:07.570218 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.566111 5222 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:15:07.570218 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.566115 5222 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:15:07.571107 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.566119 5222 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:15:07.571107 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.566121 5222 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:15:07.571107 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.566124 5222 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:15:07.571107 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.566127 5222 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:15:07.571107 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.566129 5222 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:15:07.571107 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.566132 5222 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:15:07.571107 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.566134 5222 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:15:07.571107 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.566137 5222 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:15:07.571107 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.566140 5222 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:15:07.571107 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.566149 5222 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:15:07.571107 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.566151 5222 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:15:07.571107 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.566154 5222 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:15:07.571107 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.566158 5222 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:15:07.571107 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.566160 5222 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:15:07.571107 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.566163 5222 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:15:07.571107 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.566165 5222 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:15:07.571107 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.566168 5222 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:15:07.571107 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.566171 5222 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:15:07.571107 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.566173 5222 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:15:07.571107 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.566176 5222 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:15:07.571998 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.566179 5222 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:15:07.571998 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.566181 5222 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:15:07.571998 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.566184 5222 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:15:07.571998 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.566186 5222 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:15:07.571998 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.566188 5222 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:15:07.571998 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.566191 5222 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:15:07.571998 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.566193 5222 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:15:07.571998 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.566199 5222 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:15:07.571998 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.566203 5222 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:15:07.571998 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.566208 5222 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:15:07.571998 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.566211 5222 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:15:07.571998 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.566213 5222 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:15:07.571998 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.566216 5222 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:15:07.571998 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.566218 5222 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:15:07.571998 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.566221 5222 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:15:07.571998 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.566224 5222 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:15:07.571998 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.566226 5222 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:15:07.571998 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.566228 5222 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:15:07.571998 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.566231 5222 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:15:07.572826 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.566233 5222 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:15:07.572826 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.566236 5222 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:15:07.572826 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.566238 5222 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:15:07.572826 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.566246 5222 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:15:07.572826 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.566249 5222 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:15:07.572826 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.566251 5222 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:15:07.572826 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.566254 5222 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:15:07.572826 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.566256 5222 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:15:07.572826 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.566259 5222 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:15:07.572826 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.566261 5222 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:15:07.572826 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.566264 5222 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:15:07.572826 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.566269 5222 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:15:07.572826 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.566272 5222 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:15:07.572826 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.566275 5222 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:15:07.572826 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.566278 5222 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:15:07.572826 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.566281 5222 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:15:07.572826 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.566284 5222 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:15:07.572826 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.566286 5222 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:15:07.572826 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.566289 5222 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:15:07.572826 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566407 5222 flags.go:64] FLAG: --address="0.0.0.0" Oct 31 00:15:07.573691 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566425 5222 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 31 00:15:07.573691 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566431 5222 flags.go:64] FLAG: --anonymous-auth="true" Oct 31 00:15:07.573691 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566436 5222 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 31 00:15:07.573691 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566441 5222 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 31 00:15:07.573691 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566444 5222 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 31 00:15:07.573691 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566449 5222 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 31 00:15:07.573691 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566453 5222 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 31 00:15:07.573691 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566457 5222 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 31 00:15:07.573691 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566461 5222 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 31 00:15:07.573691 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566465 5222 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 31 00:15:07.573691 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566468 5222 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 31 00:15:07.573691 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566471 5222 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 31 00:15:07.573691 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566474 5222 flags.go:64] FLAG: --cgroup-root="" Oct 31 00:15:07.573691 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566477 5222 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 31 00:15:07.573691 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566480 5222 flags.go:64] FLAG: --client-ca-file="" Oct 31 00:15:07.573691 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566483 5222 flags.go:64] FLAG: --cloud-provider="external" Oct 31 00:15:07.573691 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566492 5222 flags.go:64] FLAG: --cluster-dns="[]" Oct 31 00:15:07.573691 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566499 5222 flags.go:64] FLAG: --cluster-domain="" Oct 31 00:15:07.573691 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566502 5222 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 31 00:15:07.573691 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566505 5222 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 31 00:15:07.573691 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566508 5222 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 31 00:15:07.573691 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566512 5222 flags.go:64] FLAG: --container-log-max-files="5" Oct 31 00:15:07.573691 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566517 5222 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 31 00:15:07.573691 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566521 5222 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 31 00:15:07.574314 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566526 5222 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 31 00:15:07.574314 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566532 5222 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 31 00:15:07.574314 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566536 5222 flags.go:64] FLAG: --contention-profiling="false" Oct 31 00:15:07.574314 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566541 5222 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 31 00:15:07.574314 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566545 5222 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 31 00:15:07.574314 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566548 5222 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 31 00:15:07.574314 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566551 5222 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 31 00:15:07.574314 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566555 5222 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 31 00:15:07.574314 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566560 5222 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 31 00:15:07.574314 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566563 5222 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 31 00:15:07.574314 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566566 5222 flags.go:64] FLAG: --enable-load-reader="false" Oct 31 00:15:07.574314 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566569 5222 flags.go:64] FLAG: --enable-server="true" Oct 31 00:15:07.574314 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566572 5222 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 31 00:15:07.574314 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566579 5222 flags.go:64] FLAG: --event-burst="100" Oct 31 00:15:07.574314 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566583 5222 flags.go:64] FLAG: --event-qps="50" Oct 31 00:15:07.574314 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566585 5222 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 31 00:15:07.574314 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566590 5222 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 31 00:15:07.574314 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566593 5222 flags.go:64] FLAG: --eviction-hard="" Oct 31 00:15:07.574314 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566597 5222 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 31 00:15:07.574314 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566601 5222 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 31 00:15:07.574314 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566606 5222 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 31 00:15:07.574314 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566611 5222 flags.go:64] FLAG: --eviction-soft="" Oct 31 00:15:07.574314 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566617 5222 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 31 00:15:07.574314 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566621 5222 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 31 00:15:07.574314 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566630 5222 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 31 00:15:07.574987 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566635 5222 flags.go:64] FLAG: --experimental-mounter-path="" Oct 31 00:15:07.574987 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566638 5222 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 31 00:15:07.574987 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566642 5222 flags.go:64] FLAG: --fail-swap-on="true" Oct 31 00:15:07.574987 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566644 5222 flags.go:64] FLAG: --feature-gates="" Oct 31 00:15:07.574987 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566649 5222 flags.go:64] FLAG: --file-check-frequency="20s" Oct 31 00:15:07.574987 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566652 5222 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 31 00:15:07.574987 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566655 5222 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 31 00:15:07.574987 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566659 5222 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 31 00:15:07.574987 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566662 5222 flags.go:64] FLAG: --healthz-port="10248" Oct 31 00:15:07.574987 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566665 5222 flags.go:64] FLAG: --help="false" Oct 31 00:15:07.574987 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566668 5222 flags.go:64] FLAG: --hostname-override="ip-10-0-137-56.us-west-2.compute.internal" Oct 31 00:15:07.574987 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566671 5222 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 31 00:15:07.574987 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566674 5222 flags.go:64] FLAG: --http-check-frequency="20s" Oct 31 00:15:07.574987 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566677 5222 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 31 00:15:07.574987 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566681 5222 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 31 00:15:07.574987 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566684 5222 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 31 00:15:07.574987 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566687 5222 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 31 00:15:07.574987 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566691 5222 flags.go:64] FLAG: --image-service-endpoint="" Oct 31 00:15:07.574987 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566694 5222 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 31 00:15:07.574987 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566697 5222 flags.go:64] FLAG: --kube-api-burst="100" Oct 31 00:15:07.574987 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566700 5222 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 31 00:15:07.574987 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566703 5222 flags.go:64] FLAG: --kube-api-qps="50" Oct 31 00:15:07.574987 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566706 5222 flags.go:64] FLAG: --kube-reserved="" Oct 31 00:15:07.574987 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566711 5222 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 31 00:15:07.575603 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566714 5222 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 31 00:15:07.575603 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566717 5222 flags.go:64] FLAG: --kubelet-cgroups="" Oct 31 00:15:07.575603 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566719 5222 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 31 00:15:07.575603 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566722 5222 flags.go:64] FLAG: --lock-file="" Oct 31 00:15:07.575603 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566725 5222 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 31 00:15:07.575603 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566728 5222 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 31 00:15:07.575603 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566731 5222 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 31 00:15:07.575603 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566737 5222 flags.go:64] FLAG: --log-json-split-stream="false" Oct 31 00:15:07.575603 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566740 5222 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 31 00:15:07.575603 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566743 5222 flags.go:64] FLAG: --log-text-split-stream="false" Oct 31 00:15:07.575603 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566746 5222 flags.go:64] FLAG: --logging-format="text" Oct 31 00:15:07.575603 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566749 5222 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 31 00:15:07.575603 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566752 5222 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 31 00:15:07.575603 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566755 5222 flags.go:64] FLAG: --manifest-url="" Oct 31 00:15:07.575603 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566758 5222 flags.go:64] FLAG: --manifest-url-header="" Oct 31 00:15:07.575603 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566762 5222 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 31 00:15:07.575603 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566765 5222 flags.go:64] FLAG: --max-open-files="1000000" Oct 31 00:15:07.575603 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566769 5222 flags.go:64] FLAG: --max-pods="110" Oct 31 00:15:07.575603 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566772 5222 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 31 00:15:07.575603 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566775 5222 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 31 00:15:07.575603 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566778 5222 flags.go:64] FLAG: --memory-manager-policy="None" Oct 31 00:15:07.575603 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566781 5222 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 31 00:15:07.575603 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566784 5222 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 31 00:15:07.575603 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566787 5222 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 31 00:15:07.575603 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566791 5222 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 31 00:15:07.576274 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566799 5222 flags.go:64] FLAG: --node-status-max-images="50" Oct 31 00:15:07.576274 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566802 5222 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 31 00:15:07.576274 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566805 5222 flags.go:64] FLAG: --oom-score-adj="-999" Oct 31 00:15:07.576274 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566808 5222 flags.go:64] FLAG: --pod-cidr="" Oct 31 00:15:07.576274 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566811 5222 flags.go:64] FLAG: --pod-manifest-path="" Oct 31 00:15:07.576274 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566814 5222 flags.go:64] FLAG: --pod-max-pids="-1" Oct 31 00:15:07.576274 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566818 5222 flags.go:64] FLAG: --pods-per-core="0" Oct 31 00:15:07.576274 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566821 5222 flags.go:64] FLAG: --port="10250" Oct 31 00:15:07.576274 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566824 5222 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 31 00:15:07.576274 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566827 5222 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-024d89d5d0a1f47d6" Oct 31 00:15:07.576274 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566830 5222 flags.go:64] FLAG: --qos-reserved="" Oct 31 00:15:07.576274 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566832 5222 flags.go:64] FLAG: --read-only-port="10255" Oct 31 00:15:07.576274 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566835 5222 flags.go:64] FLAG: --register-node="true" Oct 31 00:15:07.576274 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566838 5222 flags.go:64] FLAG: --register-with-taints="" Oct 31 00:15:07.576274 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566842 5222 flags.go:64] FLAG: --registry-burst="10" Oct 31 00:15:07.576274 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566845 5222 flags.go:64] FLAG: --registry-qps="5" Oct 31 00:15:07.576274 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566847 5222 flags.go:64] FLAG: --reserved-cpus="" Oct 31 00:15:07.576274 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566851 5222 flags.go:64] FLAG: --reserved-memory="" Oct 31 00:15:07.576274 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566855 5222 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 31 00:15:07.576274 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566858 5222 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 31 00:15:07.576274 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566861 5222 flags.go:64] FLAG: --rotate-certificates="false" Oct 31 00:15:07.576274 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566864 5222 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 31 00:15:07.576274 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566867 5222 flags.go:64] FLAG: --runonce="false" Oct 31 00:15:07.576274 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566870 5222 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 31 00:15:07.576274 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566873 5222 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 31 00:15:07.576274 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566876 5222 flags.go:64] FLAG: --seccomp-default="false" Oct 31 00:15:07.576274 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566879 5222 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 31 00:15:07.576962 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566882 5222 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 31 00:15:07.576962 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566885 5222 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 31 00:15:07.576962 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566888 5222 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 31 00:15:07.576962 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566891 5222 flags.go:64] FLAG: --storage-driver-password="root" Oct 31 00:15:07.576962 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566894 5222 flags.go:64] FLAG: --storage-driver-secure="false" Oct 31 00:15:07.576962 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566897 5222 flags.go:64] FLAG: --storage-driver-table="stats" Oct 31 00:15:07.576962 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566900 5222 flags.go:64] FLAG: --storage-driver-user="root" Oct 31 00:15:07.576962 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566902 5222 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 31 00:15:07.576962 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566905 5222 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 31 00:15:07.576962 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566908 5222 flags.go:64] FLAG: --system-cgroups="" Oct 31 00:15:07.576962 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566911 5222 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 31 00:15:07.576962 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566917 5222 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 31 00:15:07.576962 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566920 5222 flags.go:64] FLAG: --tls-cert-file="" Oct 31 00:15:07.576962 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566923 5222 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 31 00:15:07.576962 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566928 5222 flags.go:64] FLAG: --tls-min-version="" Oct 31 00:15:07.576962 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566931 5222 flags.go:64] FLAG: --tls-private-key-file="" Oct 31 00:15:07.576962 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566934 5222 flags.go:64] FLAG: --topology-manager-policy="none" Oct 31 00:15:07.576962 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566937 5222 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 31 00:15:07.576962 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566940 5222 flags.go:64] FLAG: --topology-manager-scope="container" Oct 31 00:15:07.576962 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566942 5222 flags.go:64] FLAG: --v="2" Oct 31 00:15:07.576962 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566947 5222 flags.go:64] FLAG: --version="false" Oct 31 00:15:07.576962 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566951 5222 flags.go:64] FLAG: --vmodule="" Oct 31 00:15:07.576962 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566956 5222 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 31 00:15:07.576962 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.566959 5222 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 31 00:15:07.576962 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567112 5222 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:15:07.577732 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567116 5222 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:15:07.577732 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567119 5222 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:15:07.577732 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567122 5222 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:15:07.577732 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567124 5222 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:15:07.577732 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567127 5222 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:15:07.577732 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567129 5222 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:15:07.577732 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567132 5222 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:15:07.577732 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567134 5222 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:15:07.577732 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567136 5222 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:15:07.577732 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567139 5222 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:15:07.577732 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567141 5222 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:15:07.577732 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567144 5222 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:15:07.577732 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567147 5222 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:15:07.577732 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567149 5222 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:15:07.577732 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567152 5222 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:15:07.577732 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567156 5222 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:15:07.577732 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567160 5222 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:15:07.577732 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567163 5222 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:15:07.577732 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567166 5222 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:15:07.578236 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567169 5222 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:15:07.578236 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567173 5222 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:15:07.578236 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567176 5222 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:15:07.578236 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567178 5222 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:15:07.578236 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567181 5222 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:15:07.578236 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567184 5222 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:15:07.578236 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567186 5222 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:15:07.578236 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567189 5222 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:15:07.578236 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567191 5222 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:15:07.578236 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567194 5222 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:15:07.578236 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567196 5222 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:15:07.578236 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567199 5222 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:15:07.578236 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567203 5222 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:15:07.578236 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567206 5222 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:15:07.578236 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567209 5222 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:15:07.578236 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567211 5222 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:15:07.578236 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567214 5222 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:15:07.578236 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567216 5222 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:15:07.578236 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567219 5222 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:15:07.578236 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567221 5222 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:15:07.578747 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567224 5222 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:15:07.578747 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567227 5222 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:15:07.578747 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567231 5222 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:15:07.578747 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567234 5222 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:15:07.578747 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567239 5222 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:15:07.578747 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567243 5222 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:15:07.578747 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567247 5222 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:15:07.578747 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567251 5222 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:15:07.578747 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567254 5222 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:15:07.578747 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567257 5222 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:15:07.578747 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567260 5222 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:15:07.578747 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567264 5222 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:15:07.578747 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567267 5222 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:15:07.578747 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567270 5222 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:15:07.578747 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567273 5222 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:15:07.578747 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567275 5222 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:15:07.578747 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567278 5222 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:15:07.578747 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567281 5222 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:15:07.578747 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567283 5222 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:15:07.578747 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567286 5222 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:15:07.579345 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567288 5222 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:15:07.579345 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567291 5222 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:15:07.579345 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567293 5222 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:15:07.579345 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567296 5222 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:15:07.579345 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567299 5222 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:15:07.579345 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567302 5222 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:15:07.579345 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567305 5222 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:15:07.579345 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567307 5222 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:15:07.579345 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567311 5222 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:15:07.579345 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567315 5222 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:15:07.579345 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567320 5222 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:15:07.579345 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567324 5222 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:15:07.579345 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567328 5222 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:15:07.579345 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567332 5222 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:15:07.579345 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567335 5222 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:15:07.579345 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567338 5222 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:15:07.579345 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567340 5222 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:15:07.579345 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567343 5222 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:15:07.579345 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567346 5222 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:15:07.579345 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567348 5222 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:15:07.580074 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567352 5222 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:15:07.580074 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567356 5222 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:15:07.580074 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567360 5222 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:15:07.580074 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567364 5222 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:15:07.580074 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567382 5222 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:15:07.580074 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567387 5222 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:15:07.580074 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567389 5222 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:15:07.580074 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567392 5222 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:15:07.580074 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567395 5222 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:15:07.580074 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567398 5222 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:15:07.580074 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567400 5222 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:15:07.580074 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567403 5222 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:15:07.580074 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567405 5222 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:15:07.580074 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567408 5222 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:15:07.580074 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567411 5222 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:15:07.580074 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567413 5222 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:15:07.580074 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567416 5222 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:15:07.580074 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567418 5222 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:15:07.580074 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567422 5222 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:15:07.580575 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567425 5222 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:15:07.580575 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567427 5222 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:15:07.580575 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567430 5222 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:15:07.580575 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567432 5222 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:15:07.580575 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567435 5222 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:15:07.580575 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567437 5222 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:15:07.580575 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567440 5222 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:15:07.580575 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.567468 5222 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 00:15:07.580575 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567509 5222 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:15:07.580575 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567513 5222 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:15:07.580575 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567517 5222 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:15:07.580575 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567519 5222 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:15:07.580575 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567522 5222 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:15:07.580575 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567524 5222 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:15:07.580575 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567527 5222 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:15:07.580575 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567530 5222 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:15:07.580575 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567534 5222 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:15:07.580989 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567536 5222 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:15:07.580989 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567539 5222 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:15:07.580989 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567544 5222 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:15:07.580989 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567547 5222 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:15:07.580989 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567550 5222 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:15:07.580989 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567552 5222 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:15:07.580989 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567555 5222 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:15:07.580989 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567557 5222 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:15:07.580989 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567560 5222 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:15:07.580989 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567563 5222 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:15:07.580989 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567565 5222 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:15:07.580989 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567568 5222 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:15:07.580989 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567570 5222 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:15:07.580989 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567573 5222 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:15:07.580989 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567575 5222 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:15:07.580989 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567579 5222 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:15:07.580989 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567582 5222 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:15:07.580989 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567585 5222 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:15:07.580989 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567587 5222 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:15:07.580989 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567590 5222 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:15:07.581647 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567592 5222 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:15:07.581647 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567595 5222 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:15:07.581647 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567597 5222 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:15:07.581647 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567600 5222 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:15:07.581647 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567603 5222 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:15:07.581647 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567605 5222 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:15:07.581647 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567608 5222 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:15:07.581647 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567610 5222 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:15:07.581647 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567613 5222 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:15:07.581647 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567615 5222 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:15:07.581647 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567617 5222 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:15:07.581647 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567622 5222 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:15:07.581647 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567624 5222 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:15:07.581647 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567627 5222 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:15:07.581647 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567632 5222 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:15:07.581647 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567637 5222 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:15:07.581647 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567640 5222 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:15:07.581647 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567643 5222 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:15:07.581647 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567645 5222 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:15:07.582217 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567648 5222 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:15:07.582217 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567650 5222 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:15:07.582217 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567654 5222 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:15:07.582217 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567657 5222 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:15:07.582217 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567659 5222 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:15:07.582217 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567662 5222 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:15:07.582217 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567664 5222 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:15:07.582217 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567667 5222 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:15:07.582217 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567670 5222 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:15:07.582217 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567672 5222 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:15:07.582217 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567675 5222 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:15:07.582217 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567678 5222 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:15:07.582217 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567680 5222 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:15:07.582217 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567683 5222 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:15:07.582217 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567685 5222 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:15:07.582217 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567688 5222 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:15:07.582217 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567690 5222 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:15:07.582217 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567692 5222 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:15:07.582217 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567695 5222 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:15:07.582815 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567697 5222 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:15:07.582815 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567700 5222 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:15:07.582815 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567703 5222 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:15:07.582815 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567705 5222 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:15:07.582815 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567708 5222 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:15:07.582815 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567711 5222 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:15:07.582815 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567714 5222 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:15:07.582815 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567717 5222 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:15:07.582815 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567722 5222 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:15:07.582815 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567724 5222 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:15:07.582815 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567727 5222 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:15:07.582815 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567729 5222 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:15:07.582815 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567732 5222 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:15:07.582815 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567734 5222 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:15:07.582815 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567737 5222 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:15:07.582815 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567739 5222 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:15:07.582815 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567742 5222 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:15:07.582815 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567744 5222 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:15:07.582815 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567748 5222 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:15:07.583381 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567751 5222 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:15:07.583381 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567754 5222 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:15:07.583381 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567757 5222 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:15:07.583381 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567760 5222 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:15:07.583381 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567763 5222 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:15:07.583381 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567765 5222 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:15:07.583381 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567768 5222 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:15:07.583381 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567772 5222 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:15:07.583381 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567774 5222 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:15:07.583381 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567776 5222 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:15:07.583381 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567779 5222 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:15:07.583381 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567781 5222 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:15:07.583381 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567784 5222 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:15:07.583381 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567786 5222 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:15:07.583381 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567789 5222 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:15:07.583381 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567791 5222 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:15:07.583381 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567794 5222 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:15:07.583381 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567796 5222 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:15:07.583381 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567800 5222 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:15:07.583381 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.567803 5222 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:15:07.583874 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.573283 5222 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 31 00:15:07.583874 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.573308 5222 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 31 00:15:07.583874 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.573336 5222 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 31 00:15:07.583874 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.573343 5222 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 31 00:15:07.583874 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.573558 5222 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:15:07.583874 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.573567 5222 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:15:07.583874 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.573573 5222 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:15:07.583874 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.573579 5222 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:15:07.583874 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.573584 5222 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:15:07.583874 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.573589 5222 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:15:07.583874 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.573594 5222 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:15:07.583874 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.573598 5222 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:15:07.583874 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.573603 5222 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:15:07.583874 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.573607 5222 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:15:07.583874 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.573611 5222 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:15:07.583874 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.573616 5222 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:15:07.583874 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.573626 5222 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:15:07.583874 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.573633 5222 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:15:07.583874 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.573695 5222 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:15:07.583874 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.573701 5222 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:15:07.584436 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.573705 5222 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:15:07.584436 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.573710 5222 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:15:07.584436 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.573715 5222 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:15:07.584436 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.573720 5222 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:15:07.584436 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.573725 5222 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:15:07.584436 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.573730 5222 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:15:07.584436 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.573735 5222 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:15:07.584436 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.573740 5222 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:15:07.584436 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.573745 5222 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:15:07.584436 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.573749 5222 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:15:07.584436 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.573759 5222 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:15:07.584436 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.573764 5222 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:15:07.584436 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.573769 5222 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:15:07.584436 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.573773 5222 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:15:07.584436 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.573777 5222 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:15:07.584436 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.573781 5222 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:15:07.584436 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.573785 5222 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:15:07.584436 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.573790 5222 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:15:07.584436 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.573794 5222 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:15:07.584436 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.573798 5222 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:15:07.584910 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.573800 5222 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:15:07.584910 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.573803 5222 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:15:07.584910 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.573806 5222 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:15:07.584910 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.573830 5222 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:15:07.584910 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.573834 5222 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:15:07.584910 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.573937 5222 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:15:07.584910 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.573941 5222 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:15:07.584910 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.573945 5222 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:15:07.584910 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.573948 5222 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:15:07.584910 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.573951 5222 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:15:07.584910 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.573953 5222 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:15:07.584910 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.573957 5222 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:15:07.584910 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.573960 5222 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:15:07.584910 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.573962 5222 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:15:07.584910 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.573965 5222 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:15:07.584910 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.573968 5222 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:15:07.584910 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.573971 5222 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:15:07.584910 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.573973 5222 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:15:07.584910 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.573975 5222 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:15:07.584910 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.573978 5222 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:15:07.585412 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.573981 5222 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:15:07.585412 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.573983 5222 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:15:07.585412 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.573986 5222 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:15:07.585412 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.573988 5222 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:15:07.585412 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.573991 5222 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:15:07.585412 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.573993 5222 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:15:07.585412 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.573997 5222 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:15:07.585412 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574000 5222 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:15:07.585412 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574003 5222 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:15:07.585412 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574005 5222 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:15:07.585412 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574008 5222 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:15:07.585412 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574011 5222 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:15:07.585412 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574014 5222 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:15:07.585412 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574017 5222 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:15:07.585412 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574019 5222 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:15:07.585412 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574022 5222 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:15:07.585412 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574025 5222 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:15:07.585412 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574027 5222 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:15:07.585412 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574030 5222 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:15:07.585950 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574033 5222 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:15:07.585950 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574035 5222 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:15:07.585950 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574038 5222 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:15:07.585950 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574041 5222 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:15:07.585950 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574044 5222 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:15:07.585950 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574046 5222 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:15:07.585950 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574049 5222 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:15:07.585950 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574052 5222 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:15:07.585950 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574054 5222 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:15:07.585950 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574057 5222 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:15:07.585950 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574059 5222 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:15:07.585950 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574062 5222 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:15:07.585950 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574065 5222 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:15:07.585950 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574068 5222 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:15:07.585950 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574070 5222 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:15:07.585950 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574073 5222 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:15:07.585950 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574075 5222 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:15:07.585950 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574078 5222 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:15:07.585950 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574080 5222 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:15:07.585950 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574083 5222 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:15:07.586519 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574089 5222 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:15:07.586519 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574092 5222 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:15:07.586519 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574096 5222 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:15:07.586519 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574099 5222 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:15:07.586519 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574102 5222 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:15:07.586519 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574105 5222 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:15:07.586519 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574108 5222 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:15:07.586519 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574111 5222 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:15:07.586519 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574113 5222 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:15:07.586519 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574116 5222 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:15:07.586519 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574119 5222 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:15:07.586519 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.574159 5222 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 00:15:07.586519 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574311 5222 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:15:07.586519 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574317 5222 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:15:07.586519 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574320 5222 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:15:07.586519 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574323 5222 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:15:07.586941 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574326 5222 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:15:07.586941 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574330 5222 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:15:07.586941 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574333 5222 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:15:07.586941 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574336 5222 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:15:07.586941 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574338 5222 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:15:07.586941 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574341 5222 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:15:07.586941 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574343 5222 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:15:07.586941 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574347 5222 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:15:07.586941 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574352 5222 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:15:07.586941 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574355 5222 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:15:07.586941 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574358 5222 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:15:07.586941 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574361 5222 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:15:07.586941 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574363 5222 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:15:07.586941 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574386 5222 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:15:07.586941 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574389 5222 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:15:07.586941 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574392 5222 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:15:07.586941 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574394 5222 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:15:07.586941 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574397 5222 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:15:07.586941 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574400 5222 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:15:07.587424 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574403 5222 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:15:07.587424 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574407 5222 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:15:07.587424 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574409 5222 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:15:07.587424 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574412 5222 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:15:07.587424 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574414 5222 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:15:07.587424 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574417 5222 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:15:07.587424 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574420 5222 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:15:07.587424 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574422 5222 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:15:07.587424 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574425 5222 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:15:07.587424 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574428 5222 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:15:07.587424 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574430 5222 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:15:07.587424 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574433 5222 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:15:07.587424 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574435 5222 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:15:07.587424 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574438 5222 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:15:07.587424 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574440 5222 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:15:07.587424 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574443 5222 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:15:07.587424 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574446 5222 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:15:07.587424 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574449 5222 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:15:07.587424 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574451 5222 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:15:07.587424 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574454 5222 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:15:07.587944 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574456 5222 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:15:07.587944 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574458 5222 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:15:07.587944 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574461 5222 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:15:07.587944 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574463 5222 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:15:07.587944 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574465 5222 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:15:07.587944 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574468 5222 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:15:07.587944 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574470 5222 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:15:07.587944 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574473 5222 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:15:07.587944 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574475 5222 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:15:07.587944 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574477 5222 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:15:07.587944 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574480 5222 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:15:07.587944 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574482 5222 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:15:07.587944 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574485 5222 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:15:07.587944 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574488 5222 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:15:07.587944 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574490 5222 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:15:07.587944 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574493 5222 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:15:07.587944 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574496 5222 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:15:07.587944 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574498 5222 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:15:07.587944 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574501 5222 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:15:07.587944 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574503 5222 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:15:07.588505 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574506 5222 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:15:07.588505 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574509 5222 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:15:07.588505 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574511 5222 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:15:07.588505 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574514 5222 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:15:07.588505 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574516 5222 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:15:07.588505 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574519 5222 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:15:07.588505 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574521 5222 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:15:07.588505 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574524 5222 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:15:07.588505 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574527 5222 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:15:07.588505 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574529 5222 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:15:07.588505 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574533 5222 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:15:07.588505 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574536 5222 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:15:07.588505 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574539 5222 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:15:07.588505 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574541 5222 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:15:07.588505 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574544 5222 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:15:07.588505 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574546 5222 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:15:07.588505 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574549 5222 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:15:07.588505 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574551 5222 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:15:07.588505 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574554 5222 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:15:07.588505 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574557 5222 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:15:07.589027 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574559 5222 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:15:07.589027 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574562 5222 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:15:07.589027 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574565 5222 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:15:07.589027 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574568 5222 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:15:07.589027 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574571 5222 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:15:07.589027 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574573 5222 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:15:07.589027 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574576 5222 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:15:07.589027 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574579 5222 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:15:07.589027 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574581 5222 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:15:07.589027 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574584 5222 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:15:07.589027 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574586 5222 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:15:07.589027 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574589 5222 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:15:07.589027 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574591 5222 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:15:07.589027 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574597 5222 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:15:07.589027 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574625 5222 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:15:07.589027 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574629 5222 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:15:07.589027 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574633 5222 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:15:07.589027 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574639 5222 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:15:07.589027 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574642 5222 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:15:07.589505 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574645 5222 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:15:07.589505 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574648 5222 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:15:07.589505 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574651 5222 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:15:07.589505 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574654 5222 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:15:07.589505 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.574680 5222 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 00:15:07.589505 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574714 5222 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:15:07.589505 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574719 5222 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:15:07.589505 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574723 5222 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:15:07.589505 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574726 5222 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:15:07.589505 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574729 5222 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:15:07.589505 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574733 5222 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:15:07.589505 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574737 5222 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:15:07.589505 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574740 5222 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:15:07.589505 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574743 5222 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:15:07.589505 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574746 5222 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:15:07.589505 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574749 5222 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:15:07.589893 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574751 5222 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:15:07.589893 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574754 5222 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:15:07.589893 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574757 5222 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:15:07.589893 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574759 5222 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:15:07.589893 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574762 5222 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:15:07.589893 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574765 5222 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:15:07.589893 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574767 5222 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:15:07.589893 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574770 5222 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:15:07.589893 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574772 5222 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:15:07.589893 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574775 5222 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:15:07.589893 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574779 5222 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:15:07.589893 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574782 5222 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:15:07.589893 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574785 5222 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:15:07.589893 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574788 5222 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:15:07.589893 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574790 5222 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:15:07.589893 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574793 5222 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:15:07.589893 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574795 5222 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:15:07.589893 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574797 5222 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:15:07.589893 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574800 5222 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:15:07.589893 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574803 5222 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:15:07.590439 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574806 5222 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:15:07.590439 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574808 5222 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:15:07.590439 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574811 5222 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:15:07.590439 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574814 5222 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:15:07.590439 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574816 5222 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:15:07.590439 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574819 5222 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:15:07.590439 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574821 5222 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:15:07.590439 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574824 5222 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:15:07.590439 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574826 5222 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:15:07.590439 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574829 5222 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:15:07.590439 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574831 5222 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:15:07.590439 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574835 5222 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:15:07.590439 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574838 5222 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:15:07.590439 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574840 5222 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:15:07.590439 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574843 5222 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:15:07.590439 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574845 5222 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:15:07.590439 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574848 5222 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:15:07.590439 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574851 5222 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:15:07.590439 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574855 5222 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:15:07.590892 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574858 5222 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:15:07.590892 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574861 5222 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:15:07.590892 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574864 5222 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:15:07.590892 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574868 5222 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:15:07.590892 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574871 5222 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:15:07.590892 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574874 5222 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:15:07.590892 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574876 5222 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:15:07.590892 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574879 5222 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:15:07.590892 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574882 5222 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:15:07.590892 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574884 5222 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:15:07.590892 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574887 5222 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:15:07.590892 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574891 5222 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:15:07.590892 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574893 5222 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:15:07.590892 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574896 5222 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:15:07.590892 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574899 5222 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:15:07.590892 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574901 5222 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:15:07.590892 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574904 5222 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:15:07.590892 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574907 5222 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:15:07.590892 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574909 5222 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:15:07.590892 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574912 5222 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:15:07.591612 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574915 5222 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:15:07.591612 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574917 5222 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:15:07.591612 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574919 5222 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:15:07.591612 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574922 5222 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:15:07.591612 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574926 5222 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:15:07.591612 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574929 5222 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:15:07.591612 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574931 5222 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:15:07.591612 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574934 5222 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:15:07.591612 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574937 5222 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:15:07.591612 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574939 5222 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:15:07.591612 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574942 5222 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:15:07.591612 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574945 5222 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:15:07.591612 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574947 5222 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:15:07.591612 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574950 5222 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:15:07.591612 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574953 5222 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:15:07.591612 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574957 5222 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:15:07.591612 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574959 5222 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:15:07.591612 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574962 5222 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:15:07.591612 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574964 5222 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:15:07.591612 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574967 5222 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:15:07.592354 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574969 5222 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:15:07.592354 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574972 5222 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:15:07.592354 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574974 5222 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:15:07.592354 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574977 5222 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:15:07.592354 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574979 5222 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:15:07.592354 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574982 5222 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:15:07.592354 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574985 5222 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:15:07.592354 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574988 5222 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:15:07.592354 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574992 5222 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:15:07.592354 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574994 5222 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:15:07.592354 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.574997 5222 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:15:07.592354 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.575000 5222 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:15:07.592354 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.575002 5222 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:15:07.592354 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.575005 5222 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:15:07.592354 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.575007 5222 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:15:07.592354 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:07.575010 5222 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:15:07.592354 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.575073 5222 server.go:957] "Client rotation is on, will bootstrap in background" Oct 31 00:15:07.592354 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.576180 5222 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 31 00:15:07.592354 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.576252 5222 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 31 00:15:07.592948 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.576485 5222 server.go:1014] "Starting client certificate rotation" Oct 31 00:15:07.592948 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.576628 5222 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 00:15:07.592948 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.576695 5222 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 00:08:53 +0000 UTC" deadline="2028-07-07 22:39:25.45944035 +0000 UTC" Oct 31 00:15:07.592948 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.576728 5222 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14782h24m17.882715421s" Oct 31 00:15:07.592948 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.576953 5222 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 00:15:07.592948 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.578103 5222 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 00:15:07.592948 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.579340 5222 log.go:25] "Validated CRI v1 runtime API" Oct 31 00:15:07.592948 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.581224 5222 log.go:25] "Validated CRI v1 image API" Oct 31 00:15:07.592948 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.582055 5222 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 31 00:15:07.592948 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.584011 5222 fs.go:135] Filesystem UUIDs: map[23ee767d-b28d-4825-ae3e-a07f0a1c007a:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 cc168d45-902b-4a82-8b93-fb101dccc663:/dev/nvme0n1p3] Oct 31 00:15:07.593308 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.584030 5222 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/13aeedbbd5795e01fadf0288886faebf6854062a3d743cbf2896b364a908f55d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/13aeedbbd5795e01fadf0288886faebf6854062a3d743cbf2896b364a908f55d/userdata/shm major:0 minor:197 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2d6531cabcd80da4099819572fe53a24f4806363c480abd15f54c08134300f74/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2d6531cabcd80da4099819572fe53a24f4806363c480abd15f54c08134300f74/userdata/shm major:0 minor:64 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2d7517a7af7b2bf180b122c5a436e86320f74f7a9b99a3c3a6e8af9bb71f5675/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2d7517a7af7b2bf180b122c5a436e86320f74f7a9b99a3c3a6e8af9bb71f5675/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2f67477bcc4a7d7c671cdd426b7bd4aca4916983579ffc57621163ce85f1e77a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2f67477bcc4a7d7c671cdd426b7bd4aca4916983579ffc57621163ce85f1e77a/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3c171721a06bad3a5280a8be2538f726a9af19dbd55856c3aaa9873dd468bd8f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3c171721a06bad3a5280a8be2538f726a9af19dbd55856c3aaa9873dd468bd8f/userdata/shm major:0 minor:311 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/526acb8408e46da20b2ba7f24e9ac667d7b7e021118d73e3d193ec380f36efb7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/526acb8408e46da20b2ba7f24e9ac667d7b7e021118d73e3d193ec380f36efb7/userdata/shm major:0 minor:315 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6450b4ee33119ffd3082ede4ecbe8007a9275feabbf96dee54b65a703684250b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6450b4ee33119ffd3082ede4ecbe8007a9275feabbf96dee54b65a703684250b/userdata/shm major:0 minor:95 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6810965ba74ad28fc39e610471d28dd74b2bd6d74c30ae3c36b47a9a6bd80630/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6810965ba74ad28fc39e610471d28dd74b2bd6d74c30ae3c36b47a9a6bd80630/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7e1bf6c5db5b115aaee51fc3686ab440559202deb4f763f9c005dbdf951ef51d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7e1bf6c5db5b115aaee51fc3686ab440559202deb4f763f9c005dbdf951ef51d/userdata/shm major:0 minor:227 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7eae82e9ea1bf2459313e7139b9f222797c511f08b4b8774861dcd838dd46ad9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7eae82e9ea1bf2459313e7139b9f222797c511f08b4b8774861dcd838dd46ad9/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/85cdbe46bdd5e24c6593111a991360dc0159e1e871dafb1bd333367618c00f41/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/85cdbe46bdd5e24c6593111a991360dc0159e1e871dafb1bd333367618c00f41/userdata/shm major:0 minor:221 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a3f9f3fe924dfb4d4b266048dfe16c8b0550c3570ad9df858260e0e531a7a485/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a3f9f3fe924dfb4d4b266048dfe16c8b0550c3570ad9df858260e0e531a7a485/userdata/shm major:0 minor:219 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a41c4f5553eae427686ab6b45e27123b288ecab81d8c898fd14d6f2f24ab25d9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a41c4f5553eae427686ab6b45e27123b288ecab81d8c898fd14d6f2f24ab25d9/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a4c6c3886ed113a59ea0d1ca28218a244394c08b02880bb9e841d2c55dc730ef/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a4c6c3886ed113a59ea0d1ca28218a244394c08b02880bb9e841d2c55dc730ef/userdata/shm major:0 minor:280 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b1592c8b3c26564beed25f9e4abca5a54557d95546cab7063ac25aa7da5d0251/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b1592c8b3c26564beed25f9e4abca5a54557d95546cab7063ac25aa7da5d0251/userdata/shm major:0 minor:287 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b9d29c14a25ecc7102c32fec2fe49a6da0882079c0b09b5e376492bf0b7ce5d6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b9d29c14a25ecc7102c32fec2fe49a6da0882079c0b09b5e376492bf0b7ce5d6/userdata/shm major:0 minor:224 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c879df49638ffe7b92434481fc181e47422074d35dc456a77ac6926fc51595a1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c879df49638ffe7b92434481fc181e47422074d35dc456a77ac6926fc51595a1/userdata/shm major:0 minor:278 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d7f607419a7dbf03da1d4459c5376eb54185fa2270d3616edd508177ed7540eb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d7f607419a7dbf03da1d4459c5376eb54185fa2270d3616edd508177ed7540eb/userdata/shm major:0 minor:313 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d84687d8838569571481ff1562bc7bac9a6445916f8efa90c7cc3a4d986ec18c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d84687d8838569571481ff1562bc7bac9a6445916f8efa90c7cc3a4d986ec18c/userdata/shm major:0 minor:77 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/da2e414968db90f728e2dbdf6b02f29a507bfa7e488f8490650761508fefde8c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/da2e414968db90f728e2dbdf6b02f29a507bfa7e488f8490650761508fefde8c/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e352133d4a0ba734426022a4680011129d2529bcf53892b04212d2b156322dd4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e352133d4a0ba734426022a4680011129d2529bcf53892b04212d2b156322dd4/userdata/shm major:0 minor:344 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e5c3e87bed66ffaf5ebc4ef908b7b5485a99b714652eb8b0b7f3ce5237e7acab/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e5c3e87bed66ffaf5ebc4ef908b7b5485a99b714652eb8b0b7f3ce5237e7acab/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0cbf0042-931c-4135-9330-f9e798b7a567/volumes/kubernetes.io~projected/kube-api-access-ms4cs:{mountpoint:/var/lib/kubelet/pods/0cbf0042-931c-4135-9330-f9e798b7a567/volumes/kubernetes.io~projected/kube-api-access-ms4cs major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0eb4888f-40a8-4aba-85d8-3c845382bd02/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/0eb4888f-40a8-4aba-85d8-3c845382bd02/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0eb4888f-40a8-4aba-85d8-3c845382bd02/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/0eb4888f-40a8-4aba-85d8-3c845382bd02/volumes/kubernetes.io~empty-dir/tmp major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0eb4888f-40a8-4aba-85d8-3c845382bd02/volumes/kubernetes.io~projected/kube-api-access-nrkcn:{mountpoint:/var/lib/kubelet/pods/0eb4888f-40a8-4aba-85d8-3c845382bd02/volumes/kubernetes.io~projected/kube-api-access-nrkcn major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0f8c3c03-a1fc-4376-98c6-feeeefa08068/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/0f8c3c03-a1fc-4376-98c6-feeeefa08068/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0f8c3c03-a1fc-4376-98c6-feeeefa08068/volumes/kubernetes.io~projected/kube-api-access-8wrw4:{mountpoint:/var/lib/kubelet/pods/0f8c3c03-a1fc-4376-98c6-feeeefa08068/volumes/kubernetes.io~projected/kube-api-access-8wrw4 major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0f8c3c03-a1fc-4376-98c6-feeeefa08068/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/0f8c3c03-a1fc-4376-98c6-feeeefa08068/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1b88d3b9-589f-4f4e-b8a6-872f4fb5536b/volumes/kubernetes.io~projected/kube-api-access-dqw94:{mountpoint:/var/lib/kubelet/pods/1b88d3b9-589f-4f4e-b8a6-872f4fb5536b/volumes/kubernetes.io~projected/kube-api-access-dqw94 major:0 minor:309 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1b88d3b9-589f-4f4e-b8a6-872f4fb5536b/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/1b88d3b9-589f-4f4e-b8a6-872f4fb5536b/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:306 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1b88d3b9-589f-4f4e-b8a6-872f4fb5536b/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/1b88d3b9-589f-4f4e-b8a6-872f4fb5536b/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:308 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1bc3d212-b67c-4368-90a3-8e080a4f1452/volumes/kubernetes.io~projected/kube-api-access-wm7wn:{mountpoint:/var/lib/kubelet/pods/1bc3d212-b67c-4368-90a3-8e080a4f1452/volumes/kubernetes.io~projected/kube-api-access-wm7wn major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/26aba8fc-e2da-430f-b0d8-6ee45468441d/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/26aba8fc-e2da-430f-b0d8-6ee45468441d/volumes/kubernetes.io~empty-dir/config-out major:0 minor:341 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/26aba8fc-e2da-430f-b0d8-6ee45468441d/volumes/kubernetes.io~projected/kube-api-access-4v246:{mountpoint:/var/lib/kubelet/pods/26aba8fc-e2da-430f-b0d8-6ee45468441d/volumes/kubernetes.io~projected/kube-api-access-4v246 major:0 minor:342 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/26aba8fc-e2da-430f-b0d8-6ee45468441d/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/26aba8fc-e2da-430f-b0d8-6ee45468441d/volumes/kubernetes.io~projected/tls-assets major:0 minor:335 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/26aba8fc-e2da-430f-b0d8-6ee45468441d/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/26aba8fc-e2da-430f-b0d8-6ee45468441d/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:336 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/26aba8fc-e2da-430f-b0d8-6ee45468441d/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/26aba8fc-e2da-430f-b0d8-6ee45468441d/volumes/kubernetes.io~secret/config-volume major:0 minor:337 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/26aba8fc-e2da-430f-b0d8-6ee45468441d/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/26aba8fc-e2da-430f-b0d8-6ee45468441d/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:340 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/26aba8fc-e2da-430f-b0d8-6ee45468441d/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/26aba8fc-e2da-430f-b0d8-6ee45468441d/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:343 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/26aba8fc-e2da-430f-b0d8-6ee45468441d/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/26aba8fc-e2da-430f-b0d8-6ee45468441d/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:331 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/26aba8fc-e2da-430f-b0d8-6ee45468441d/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/26aba8fc-e2da-430f-b0d8-6ee45468441d/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:339 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/26aba8fc-e2da-430f-b0d8-6ee45468441d/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/26aba8fc-e2da-430f-b0d8-6ee45468441d/volumes/kubernetes.io~secret/web-config major:0 minor:338 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2ed1e615-cbca-482c-89cc-0fe7d8e7e597/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/2ed1e615-cbca-482c-89cc-0fe7d8e7e597/volumes/kubernetes.io~secret/agent-certs major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5c58f866-ef80-42d5-aab0-0cdefbc7a158/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/5c58f866-ef80-42d5-aab0-0cdefbc7a158/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5c58f866-ef80-42d5-aab0-0cdefbc7a158/volumes/kubernetes.io~projected/kube-api-access-n5hld:{mountpoint:/var/lib/kubelet/pods/5c58f866-ef80-42d5-aab0-0cdefbc7a158/volumes/kubernetes.io~projected/kube-api-access-n5hld major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5c58f866-ef80-42d5-aab0-0cdefbc7a158/volumes/kubernetes.io~projected/registry-tls:{mountpoint:/var/lib/kubelet/pods/5c58f866-ef80-42d5-aab0-0cdefbc7a158/volumes/kubernetes.io~projected/registry-tls major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5c58f866-ef80-42d5-aab0-0cdefbc7a158/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/5c58f866-ef80-42d5-aab0-0cdefbc7a158/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5c58f866-ef80-42d5-aab0-0cdefbc7a158/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/5c58f866-ef80-42d5-aab0-0cdefbc7a158/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/652defc0-301f-4679-8e49-6729381259dd/volumes/kubernetes.io~projected/kube-api-access-jwf5l:{mountpoint:/var/lib/kubelet/pods/652defc0-301f-4679-8e49-6729381259dd/volumes/kubernetes.io~projected/kube-api-access-jwf5l major:0 minor:304 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/652defc0-301f-4679-8e49-6729381259dd/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/652defc0-301f-4679-8e49-6729381259dd/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:299 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/652defc0-301f-4679-8e49-6729381259dd/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/652defc0-301f-4679-8e49-6729381259dd/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:303 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6a21f264-1e80-4eab-8c6e-27ca9ca19f0d/volumes/kubernetes.io~projected/kube-api-access-8j6tz:{mountpoint:/var/lib/kubelet/pods/6a21f264-1e80-4eab-8c6e-27ca9ca19f0d/volumes/kubernetes.io~projected/kube-api-access-8j6tz major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/83d351d1-c6e7-43a6-a0fb-eaecefa8c5b2/volumes/kubernetes.io~projected/kube-api-access-5lsrt:{mountpoint:/var/lib/kubelet/pods/83d351d1-c6e7-43a6-a0fb-eaecefa8c5b2/volumes/kubernetes.io~projected/kube-api-access-5lsrt major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/83d351d1-c6e7-43a6-a0fb-eaecefa8c5b2/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/83d351d1-c6e7-43a6-a0fb-eaecefa8c5b2/volumes/kubernetes.io~secret/metrics-tls major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/978a9050-dddb-4869-85d8-84a001554c5b/volumes/kubernetes.io~projected/kube-api-access-gbpb7:{mountpoint:/var/lib/kubelet/pods/978a9050-dddb-4869-85d8-84a001554c5b/volumes/kubernetes.io~projected/kube-api-access-gbpb7 major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/97d11160-2114-4fbf-95c2-fa7b4c9d2260/volumes/kubernetes.io~projected/kube-api-access-6pc97:{mountpoint:/var/lib/kubelet/pods/97d11160-2114-4fbf-95c2-fa7b4c9d2260/volumes/kubernetes.io~projected/kube-api-access-6pc97 major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/97d11160-2114-4fbf-95c2-fa7b4c9d2260/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/97d11160-2114-4fbf-95c2-fa7b4c9d2260/volumes/kubernetes.io~secret/cert major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9801ff00-9e9a-4141-8212-ca32bf0251b3/volumes/kubernetes.io~projected/kube-api-access-xzs4x:{mountpoint:/var/lib/kubelet/pods/9801ff00-9e9a-4141-8212-ca32bf0251b3/volumes/kubernetes.io~projected/kube-api-access-xzs4x major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a077dea4-b5a3-4335-bb6a-7618e794eb8f/volumes/kubernetes.io~projected/kube-api-access-xc9bh:{mountpoint:/var/lib/kubelet/pods/a077dea4-b5a3-4335-bb6a-7618e794eb8f/volumes/kubernetes.io~projected/kube-api-access-xc9bh major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a077dea4-b5a3-4335-bb6a-7618e794eb8f/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/a077dea4-b5a3-4335-bb6a-7618e794eb8f/volumes/kubernetes.io~secret/metrics-certs major:0 minor:272 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a7d0a920-91fe-4c4a-b9bc-67550c324e7f/volumes/kubernetes.io~projected/kube-api-access-djjgz:{mountpoint:/var/lib/kubelet/pods/a7d0a920-91fe-4c4a-b9bc-67550c324e7f/volumes/kubernetes.io~projected/kube-api-access-djjgz major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c61956a7-ba51-4063-bc25-aaa418168bc2/volumes/kubernetes.io~projected/kube-api-access-cspld:{mountpoint:/var/lib/kubelet/pods/c61956a7-ba51-4063-bc25-aaa418168bc2/volumes/kubernetes.io~projected/kube-api-access-cspld major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d02ae6b5-4860-45b3-8d16-4a49531b77d4/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/d02ae6b5-4860-45b3-8d16-4a49531b77d4/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:277 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d22409bc-1bcf-46b1-9789-6f7298566a7f/volumes/kubernetes.io~projected/kube-api-access-srgbh:{mountpoint:/var/lib/kubelet/pods/d22409bc-1bcf-46b1-9789-6f7298566a7f/volumes/kubernetes.io~projected/kube-api-access-srgbh major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d22409bc-1bcf-46b1-9789-6f7298566a7f/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/d22409bc-1bcf-46b1-9789-6f7298566a7f/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ea9a5160-4696-4b75-8a10-0fbc54b2ad04/volumes/kubernetes.io~projected/kube-api-access-v7hn4:{mountpoint:/var/lib/kubelet/pods/ea9a5160-4696-4b75-8a10-0fbc54b2ad04/volumes/kubernetes.io~projected/kube-api-access-v7hn4 major:0 minor:310 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ea9a5160-4696-4b75-8a10-0fbc54b2ad04/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/ea9a5160-4696-4b75-8a10-0fbc54b2ad04/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:307 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ea9a5160-4696-4b75-8a10-0fbc54b2ad04/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/ea9a5160-4696-4b75-8a10-0fbc54b2ad04/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:305 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eadb6436-5e5b-4116-8ed7-26aa935fa749/volumes/kubernetes.io~projected/kube-api-access-nz8r2:{mountpoint:/var/lib/kubelet/pods/eadb6436-5e5b-4116-8ed7-26aa935fa749/volumes/kubernetes.io~projected/kube-api-access-nz8r2 major:0 minor:56 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/d107547f64a0dddd504257db0e20ab4f99219ccc2784c2a2d38161514a91f134/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/0e7687353a7aafe59a128c8245c2e28ebd113dd50256c46c7e4a8dd77e9902fe/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/92c36e578d85dd78e698d0cc44abb96e8b8f629c64b7a93c2f1773dc02683b32/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/e2e25cebe95e12aa1f8daf34d1a3badeb27c04b622865cf23ee22499d5de17b9/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/8dabdd2fadc52381918f38804713570c36505479401cfaeb67c41d650b310ba2/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/f844e9612f2afa13fe055542a3a1d74a1634a27c69d4655dfef4844afc12bf5a/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/089f54f5ef4dee526d24e3776c1908242c6d4b9eb64017bfb2cce05b12735249/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/9c48adfd10e1b63cc4fe66a5cb7d7e89865861d2772ba28cb6855c520a9931e4/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-133:{mountpoint:/var/lib/containers/storage/overlay/9e571e2b40df8c20d087c093da0c3ec482fa8b795e6fbb06182c7053459630e4/merged major:0 minor:133 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/de5d666db11d9527ab9a4aaeab95a6ba653837a9c596dcec21f25eb620d5fdb3/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-138:{mountpoint:/var/lib/containers/storage/overlay/77bd994da268868cd7c3ffe117d405696959ee8ad1346db6e081fdd9c88996b7/merged major:0 minor:138 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/628de5786e9de60cfb787e658600948d472465dd23e0f7410cdd82b71cdccb68/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/f663cebc2911e18c8ce4e3aa031b5ae6b1b78ffeef3ca99ed8b0565b0d62e847/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/8e65308acd0d63b39772ecbdc8425aeb7f28b0a50c4bf43a3c4699eae56b6fef/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-164:{mountpoint:/var/lib/containers/storage/overlay/28aab9b0449b2e3d5ac39352c05ba68ecf0c2ce284dbb16cc272457b33134af0/merged major:0 minor:164 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/e675285176bb375e9c46246a24084fa49b80d7fcf4cb5639c77ff4084e5511c2/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/8671ab3fc7117e797edcc2267efa1a1fc3221d6e2a4e6ac51372deda7074eec9/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/8b0387f9622639dcf73f20bc1b7338b413c6858e3af4f8bebcddd9b0b8910fe6/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/dd45717ccb2a251c56b10771a6b4172678900fb8d96e171478b808ec6fc85307/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/c6af2bf703ae6ba1883020cea1627614a2aac3b64c00303cd11acbbfc99c160c/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/568db1eec225f5c4691b8b328a60aaa4b702a8b068f3586121a6a2edb021da9c/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-202:{mountpoint:/var/lib/containers/storage/overlay/62cc3a3f8471fb671ba9bb434592dec2a45752acc2697e9074a697a6cc7dc4f9/merged major:0 minor:202 fsType:overlay blockSize:0} overlay_0-204:{mountpoint:/var/lib/containers/storage/overlay/a4450fa9634181995656452efe595ce2d5580f03abda04d44bdaf2bc4ee1a5e0/merged major:0 minor:204 fsType:overlay blockSize:0} overlay_0-229:{mountpoint:/var/lib/containers/storage/overlay/23113e08845c26cdd872d6f31fa862e9a7a24314e243513c488ac2752c0652ac/merged major:0 minor:229 fsType:overlay blockSize:0} overlay_0-231:{mountpoint:/var/lib/containers/storage/overlay/80e9dc3adccf9fc43301f8bf08c9ba41287f4218aec869a0e3a1e66f56ff2153/merged major:0 minor:231 fsType:overlay blockSize:0} overlay_0-233:{mountpoint:/var/lib/containers/storage/overlay/4d1def1a4b19b0714cde4e51de5f7071dfe18681cf1049ded204614f9ca13526/merged major:0 minor:233 fsType:overlay blockSize:0} overlay_0-235:{mountpoint:/var/lib/containers/storage/overlay/ffbf5bade61f8f75e908a743857f1939d69e680e904132816668d4821fff8aa2/merged major:0 minor:235 fsType:overlay blockSize:0} overlay_0-237:{mountpoint:/var/lib/containers/storage/overlay/c1fbf6afb41e9ffe15e12b9d5c5b6797476e158608087ba15d608d3f97cf83c9/merged major:0 minor:237 fsType:overlay blockSize:0} overlay_0-243:{mountpoint:/var/lib/containers/storage/overlay/1b5b4ed0aeaa968be7e378139ef01a4f784e3798c7cbeceda52d91d1ca3ea368/merged major:0 minor:243 fsType:overlay blockSize:0} overlay_0-245:{mountpoint:/var/lib/containers/storage/overlay/0a10093f46593e5de97dc3d91038fda19f75c22be9436daee70e24fa9c0461e2/merged major:0 minor:245 fsType:overlay blockSize:0} overlay_0-251:{mountpoint:/var/lib/containers/storage/overlay/c9a169b9a8de7af5de085aa6dcaeab34dcbe22ad6fd180e7f902a234d758eca9/merged major:0 minor:251 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/29c3966f6cecb2c73bb918f4cf2f3fdbfddb00da638e7d7c5c12c381cfa2bb75/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/9d62ee4642b9dcc9e7580b68c48db466d0c901ee1875f16176cf062c3d8404d8/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-267:{mountpoint:/var/lib/containers/storage/overlay/58f60946db19f47e56a4d28fe21680146ebaba41b2cc88941b4195e9878b0434/merged major:0 minor:267 fsType:overlay blockSize:0} overlay_0-282:{mountpoint:/var/lib/containers/storage/overlay/7795c6b55613dfb02e1de34f00f061284ced2af9be0fbb893eb9cf4cb1387d7a/merged major:0 minor:282 fsType:overlay blockSize:0} overlay_0-284:{mountpoint:/var/lib/containers/storage/overlay/e6853655aa36aa44ae9cb25af152a400d4d3f12c006735cca3f6e0461b8d48f4/merged major:0 minor:284 fsType:overlay blockSize:0} overlay_0-289:{mountpoint:/var/lib/containers/storage/overlay/a862639ad12c02d9efa8b03ea97032a6ac838db4db50da41059b04b1f5f7d80b/merged major:0 minor:289 fsType:overlay blockSize:0} overlay_0-291:{mountpoint:/var/lib/containers/storage/overlay/6abe6b4505f595f75f29a034e6635ca00d87d7ad0fc5b3f2035e93e3365859b1/merged major:0 minor:291 fsType:overlay blockSize:0} overlay_0-293:{mountpoint:/var/lib/containers/storage/overlay/4c6a0b9453d233f979206c0c3598133df8f72d40274fe9689ba3f085b3732d35/merged major:0 minor:293 fsType:overlay blockSize:0} overlay_0-317:{mountpoint:/var/lib/containers/storage/overlay/c6f749744eec21253546226c106daf78da2fada8e2ae2b716751d38d60a9463b/merged major:0 minor:317 fsType:overlay blockSize:0} overlay_0-319:{mountpoint:/var/lib/containers/storage/overlay/884db8eb55223a3c0e6d0571addf10873e911e231778218f12df08d8cebcada2/merged major:0 minor:319 fsType:overlay blockSize:0} overlay_0-321:{mountpoint:/var/lib/containers/storage/overlay/d45e5799aa8412a54a6e81345f3c0c1f2aa0b8fe9c18239a9470a343c81fc722/merged major:0 minor:321 fsType:overlay blockSize:0} overlay_0-323:{mountpoint:/var/lib/containers/storage/overlay/74b3a32771420b3d7a467fb1d6f1a8312f012bc3678e17d26eba675799cde89f/merged major:0 minor:323 fsType:overlay blockSize:0} overlay_0-325:{mountpoint:/var/lib/containers/storage/overlay/7bc13c8a3c5d5ad68dbe618680c44692042544c08f338a8ad9527de25bb1f802/merged major:0 minor:325 fsType:overlay blockSize:0} overlay_0-346:{mountpoint:/var/lib/containers/storage/overlay/cc39448341bc79dbafeb9ca064fac6215a5f388ed777a7795c6841816a84ff79/merged major:0 minor:346 fsType:overlay blockSize:0} overlay_0-348:{mountpoint:/var/lib/containers/storage/overlay/7ef5b62a45224e7e463c53289213034bd9ae576aca347a57625f00fdf4a91695/merged major:0 minor:348 fsType:overlay blockSize:0} overlay_0-350:{mountpoint:/var/lib/containers/storage/overlay/a26460df84a3a56f8bf07286fb4364d6c6c375caf17f58a87a83ea75c174fe98/merged major:0 minor:350 fsType:overlay blockSize:0} overlay_0-62:{mountpoint:/var/lib/containers/storage/overlay/efc608e4df66fa42c0f2a45da2bb2cc9f8b9a384437e2d8387007f7e36f8e903/merged major:0 minor:62 fsType:overlay blockSize:0} overlay_0-66:{mountpoint:/var/lib/containers/storage/overlay/2ec40c094bb081ad3d113122c4ab9a35b3ce6e8156fb996065e97cbf33602a4f/merged major:0 minor:66 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/cfc0ec0bb5b3e938462bb9978172a957e8a7fc17a91efa090c7dcb6aec93cfca/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/465861d591f9d088ef04caa3584530fb8314f8828d41eb5381e37b68925fff5a/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/8121d2230e62d4c976cd5f362db8e689852183403db730c9cbcf308b704a0cb6/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/e5332ef4ef8bcd83f86b8471eb71fee82cc4167131f00e2d7cf4c8e9a23094e2/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/edb66282deefb520b835c0c6c10071a3f79d6f03bc1c4897d572b23fdbc8f34b/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/4b4a36e5ebd03ca3f8d717c8b4e266a57dbd2376ac7c1e0f9d4c303672b48e16/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/2daaab63b656f5d52ae11b430ed98eb9957e20608a64f28dc474870f7bd46b94/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-97:{mountpoint:/var/lib/containers/storage/overlay/777529b2297b61a8772f571e5a801ca5deb709c3cec26394cc391cf602e056a6/merged major:0 minor:97 fsType:overlay blockSize:0}] Oct 31 00:15:07.594076 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.592939 5222 manager.go:217] Machine: {Timestamp:2026-10-31 00:15:07.592299135 +0000 UTC m=+0.063468879 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3102376 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2a538df4d8ddd6a1300a4438b9b447 SystemUUID:ec2a538d-f4d8-ddd6-a130-0a4438b9b447 BootID:af8b0a00-c27e-4d39-bc3a-3bf1c043eb45 Filesystems:[{Device:/var/lib/kubelet/pods/26aba8fc-e2da-430f-b0d8-6ee45468441d/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:337 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/26aba8fc-e2da-430f-b0d8-6ee45468441d/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:343 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-350 DeviceMajor:0 DeviceMinor:350 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-133 DeviceMajor:0 DeviceMinor:133 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3c171721a06bad3a5280a8be2538f726a9af19dbd55856c3aaa9873dd468bd8f/userdata/shm DeviceMajor:0 DeviceMinor:311 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/26aba8fc-e2da-430f-b0d8-6ee45468441d/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:341 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/0f8c3c03-a1fc-4376-98c6-feeeefa08068/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/0cbf0042-931c-4135-9330-f9e798b7a567/volumes/kubernetes.io~projected/kube-api-access-ms4cs DeviceMajor:0 DeviceMinor:45 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d22409bc-1bcf-46b1-9789-6f7298566a7f/volumes/kubernetes.io~projected/kube-api-access-srgbh DeviceMajor:0 DeviceMinor:215 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-233 DeviceMajor:0 DeviceMinor:233 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-291 DeviceMajor:0 DeviceMinor:291 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-346 DeviceMajor:0 DeviceMinor:346 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a077dea4-b5a3-4335-bb6a-7618e794eb8f/volumes/kubernetes.io~projected/kube-api-access-xc9bh DeviceMajor:0 DeviceMinor:51 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2d6531cabcd80da4099819572fe53a24f4806363c480abd15f54c08134300f74/userdata/shm DeviceMajor:0 DeviceMinor:64 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/85cdbe46bdd5e24c6593111a991360dc0159e1e871dafb1bd333367618c00f41/userdata/shm DeviceMajor:0 DeviceMinor:221 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/526acb8408e46da20b2ba7f24e9ac667d7b7e021118d73e3d193ec380f36efb7/userdata/shm DeviceMajor:0 DeviceMinor:315 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/26aba8fc-e2da-430f-b0d8-6ee45468441d/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:335 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-251 DeviceMajor:0 DeviceMinor:251 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0eb4888f-40a8-4aba-85d8-3c845382bd02/volumes/kubernetes.io~projected/kube-api-access-nrkcn DeviceMajor:0 DeviceMinor:59 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2d7517a7af7b2bf180b122c5a436e86320f74f7a9b99a3c3a6e8af9bb71f5675/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-235 DeviceMajor:0 DeviceMinor:235 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-289 DeviceMajor:0 DeviceMinor:289 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-348 DeviceMajor:0 DeviceMinor:348 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a7d0a920-91fe-4c4a-b9bc-67550c324e7f/volumes/kubernetes.io~projected/kube-api-access-djjgz DeviceMajor:0 DeviceMinor:52 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7eae82e9ea1bf2459313e7139b9f222797c511f08b4b8774861dcd838dd46ad9/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a41c4f5553eae427686ab6b45e27123b288ecab81d8c898fd14d6f2f24ab25d9/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/652defc0-301f-4679-8e49-6729381259dd/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:303 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9801ff00-9e9a-4141-8212-ca32bf0251b3/volumes/kubernetes.io~projected/kube-api-access-xzs4x DeviceMajor:0 DeviceMinor:57 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-62 DeviceMajor:0 DeviceMinor:62 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/13aeedbbd5795e01fadf0288886faebf6854062a3d743cbf2896b364a908f55d/userdata/shm DeviceMajor:0 DeviceMinor:197 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/97d11160-2114-4fbf-95c2-fa7b4c9d2260/volumes/kubernetes.io~projected/kube-api-access-6pc97 DeviceMajor:0 DeviceMinor:214 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-321 DeviceMajor:0 DeviceMinor:321 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a4c6c3886ed113a59ea0d1ca28218a244394c08b02880bb9e841d2c55dc730ef/userdata/shm DeviceMajor:0 DeviceMinor:280 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ea9a5160-4696-4b75-8a10-0fbc54b2ad04/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:305 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ea9a5160-4696-4b75-8a10-0fbc54b2ad04/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:307 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e352133d4a0ba734426022a4680011129d2529bcf53892b04212d2b156322dd4/userdata/shm DeviceMajor:0 DeviceMinor:344 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5c58f866-ef80-42d5-aab0-0cdefbc7a158/volumes/kubernetes.io~projected/registry-tls DeviceMajor:0 DeviceMinor:216 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-319 DeviceMajor:0 DeviceMinor:319 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0eb4888f-40a8-4aba-85d8-3c845382bd02/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:48 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d84687d8838569571481ff1562bc7bac9a6445916f8efa90c7cc3a4d986ec18c/userdata/shm DeviceMajor:0 DeviceMinor:77 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-202 DeviceMajor:0 DeviceMinor:202 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-229 DeviceMajor:0 DeviceMinor:229 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/83d351d1-c6e7-43a6-a0fb-eaecefa8c5b2/volumes/kubernetes.io~projected/kube-api-access-5lsrt DeviceMajor:0 DeviceMinor:213 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/26aba8fc-e2da-430f-b0d8-6ee45468441d/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:331 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/26aba8fc-e2da-430f-b0d8-6ee45468441d/volumes/kubernetes.io~projected/kube-api-access-4v246 DeviceMajor:0 DeviceMinor:342 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b9d29c14a25ecc7102c32fec2fe49a6da0882079c0b09b5e376492bf0b7ce5d6/userdata/shm DeviceMajor:0 DeviceMinor:224 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/5c58f866-ef80-42d5-aab0-0cdefbc7a158/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:225 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/26aba8fc-e2da-430f-b0d8-6ee45468441d/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:336 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/6a21f264-1e80-4eab-8c6e-27ca9ca19f0d/volumes/kubernetes.io~projected/kube-api-access-8j6tz DeviceMajor:0 DeviceMinor:54 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/eadb6436-5e5b-4116-8ed7-26aa935fa749/volumes/kubernetes.io~projected/kube-api-access-nz8r2 DeviceMajor:0 DeviceMinor:56 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-317 DeviceMajor:0 DeviceMinor:317 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2f67477bcc4a7d7c671cdd426b7bd4aca4916983579ffc57621163ce85f1e77a/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/83d351d1-c6e7-43a6-a0fb-eaecefa8c5b2/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:206 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-245 DeviceMajor:0 DeviceMinor:245 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1b88d3b9-589f-4f4e-b8a6-872f4fb5536b/volumes/kubernetes.io~projected/kube-api-access-dqw94 DeviceMajor:0 DeviceMinor:309 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d22409bc-1bcf-46b1-9789-6f7298566a7f/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:212 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-243 DeviceMajor:0 DeviceMinor:243 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-267 DeviceMajor:0 DeviceMinor:267 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-138 DeviceMajor:0 DeviceMinor:138 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-323 DeviceMajor:0 DeviceMinor:323 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/26aba8fc-e2da-430f-b0d8-6ee45468441d/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:338 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/0eb4888f-40a8-4aba-85d8-3c845382bd02/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:50 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-293 DeviceMajor:0 DeviceMinor:293 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6810965ba74ad28fc39e610471d28dd74b2bd6d74c30ae3c36b47a9a6bd80630/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-282 DeviceMajor:0 DeviceMinor:282 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b1592c8b3c26564beed25f9e4abca5a54557d95546cab7063ac25aa7da5d0251/userdata/shm DeviceMajor:0 DeviceMinor:287 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/2ed1e615-cbca-482c-89cc-0fe7d8e7e597/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:49 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/da2e414968db90f728e2dbdf6b02f29a507bfa7e488f8490650761508fefde8c/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1bc3d212-b67c-4368-90a3-8e080a4f1452/volumes/kubernetes.io~projected/kube-api-access-wm7wn DeviceMajor:0 DeviceMinor:53 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-66 DeviceMajor:0 DeviceMinor:66 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e5c3e87bed66ffaf5ebc4ef908b7b5485a99b714652eb8b0b7f3ce5237e7acab/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-231 DeviceMajor:0 DeviceMinor:231 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a077dea4-b5a3-4335-bb6a-7618e794eb8f/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:272 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/978a9050-dddb-4869-85d8-84a001554c5b/volumes/kubernetes.io~projected/kube-api-access-gbpb7 DeviceMajor:0 DeviceMinor:55 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-164 DeviceMajor:0 DeviceMinor:164 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0f8c3c03-a1fc-4376-98c6-feeeefa08068/volumes/kubernetes.io~projected/kube-api-access-8wrw4 DeviceMajor:0 DeviceMinor:58 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c879df49638ffe7b92434481fc181e47422074d35dc456a77ac6926fc51595a1/userdata/shm DeviceMajor:0 DeviceMinor:278 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/c61956a7-ba51-4063-bc25-aaa418168bc2/volumes/kubernetes.io~projected/kube-api-access-cspld DeviceMajor:0 DeviceMinor:286 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/652defc0-301f-4679-8e49-6729381259dd/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:299 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/652defc0-301f-4679-8e49-6729381259dd/volumes/kubernetes.io~projected/kube-api-access-jwf5l DeviceMajor:0 DeviceMinor:304 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5c58f866-ef80-42d5-aab0-0cdefbc7a158/volumes/kubernetes.io~projected/kube-api-access-n5hld DeviceMajor:0 DeviceMinor:223 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1b88d3b9-589f-4f4e-b8a6-872f4fb5536b/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:308 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6450b4ee33119ffd3082ede4ecbe8007a9275feabbf96dee54b65a703684250b/userdata/shm DeviceMajor:0 DeviceMinor:95 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-204 DeviceMajor:0 DeviceMinor:204 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a3f9f3fe924dfb4d4b266048dfe16c8b0550c3570ad9df858260e0e531a7a485/userdata/shm DeviceMajor:0 DeviceMinor:219 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7e1bf6c5db5b115aaee51fc3686ab440559202deb4f763f9c005dbdf951ef51d/userdata/shm DeviceMajor:0 DeviceMinor:227 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-237 DeviceMajor:0 DeviceMinor:237 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/26aba8fc-e2da-430f-b0d8-6ee45468441d/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:339 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/97d11160-2114-4fbf-95c2-fa7b4c9d2260/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:211 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d7f607419a7dbf03da1d4459c5376eb54185fa2270d3616edd508177ed7540eb/userdata/shm DeviceMajor:0 DeviceMinor:313 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/26aba8fc-e2da-430f-b0d8-6ee45468441d/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:340 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/0f8c3c03-a1fc-4376-98c6-feeeefa08068/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:47 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-97 DeviceMajor:0 DeviceMinor:97 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5c58f866-ef80-42d5-aab0-0cdefbc7a158/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:218 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d02ae6b5-4860-45b3-8d16-4a49531b77d4/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:277 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-284 DeviceMajor:0 DeviceMinor:284 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1b88d3b9-589f-4f4e-b8a6-872f4fb5536b/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:306 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/5c58f866-ef80-42d5-aab0-0cdefbc7a158/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:217 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ea9a5160-4696-4b75-8a10-0fbc54b2ad04/volumes/kubernetes.io~projected/kube-api-access-v7hn4 DeviceMajor:0 DeviceMinor:310 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-325 DeviceMajor:0 DeviceMinor:325 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:13aeedbbd5795e0 MacAddress:7a:fa:d2:fe:5f:87 Speed:10000 Mtu:8901} {Name:3c171721a06bad3 MacAddress:92:a0:f6:12:77:3f Speed:10000 Mtu:8901} {Name:7e1bf6c5db5b115 MacAddress:12:8f:de:3c:51:50 Speed:10000 Mtu:8901} {Name:85cdbe46bdd5e24 MacAddress:9e:fa:c3:65:3c:22 Speed:10000 Mtu:8901} {Name:a3f9f3fe924dfb4 MacAddress:16:8d:25:82:74:a0 Speed:10000 Mtu:8901} {Name:a4c6c3886ed113a MacAddress:b6:cf:55:f3:03:3c Speed:10000 Mtu:8901} {Name:b1592c8b3c26564 MacAddress:6e:85:e8:3e:1d:14 Speed:10000 Mtu:8901} {Name:b9d29c14a25ecc7 MacAddress:a2:95:a7:7e:f2:4a Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:d2:28:5e:33 Speed:0 Mtu:9001} {Name:br-int MacAddress:62:25:d4:c9:1e:38 Speed:0 Mtu:8901} {Name:c879df49638ffe7 MacAddress:ea:72:94:00:85:da Speed:10000 Mtu:8901} {Name:d7f607419a7dbf0 MacAddress:d6:eb:49:e5:15:f0 Speed:10000 Mtu:8901} {Name:e352133d4a0ba73 MacAddress:f6:5c:f1:e6:7b:81 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:06:ff:d2:28:5e:33 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:3e:02:69:e8:8e:fc Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:4a:86:01:71:a3:2b Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 31 00:15:07.594320 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.594090 5222 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 31 00:15:07.594320 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.594169 5222 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 31 00:15:07.594573 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.594542 5222 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 31 00:15:07.594776 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.594577 5222 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-56.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 31 00:15:07.594818 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.594782 5222 topology_manager.go:143] "Creating topology manager with none policy" Oct 31 00:15:07.594818 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.594793 5222 container_manager_linux.go:311] "Creating device plugin manager" Oct 31 00:15:07.594818 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.594807 5222 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 00:15:07.594893 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.594822 5222 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 00:15:07.594938 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.594930 5222 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 31 00:15:07.595117 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.595099 5222 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 31 00:15:07.595246 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.595234 5222 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 31 00:15:07.595303 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.595286 5222 kubelet.go:493] "Attempting to sync node with API server" Oct 31 00:15:07.595360 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.595307 5222 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 31 00:15:07.595360 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.595322 5222 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 31 00:15:07.595360 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.595337 5222 kubelet.go:405] "Adding apiserver pod source" Oct 31 00:15:07.595360 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.595348 5222 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 31 00:15:07.596240 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.596221 5222 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 31 00:15:07.596240 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.596239 5222 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 00:15:07.596833 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.596811 5222 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 31 00:15:07.605464 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.605443 5222 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 31 00:15:07.605464 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.605461 5222 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 00:15:07.605611 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.605581 5222 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 31 00:15:07.605791 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.605780 5222 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 31 00:15:07.605821 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.605810 5222 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 31 00:15:07.605928 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.605913 5222 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 31 00:15:07.605928 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.605929 5222 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 31 00:15:07.606060 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.605935 5222 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 31 00:15:07.606060 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.605941 5222 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 31 00:15:07.606060 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.605947 5222 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 31 00:15:07.606060 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.605952 5222 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 31 00:15:07.606060 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.605958 5222 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 31 00:15:07.606060 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.605964 5222 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 31 00:15:07.606060 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.605971 5222 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 31 00:15:07.606060 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.605977 5222 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 31 00:15:07.606060 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.605985 5222 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 31 00:15:07.606060 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.605994 5222 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 31 00:15:07.606060 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.606019 5222 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 31 00:15:07.606060 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.606025 5222 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 31 00:15:07.609536 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.609511 5222 server.go:1263] "Started kubelet" Oct 31 00:15:07.609668 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.609607 5222 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 31 00:15:07.609721 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.609691 5222 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 31 00:15:07.609765 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.609619 5222 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 31 00:15:07.610664 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.609992 5222 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 31 00:15:07.610043 ip-10-0-137-56 systemd[1]: Started Kubernetes Kubelet. Oct 31 00:15:07.611357 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.611337 5222 server.go:317] "Adding debug handlers to kubelet server" Oct 31 00:15:07.614582 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.614550 5222 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 31 00:15:07.614684 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.614633 5222 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 31 00:15:07.614934 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.614873 5222 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 00:08:54 +0000 UTC" deadline="2028-05-14 03:34:35.21801394 +0000 UTC" Oct 31 00:15:07.615020 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.614935 5222 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13467h19m27.603083435s" Oct 31 00:15:07.615020 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.614961 5222 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 31 00:15:07.615020 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.614974 5222 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 31 00:15:07.615151 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.615093 5222 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 31 00:15:07.615496 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:15:07.615468 5222 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-56.us-west-2.compute.internal\" not found" Oct 31 00:15:07.617839 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.617820 5222 factory.go:55] Registering systemd factory Oct 31 00:15:07.617944 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.617841 5222 factory.go:223] Registration of the systemd container factory successfully Oct 31 00:15:07.618338 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.618284 5222 factory.go:153] Registering CRI-O factory Oct 31 00:15:07.618338 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.618306 5222 factory.go:223] Registration of the crio container factory successfully Oct 31 00:15:07.618507 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.618477 5222 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 31 00:15:07.618507 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.618504 5222 factory.go:103] Registering Raw factory Oct 31 00:15:07.618618 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.618518 5222 manager.go:1199] Started watching for new ooms in manager Oct 31 00:15:07.618772 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.618755 5222 manager.go:319] Starting recovery of all containers Oct 31 00:15:07.619442 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.619399 5222 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="26aba8fc-e2da-430f-b0d8-6ee45468441d" volumeName="kubernetes.io/configmap/26aba8fc-e2da-430f-b0d8-6ee45468441d-metrics-client-ca" seLinuxMountContext="" Oct 31 00:15:07.619442 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.619446 5222 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5c58f866-ef80-42d5-aab0-0cdefbc7a158" volumeName="kubernetes.io/configmap/5c58f866-ef80-42d5-aab0-0cdefbc7a158-trusted-ca" seLinuxMountContext="" Oct 31 00:15:07.619595 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.619462 5222 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="652defc0-301f-4679-8e49-6729381259dd" volumeName="kubernetes.io/projected/652defc0-301f-4679-8e49-6729381259dd-kube-api-access-jwf5l" seLinuxMountContext="" Oct 31 00:15:07.619595 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.619475 5222 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6a21f264-1e80-4eab-8c6e-27ca9ca19f0d" volumeName="kubernetes.io/configmap/6a21f264-1e80-4eab-8c6e-27ca9ca19f0d-cni-binary-copy" seLinuxMountContext="" Oct 31 00:15:07.619595 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.619487 5222 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a077dea4-b5a3-4335-bb6a-7618e794eb8f" volumeName="kubernetes.io/secret/a077dea4-b5a3-4335-bb6a-7618e794eb8f-metrics-certs" seLinuxMountContext="" Oct 31 00:15:07.619595 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.619499 5222 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a7d0a920-91fe-4c4a-b9bc-67550c324e7f" volumeName="kubernetes.io/empty-dir/a7d0a920-91fe-4c4a-b9bc-67550c324e7f-tmp-dir" seLinuxMountContext="" Oct 31 00:15:07.619595 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.619513 5222 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eadb6436-5e5b-4116-8ed7-26aa935fa749" volumeName="kubernetes.io/projected/eadb6436-5e5b-4116-8ed7-26aa935fa749-kube-api-access-nz8r2" seLinuxMountContext="" Oct 31 00:15:07.619595 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.619526 5222 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="97d11160-2114-4fbf-95c2-fa7b4c9d2260" volumeName="kubernetes.io/projected/97d11160-2114-4fbf-95c2-fa7b4c9d2260-kube-api-access-6pc97" seLinuxMountContext="" Oct 31 00:15:07.619595 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.619541 5222 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a077dea4-b5a3-4335-bb6a-7618e794eb8f" volumeName="kubernetes.io/projected/a077dea4-b5a3-4335-bb6a-7618e794eb8f-kube-api-access-xc9bh" seLinuxMountContext="" Oct 31 00:15:07.619595 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.619553 5222 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a7d0a920-91fe-4c4a-b9bc-67550c324e7f" volumeName="kubernetes.io/projected/a7d0a920-91fe-4c4a-b9bc-67550c324e7f-kube-api-access-djjgz" seLinuxMountContext="" Oct 31 00:15:07.619595 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.619581 5222 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d02ae6b5-4860-45b3-8d16-4a49531b77d4" volumeName="kubernetes.io/secret/d02ae6b5-4860-45b3-8d16-4a49531b77d4-original-pull-secret" seLinuxMountContext="" Oct 31 00:15:07.619595 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.619594 5222 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="26aba8fc-e2da-430f-b0d8-6ee45468441d" volumeName="kubernetes.io/secret/26aba8fc-e2da-430f-b0d8-6ee45468441d-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext="" Oct 31 00:15:07.620131 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.619607 5222 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0eb4888f-40a8-4aba-85d8-3c845382bd02" volumeName="kubernetes.io/empty-dir/0eb4888f-40a8-4aba-85d8-3c845382bd02-tmp" seLinuxMountContext="" Oct 31 00:15:07.620131 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.619620 5222 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1b88d3b9-589f-4f4e-b8a6-872f4fb5536b" volumeName="kubernetes.io/configmap/1b88d3b9-589f-4f4e-b8a6-872f4fb5536b-node-exporter-accelerators-collector-config" seLinuxMountContext="" Oct 31 00:15:07.620131 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.619642 5222 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1b88d3b9-589f-4f4e-b8a6-872f4fb5536b" volumeName="kubernetes.io/projected/1b88d3b9-589f-4f4e-b8a6-872f4fb5536b-kube-api-access-dqw94" seLinuxMountContext="" Oct 31 00:15:07.620131 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.619655 5222 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5c58f866-ef80-42d5-aab0-0cdefbc7a158" volumeName="kubernetes.io/secret/5c58f866-ef80-42d5-aab0-0cdefbc7a158-image-registry-private-configuration" seLinuxMountContext="" Oct 31 00:15:07.620131 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.619668 5222 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d22409bc-1bcf-46b1-9789-6f7298566a7f" volumeName="kubernetes.io/secret/d22409bc-1bcf-46b1-9789-6f7298566a7f-insights-runtime-extractor-tls" seLinuxMountContext="" Oct 31 00:15:07.620131 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.619681 5222 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ea9a5160-4696-4b75-8a10-0fbc54b2ad04" volumeName="kubernetes.io/secret/ea9a5160-4696-4b75-8a10-0fbc54b2ad04-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Oct 31 00:15:07.620131 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.619694 5222 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f8c3c03-a1fc-4376-98c6-feeeefa08068" volumeName="kubernetes.io/configmap/0f8c3c03-a1fc-4376-98c6-feeeefa08068-ovnkube-script-lib" seLinuxMountContext="" Oct 31 00:15:07.620131 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.619707 5222 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9801ff00-9e9a-4141-8212-ca32bf0251b3" volumeName="kubernetes.io/configmap/9801ff00-9e9a-4141-8212-ca32bf0251b3-multus-daemon-config" seLinuxMountContext="" Oct 31 00:15:07.620131 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.619719 5222 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="83d351d1-c6e7-43a6-a0fb-eaecefa8c5b2" volumeName="kubernetes.io/secret/83d351d1-c6e7-43a6-a0fb-eaecefa8c5b2-metrics-tls" seLinuxMountContext="" Oct 31 00:15:07.620131 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.619732 5222 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9801ff00-9e9a-4141-8212-ca32bf0251b3" volumeName="kubernetes.io/configmap/9801ff00-9e9a-4141-8212-ca32bf0251b3-cni-binary-copy" seLinuxMountContext="" Oct 31 00:15:07.620131 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.619744 5222 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="652defc0-301f-4679-8e49-6729381259dd" volumeName="kubernetes.io/configmap/652defc0-301f-4679-8e49-6729381259dd-metrics-client-ca" seLinuxMountContext="" Oct 31 00:15:07.620131 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.619756 5222 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d22409bc-1bcf-46b1-9789-6f7298566a7f" volumeName="kubernetes.io/projected/d22409bc-1bcf-46b1-9789-6f7298566a7f-kube-api-access-srgbh" seLinuxMountContext="" Oct 31 00:15:07.620131 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.619769 5222 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eadb6436-5e5b-4116-8ed7-26aa935fa749" volumeName="kubernetes.io/configmap/eadb6436-5e5b-4116-8ed7-26aa935fa749-serviceca" seLinuxMountContext="" Oct 31 00:15:07.620131 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.619782 5222 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f8c3c03-a1fc-4376-98c6-feeeefa08068" volumeName="kubernetes.io/secret/0f8c3c03-a1fc-4376-98c6-feeeefa08068-ovn-node-metrics-cert" seLinuxMountContext="" Oct 31 00:15:07.620131 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.619820 5222 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1b88d3b9-589f-4f4e-b8a6-872f4fb5536b" volumeName="kubernetes.io/empty-dir/1b88d3b9-589f-4f4e-b8a6-872f4fb5536b-node-exporter-textfile" seLinuxMountContext="" Oct 31 00:15:07.620131 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.619833 5222 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1b88d3b9-589f-4f4e-b8a6-872f4fb5536b" volumeName="kubernetes.io/secret/1b88d3b9-589f-4f4e-b8a6-872f4fb5536b-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Oct 31 00:15:07.620131 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.619851 5222 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="26aba8fc-e2da-430f-b0d8-6ee45468441d" volumeName="kubernetes.io/projected/26aba8fc-e2da-430f-b0d8-6ee45468441d-tls-assets" seLinuxMountContext="" Oct 31 00:15:07.620131 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.619864 5222 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ed1e615-cbca-482c-89cc-0fe7d8e7e597" volumeName="kubernetes.io/configmap/2ed1e615-cbca-482c-89cc-0fe7d8e7e597-konnectivity-ca" seLinuxMountContext="" Oct 31 00:15:07.620131 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.619879 5222 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5c58f866-ef80-42d5-aab0-0cdefbc7a158" volumeName="kubernetes.io/secret/5c58f866-ef80-42d5-aab0-0cdefbc7a158-installation-pull-secrets" seLinuxMountContext="" Oct 31 00:15:07.620131 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.619893 5222 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="978a9050-dddb-4869-85d8-84a001554c5b" volumeName="kubernetes.io/projected/978a9050-dddb-4869-85d8-84a001554c5b-kube-api-access-gbpb7" seLinuxMountContext="" Oct 31 00:15:07.620131 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.619906 5222 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1b88d3b9-589f-4f4e-b8a6-872f4fb5536b" volumeName="kubernetes.io/configmap/1b88d3b9-589f-4f4e-b8a6-872f4fb5536b-metrics-client-ca" seLinuxMountContext="" Oct 31 00:15:07.620131 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.619920 5222 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="26aba8fc-e2da-430f-b0d8-6ee45468441d" volumeName="kubernetes.io/projected/26aba8fc-e2da-430f-b0d8-6ee45468441d-kube-api-access-4v246" seLinuxMountContext="" Oct 31 00:15:07.620131 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.619933 5222 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ed1e615-cbca-482c-89cc-0fe7d8e7e597" volumeName="kubernetes.io/secret/2ed1e615-cbca-482c-89cc-0fe7d8e7e597-agent-certs" seLinuxMountContext="" Oct 31 00:15:07.620131 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.619947 5222 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="26aba8fc-e2da-430f-b0d8-6ee45468441d" volumeName="kubernetes.io/secret/26aba8fc-e2da-430f-b0d8-6ee45468441d-web-config" seLinuxMountContext="" Oct 31 00:15:07.620131 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.619960 5222 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d22409bc-1bcf-46b1-9789-6f7298566a7f" volumeName="kubernetes.io/configmap/d22409bc-1bcf-46b1-9789-6f7298566a7f-kube-rbac-proxy-cm" seLinuxMountContext="" Oct 31 00:15:07.620131 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.619974 5222 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="26aba8fc-e2da-430f-b0d8-6ee45468441d" volumeName="kubernetes.io/secret/26aba8fc-e2da-430f-b0d8-6ee45468441d-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext="" Oct 31 00:15:07.620131 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.619988 5222 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="652defc0-301f-4679-8e49-6729381259dd" volumeName="kubernetes.io/secret/652defc0-301f-4679-8e49-6729381259dd-openshift-state-metrics-tls" seLinuxMountContext="" Oct 31 00:15:07.620131 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.620001 5222 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="97d11160-2114-4fbf-95c2-fa7b4c9d2260" volumeName="kubernetes.io/secret/97d11160-2114-4fbf-95c2-fa7b4c9d2260-cert" seLinuxMountContext="" Oct 31 00:15:07.620131 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.620014 5222 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c61956a7-ba51-4063-bc25-aaa418168bc2" volumeName="kubernetes.io/projected/c61956a7-ba51-4063-bc25-aaa418168bc2-kube-api-access-cspld" seLinuxMountContext="" Oct 31 00:15:07.620131 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.620028 5222 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0eb4888f-40a8-4aba-85d8-3c845382bd02" volumeName="kubernetes.io/projected/0eb4888f-40a8-4aba-85d8-3c845382bd02-kube-api-access-nrkcn" seLinuxMountContext="" Oct 31 00:15:07.620131 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.620042 5222 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="26aba8fc-e2da-430f-b0d8-6ee45468441d" volumeName="kubernetes.io/secret/26aba8fc-e2da-430f-b0d8-6ee45468441d-cluster-tls-config" seLinuxMountContext="" Oct 31 00:15:07.620131 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.620055 5222 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="26aba8fc-e2da-430f-b0d8-6ee45468441d" volumeName="kubernetes.io/secret/26aba8fc-e2da-430f-b0d8-6ee45468441d-secret-alertmanager-main-tls" seLinuxMountContext="" Oct 31 00:15:07.620131 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.620066 5222 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5c58f866-ef80-42d5-aab0-0cdefbc7a158" volumeName="kubernetes.io/configmap/5c58f866-ef80-42d5-aab0-0cdefbc7a158-registry-certificates" seLinuxMountContext="" Oct 31 00:15:07.620131 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.620079 5222 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5c58f866-ef80-42d5-aab0-0cdefbc7a158" volumeName="kubernetes.io/projected/5c58f866-ef80-42d5-aab0-0cdefbc7a158-registry-tls" seLinuxMountContext="" Oct 31 00:15:07.620131 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.620094 5222 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c61956a7-ba51-4063-bc25-aaa418168bc2" volumeName="kubernetes.io/empty-dir/c61956a7-ba51-4063-bc25-aaa418168bc2-tmp" seLinuxMountContext="" Oct 31 00:15:07.620131 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.620107 5222 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d22409bc-1bcf-46b1-9789-6f7298566a7f" volumeName="kubernetes.io/empty-dir/d22409bc-1bcf-46b1-9789-6f7298566a7f-data-volume" seLinuxMountContext="" Oct 31 00:15:07.620131 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.620122 5222 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ea9a5160-4696-4b75-8a10-0fbc54b2ad04" volumeName="kubernetes.io/configmap/ea9a5160-4696-4b75-8a10-0fbc54b2ad04-metrics-client-ca" seLinuxMountContext="" Oct 31 00:15:07.620131 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.620134 5222 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1bc3d212-b67c-4368-90a3-8e080a4f1452" volumeName="kubernetes.io/projected/1bc3d212-b67c-4368-90a3-8e080a4f1452-kube-api-access-wm7wn" seLinuxMountContext="" Oct 31 00:15:07.620131 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.620148 5222 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5c58f866-ef80-42d5-aab0-0cdefbc7a158" volumeName="kubernetes.io/projected/5c58f866-ef80-42d5-aab0-0cdefbc7a158-bound-sa-token" seLinuxMountContext="" Oct 31 00:15:07.620131 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.620162 5222 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="83d351d1-c6e7-43a6-a0fb-eaecefa8c5b2" volumeName="kubernetes.io/empty-dir/83d351d1-c6e7-43a6-a0fb-eaecefa8c5b2-tmp-dir" seLinuxMountContext="" Oct 31 00:15:07.622039 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.620174 5222 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0cbf0042-931c-4135-9330-f9e798b7a567" volumeName="kubernetes.io/projected/0cbf0042-931c-4135-9330-f9e798b7a567-kube-api-access-ms4cs" seLinuxMountContext="" Oct 31 00:15:07.622039 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.620188 5222 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0eb4888f-40a8-4aba-85d8-3c845382bd02" volumeName="kubernetes.io/empty-dir/0eb4888f-40a8-4aba-85d8-3c845382bd02-etc-tuned" seLinuxMountContext="" Oct 31 00:15:07.622039 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.620201 5222 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f8c3c03-a1fc-4376-98c6-feeeefa08068" volumeName="kubernetes.io/configmap/0f8c3c03-a1fc-4376-98c6-feeeefa08068-env-overrides" seLinuxMountContext="" Oct 31 00:15:07.622039 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.620214 5222 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f8c3c03-a1fc-4376-98c6-feeeefa08068" volumeName="kubernetes.io/projected/0f8c3c03-a1fc-4376-98c6-feeeefa08068-kube-api-access-8wrw4" seLinuxMountContext="" Oct 31 00:15:07.622039 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.620237 5222 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="26aba8fc-e2da-430f-b0d8-6ee45468441d" volumeName="kubernetes.io/empty-dir/26aba8fc-e2da-430f-b0d8-6ee45468441d-alertmanager-main-db" seLinuxMountContext="" Oct 31 00:15:07.622039 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.620249 5222 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="26aba8fc-e2da-430f-b0d8-6ee45468441d" volumeName="kubernetes.io/secret/26aba8fc-e2da-430f-b0d8-6ee45468441d-config-volume" seLinuxMountContext="" Oct 31 00:15:07.622039 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.620261 5222 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5c58f866-ef80-42d5-aab0-0cdefbc7a158" volumeName="kubernetes.io/empty-dir/5c58f866-ef80-42d5-aab0-0cdefbc7a158-ca-trust-extracted" seLinuxMountContext="" Oct 31 00:15:07.622039 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.620274 5222 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5c58f866-ef80-42d5-aab0-0cdefbc7a158" volumeName="kubernetes.io/projected/5c58f866-ef80-42d5-aab0-0cdefbc7a158-kube-api-access-n5hld" seLinuxMountContext="" Oct 31 00:15:07.622039 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.620289 5222 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="26aba8fc-e2da-430f-b0d8-6ee45468441d" volumeName="kubernetes.io/configmap/26aba8fc-e2da-430f-b0d8-6ee45468441d-alertmanager-trusted-ca-bundle" seLinuxMountContext="" Oct 31 00:15:07.622039 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.620302 5222 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="26aba8fc-e2da-430f-b0d8-6ee45468441d" volumeName="kubernetes.io/empty-dir/26aba8fc-e2da-430f-b0d8-6ee45468441d-config-out" seLinuxMountContext="" Oct 31 00:15:07.622039 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.620315 5222 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="652defc0-301f-4679-8e49-6729381259dd" volumeName="kubernetes.io/secret/652defc0-301f-4679-8e49-6729381259dd-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Oct 31 00:15:07.622039 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.620327 5222 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6a21f264-1e80-4eab-8c6e-27ca9ca19f0d" volumeName="kubernetes.io/configmap/6a21f264-1e80-4eab-8c6e-27ca9ca19f0d-cni-sysctl-allowlist" seLinuxMountContext="" Oct 31 00:15:07.622039 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.620340 5222 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6a21f264-1e80-4eab-8c6e-27ca9ca19f0d" volumeName="kubernetes.io/configmap/6a21f264-1e80-4eab-8c6e-27ca9ca19f0d-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 31 00:15:07.622039 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.620353 5222 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6a21f264-1e80-4eab-8c6e-27ca9ca19f0d" volumeName="kubernetes.io/projected/6a21f264-1e80-4eab-8c6e-27ca9ca19f0d-kube-api-access-8j6tz" seLinuxMountContext="" Oct 31 00:15:07.622039 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.620381 5222 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9801ff00-9e9a-4141-8212-ca32bf0251b3" volumeName="kubernetes.io/projected/9801ff00-9e9a-4141-8212-ca32bf0251b3-kube-api-access-xzs4x" seLinuxMountContext="" Oct 31 00:15:07.622039 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.620394 5222 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ea9a5160-4696-4b75-8a10-0fbc54b2ad04" volumeName="kubernetes.io/projected/ea9a5160-4696-4b75-8a10-0fbc54b2ad04-kube-api-access-v7hn4" seLinuxMountContext="" Oct 31 00:15:07.622039 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.620404 5222 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="26aba8fc-e2da-430f-b0d8-6ee45468441d" volumeName="kubernetes.io/secret/26aba8fc-e2da-430f-b0d8-6ee45468441d-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext="" Oct 31 00:15:07.622039 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.620411 5222 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="83d351d1-c6e7-43a6-a0fb-eaecefa8c5b2" volumeName="kubernetes.io/configmap/83d351d1-c6e7-43a6-a0fb-eaecefa8c5b2-config-volume" seLinuxMountContext="" Oct 31 00:15:07.622039 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.620420 5222 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f8c3c03-a1fc-4376-98c6-feeeefa08068" volumeName="kubernetes.io/configmap/0f8c3c03-a1fc-4376-98c6-feeeefa08068-ovnkube-config" seLinuxMountContext="" Oct 31 00:15:07.622039 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.620434 5222 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1b88d3b9-589f-4f4e-b8a6-872f4fb5536b" volumeName="kubernetes.io/secret/1b88d3b9-589f-4f4e-b8a6-872f4fb5536b-node-exporter-tls" seLinuxMountContext="" Oct 31 00:15:07.622039 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.620445 5222 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ea9a5160-4696-4b75-8a10-0fbc54b2ad04" volumeName="kubernetes.io/secret/ea9a5160-4696-4b75-8a10-0fbc54b2ad04-kube-state-metrics-tls" seLinuxMountContext="" Oct 31 00:15:07.622039 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.620457 5222 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="83d351d1-c6e7-43a6-a0fb-eaecefa8c5b2" volumeName="kubernetes.io/projected/83d351d1-c6e7-43a6-a0fb-eaecefa8c5b2-kube-api-access-5lsrt" seLinuxMountContext="" Oct 31 00:15:07.622039 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.620469 5222 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="978a9050-dddb-4869-85d8-84a001554c5b" volumeName="kubernetes.io/configmap/978a9050-dddb-4869-85d8-84a001554c5b-iptables-alerter-script" seLinuxMountContext="" Oct 31 00:15:07.622039 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.620482 5222 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ea9a5160-4696-4b75-8a10-0fbc54b2ad04" volumeName="kubernetes.io/configmap/ea9a5160-4696-4b75-8a10-0fbc54b2ad04-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext="" Oct 31 00:15:07.622039 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.620493 5222 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ea9a5160-4696-4b75-8a10-0fbc54b2ad04" volumeName="kubernetes.io/empty-dir/ea9a5160-4696-4b75-8a10-0fbc54b2ad04-volume-directive-shadow" seLinuxMountContext="" Oct 31 00:15:07.622039 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.620501 5222 reconstruct.go:97] "Volume reconstruction finished" Oct 31 00:15:07.622039 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.620507 5222 reconciler.go:29] "Reconciler: start to sync state" Oct 31 00:15:07.639016 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.638976 5222 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h9hsf_0f8c3c03-a1fc-4376-98c6-feeeefa08068/ovn-acl-logging/0.log" Oct 31 00:15:07.642591 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.642562 5222 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h9hsf_0f8c3c03-a1fc-4376-98c6-feeeefa08068/ovn-acl-logging/0.log" Oct 31 00:15:07.646579 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.646535 5222 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 31 00:15:07.647893 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.647874 5222 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 31 00:15:07.647893 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.647895 5222 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 31 00:15:07.648071 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.647919 5222 kubelet.go:2521] "Starting kubelet main sync loop" Oct 31 00:15:07.648071 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:15:07.647962 5222 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 31 00:15:07.665543 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.665469 5222 generic.go:356] "Generic (PLEG): container finished" podID="f0db3b99b7954e19a6b4da51857943e0" containerID="2c03b055a0212025b10c111f4d14a1de5905eb9e16e87dffd9dd16492c9cde1b" exitCode=0 Oct 31 00:15:07.667080 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.667052 5222 generic.go:356] "Generic (PLEG): container finished" podID="1b88d3b9-589f-4f4e-b8a6-872f4fb5536b" containerID="2e0c1643151cb96f6291d00f280436693f66d88fa4bae34159ad9b46c0358113" exitCode=0 Oct 31 00:15:07.675873 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.675848 5222 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h9hsf_0f8c3c03-a1fc-4376-98c6-feeeefa08068/ovn-acl-logging/0.log" Oct 31 00:15:07.676272 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.676241 5222 generic.go:356] "Generic (PLEG): container finished" podID="0f8c3c03-a1fc-4376-98c6-feeeefa08068" containerID="e809bac0da7017764d3d8602eb7c48a7076f0a38548a4d2eb49344ce7cff62c6" exitCode=1 Oct 31 00:15:07.695094 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.695061 5222 generic.go:356] "Generic (PLEG): container finished" podID="6a21f264-1e80-4eab-8c6e-27ca9ca19f0d" containerID="440774ebb1e3724120772862c008f21d5fd77338c168c6fc0def1b9c97cfc0e5" exitCode=0 Oct 31 00:15:07.695094 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.695089 5222 generic.go:356] "Generic (PLEG): container finished" podID="6a21f264-1e80-4eab-8c6e-27ca9ca19f0d" containerID="9c9530986ffadecb481dc14c4227502352d72f18ed883588fc22c6909388577e" exitCode=0 Oct 31 00:15:07.695094 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.695096 5222 generic.go:356] "Generic (PLEG): container finished" podID="6a21f264-1e80-4eab-8c6e-27ca9ca19f0d" containerID="c795df5e107ca603cfcc0faff37467699bbf3e7b6bb9b8a695e9b270f5deeaa2" exitCode=0 Oct 31 00:15:07.695094 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.695102 5222 generic.go:356] "Generic (PLEG): container finished" podID="6a21f264-1e80-4eab-8c6e-27ca9ca19f0d" containerID="930d8f45dd17e51267539e5fa1767003a4800c189d8076391923826c00d17121" exitCode=0 Oct 31 00:15:07.695094 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.695107 5222 generic.go:356] "Generic (PLEG): container finished" podID="6a21f264-1e80-4eab-8c6e-27ca9ca19f0d" containerID="52415b55a41cf87669b44191c47b1cde246de5060df99a6510cc3064d3d014e3" exitCode=0 Oct 31 00:15:07.695480 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.695112 5222 generic.go:356] "Generic (PLEG): container finished" podID="6a21f264-1e80-4eab-8c6e-27ca9ca19f0d" containerID="fcdf3d9574e31e0b3ce80efda5296f44c4b2d23a51348c64a8650eed1c376191" exitCode=0 Oct 31 00:15:07.708853 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.708832 5222 manager.go:324] Recovery completed Oct 31 00:15:07.715551 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:15:07.715522 5222 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-56.us-west-2.compute.internal\" not found" Oct 31 00:15:07.727386 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.727340 5222 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 00:15:07.730337 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.730315 5222 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-56.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 00:15:07.730473 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.730347 5222 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-56.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 00:15:07.730473 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.730358 5222 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-56.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 31 00:15:07.735659 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.735637 5222 cpu_manager.go:226] "Starting" policy="none" Oct 31 00:15:07.735659 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.735652 5222 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 31 00:15:07.735659 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.735658 5222 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 00:15:07.735898 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.735675 5222 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 31 00:15:07.735898 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.735815 5222 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 31 00:15:07.735898 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.735830 5222 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 31 00:15:07.735898 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.735851 5222 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 31 00:15:07.735898 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.735857 5222 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 31 00:15:07.735898 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.735862 5222 policy_none.go:50] "Start" Oct 31 00:15:07.735898 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.735871 5222 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 31 00:15:07.735898 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.735880 5222 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 31 00:15:07.736864 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.736845 5222 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-56.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 00:15:07.736953 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.736883 5222 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-56.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 00:15:07.736953 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.736899 5222 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-56.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 31 00:15:07.737228 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.737214 5222 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 31 00:15:07.737273 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.737231 5222 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 31 00:15:07.737273 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.737238 5222 policy_none.go:44] "Start" Oct 31 00:15:07.741984 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.741952 5222 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 31 00:15:07.745846 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.745831 5222 manager.go:341] "Starting Device Plugin manager" Oct 31 00:15:07.745943 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:15:07.745862 5222 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 31 00:15:07.745943 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.745874 5222 server.go:87] "Starting device plugin registration server" Oct 31 00:15:07.746198 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.746184 5222 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 31 00:15:07.746261 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.746199 5222 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 31 00:15:07.746305 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.746282 5222 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 31 00:15:07.746439 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.746399 5222 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 31 00:15:07.746439 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.746443 5222 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 31 00:15:07.746608 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.746449 5222 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 31 00:15:07.746608 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.746505 5222 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T00:15:07.746432531Z","UUID":"be2c791d-bb28-4b74-8596-a0b26937fd57","Handler":null,"Name":"","Endpoint":""} Oct 31 00:15:07.748204 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.748161 5222 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-56.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-56.us-west-2.compute.internal"] Oct 31 00:15:07.748315 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:15:07.748233 5222 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 31 00:15:07.748395 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:15:07.748347 5222 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-56.us-west-2.compute.internal\" not found" Oct 31 00:15:07.748467 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.748358 5222 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 31 00:15:07.748467 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.748420 5222 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 31 00:15:07.748580 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.748516 5222 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 00:15:07.749987 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.749965 5222 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-56.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 00:15:07.750124 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.750006 5222 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-56.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 00:15:07.750124 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.750024 5222 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-56.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 31 00:15:07.750599 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.750577 5222 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 00:15:07.750811 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.750793 5222 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 00:15:07.754733 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.754713 5222 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-56.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 00:15:07.754830 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.754744 5222 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-56.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 00:15:07.754830 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.754758 5222 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-56.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 31 00:15:07.754830 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.754786 5222 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-56.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 00:15:07.754830 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.754810 5222 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-56.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 00:15:07.754830 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.754824 5222 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-56.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 31 00:15:07.755058 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:15:07.754922 5222 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-56.us-west-2.compute.internal\" not found" node="ip-10-0-137-56.us-west-2.compute.internal" Oct 31 00:15:07.755147 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.755113 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-56.us-west-2.compute.internal" event={"ID":"f0db3b99b7954e19a6b4da51857943e0","Type":"ContainerStarted","Data":"5dffef95bb6a69122bd4e904e0470bd2df4e0f09c15d0bf8f6008912321e57f1"} Oct 31 00:15:07.755178 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.755156 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-56.us-west-2.compute.internal" event={"ID":"f0db3b99b7954e19a6b4da51857943e0","Type":"ContainerDied","Data":"2c03b055a0212025b10c111f4d14a1de5905eb9e16e87dffd9dd16492c9cde1b"} Oct 31 00:15:07.755178 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.755171 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-56.us-west-2.compute.internal" event={"ID":"f0db3b99b7954e19a6b4da51857943e0","Type":"ContainerStarted","Data":"2d6531cabcd80da4099819572fe53a24f4806363c480abd15f54c08134300f74"} Oct 31 00:15:07.755230 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.755220 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-56.us-west-2.compute.internal" event={"ID":"e15cd53a0cf50b999ecad893fad25259","Type":"ContainerStarted","Data":"f3b77481bf449c6e6b4bdb7a5cea467c340f76f475aa912f461f3c205b5789bb"} Oct 31 00:15:07.755271 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.755237 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-56.us-west-2.compute.internal" event={"ID":"e15cd53a0cf50b999ecad893fad25259","Type":"ContainerStarted","Data":"7eae82e9ea1bf2459313e7139b9f222797c511f08b4b8774861dcd838dd46ad9"} Oct 31 00:15:07.755305 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.755276 5222 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 00:15:07.756105 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.756087 5222 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-56.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 00:15:07.756184 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.756114 5222 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-56.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 00:15:07.756184 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.756125 5222 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-56.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 31 00:15:07.756264 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:15:07.756252 5222 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-56.us-west-2.compute.internal\" not found" node="ip-10-0-137-56.us-west-2.compute.internal" Oct 31 00:15:07.813236 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.813202 5222 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:15:07.813402 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.813272 5222 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 31 00:15:07.814565 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.814550 5222 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:15:07.816342 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.816327 5222 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:15:07.846324 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.846292 5222 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 00:15:07.847422 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.847407 5222 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-56.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 00:15:07.847494 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.847437 5222 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-56.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 00:15:07.847494 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.847447 5222 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-56.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 31 00:15:07.847494 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.847490 5222 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-137-56.us-west-2.compute.internal" Oct 31 00:15:07.914005 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.913968 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f0db3b99b7954e19a6b4da51857943e0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-56.us-west-2.compute.internal\" (UID: \"f0db3b99b7954e19a6b4da51857943e0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-56.us-west-2.compute.internal" Oct 31 00:15:07.914005 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.914004 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f0db3b99b7954e19a6b4da51857943e0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-56.us-west-2.compute.internal\" (UID: \"f0db3b99b7954e19a6b4da51857943e0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-56.us-west-2.compute.internal" Oct 31 00:15:07.914179 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.914030 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e15cd53a0cf50b999ecad893fad25259-config\") pod \"kube-apiserver-proxy-ip-10-0-137-56.us-west-2.compute.internal\" (UID: \"e15cd53a0cf50b999ecad893fad25259\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-56.us-west-2.compute.internal" Oct 31 00:15:07.915763 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.915697 5222 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-56.us-west-2.compute.internal" Oct 31 00:15:07.915869 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.915848 5222 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:15:07.971166 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.971137 5222 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-137-56.us-west-2.compute.internal" Oct 31 00:15:07.971348 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.971232 5222 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-137-56.us-west-2.compute.internal" Oct 31 00:15:07.971883 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.971858 5222 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-56.us-west-2.compute.internal" event="NodeNotReady" Oct 31 00:15:07.971945 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.971891 5222 setters.go:546] "Node became not ready" node="ip-10-0-137-56.us-west-2.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-10-31T00:15:07Z","lastTransitionTime":"2026-10-31T00:15:07Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"} Oct 31 00:15:07.985219 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.985192 5222 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 00:15:07.985391 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:15:07.985247 5222 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-137-56.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-56.us-west-2.compute.internal" Oct 31 00:15:07.985391 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:07.985265 5222 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-56.us-west-2.compute.internal" Oct 31 00:15:08.014444 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.014414 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f0db3b99b7954e19a6b4da51857943e0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-56.us-west-2.compute.internal\" (UID: \"f0db3b99b7954e19a6b4da51857943e0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-56.us-west-2.compute.internal" Oct 31 00:15:08.014444 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.014434 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f0db3b99b7954e19a6b4da51857943e0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-56.us-west-2.compute.internal\" (UID: \"f0db3b99b7954e19a6b4da51857943e0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-56.us-west-2.compute.internal" Oct 31 00:15:08.014585 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.014468 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f0db3b99b7954e19a6b4da51857943e0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-56.us-west-2.compute.internal\" (UID: \"f0db3b99b7954e19a6b4da51857943e0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-56.us-west-2.compute.internal" Oct 31 00:15:08.014585 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.014491 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e15cd53a0cf50b999ecad893fad25259-config\") pod \"kube-apiserver-proxy-ip-10-0-137-56.us-west-2.compute.internal\" (UID: \"e15cd53a0cf50b999ecad893fad25259\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-56.us-west-2.compute.internal" Oct 31 00:15:08.014585 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.014525 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f0db3b99b7954e19a6b4da51857943e0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-56.us-west-2.compute.internal\" (UID: \"f0db3b99b7954e19a6b4da51857943e0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-56.us-west-2.compute.internal" Oct 31 00:15:08.014585 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.014538 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e15cd53a0cf50b999ecad893fad25259-config\") pod \"kube-apiserver-proxy-ip-10-0-137-56.us-west-2.compute.internal\" (UID: \"e15cd53a0cf50b999ecad893fad25259\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-56.us-west-2.compute.internal" Oct 31 00:15:08.035143 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.035113 5222 nodeinfomanager.go:471] existing CSINode "ip-10-0-137-56.us-west-2.compute.internal" is owned by different node (oldNodeID="fce9b99e-a31b-45e6-8194-66c9ce8e8069", newNodeID=""), cleaning up... Oct 31 00:15:08.086011 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.085964 5222 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 00:15:08.086157 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:15:08.086025 5222 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-137-56.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-56.us-west-2.compute.internal" Oct 31 00:15:08.345441 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.345413 5222 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-56.us-west-2.compute.internal" event="NodeReady" Oct 31 00:15:08.345609 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.345512 5222 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 31 00:15:08.595850 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.595770 5222 apiserver.go:52] "Watching apiserver" Oct 31 00:15:08.710916 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.710891 5222 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-56.us-west-2.compute.internal" Oct 31 00:15:08.711096 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.711019 5222 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-56.us-west-2.compute.internal" Oct 31 00:15:08.786660 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.786627 5222 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 00:15:08.786829 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:15:08.786700 5222 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-137-56.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-56.us-west-2.compute.internal" Oct 31 00:15:08.787019 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.786999 5222 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 00:15:08.787078 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:15:08.787051 5222 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-137-56.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-56.us-west-2.compute.internal" Oct 31 00:15:08.788906 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.788882 5222 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 31 00:15:08.789474 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.789454 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-4nwn7","openshift-dns/node-resolver-gkm2c","openshift-monitoring/node-exporter-dbcsc","openshift-network-diagnostics/network-check-target-ml8n5","kube-system/konnectivity-agent-lvp2h","openshift-monitoring/kube-state-metrics-5dc8d7d89b-rsf6d","openshift-monitoring/openshift-state-metrics-7bd7489957-xd6zl","openshift-ovn-kubernetes/ovnkube-node-h9hsf","kube-system/global-pull-secret-syncer-fwgtx","openshift-insights/insights-runtime-extractor-nq56p","kube-system/kube-apiserver-proxy-ip-10-0-137-56.us-west-2.compute.internal","openshift-multus/network-metrics-daemon-rjfn2","openshift-console/downloads-6dd789c8b7-nbn2h","openshift-monitoring/alertmanager-main-0","openshift-cluster-node-tuning-operator/tuned-vsfrc","openshift-image-registry/image-registry-6644876fb8-9bszd","openshift-multus/multus-additional-cni-plugins-jsctc","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-56.us-west-2.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ndd7n","openshift-dns/dns-default-lrh4r","openshift-multus/multus-b6vpd","openshift-ingress-canary/ingress-canary-x5s75","openshift-image-registry/node-ca-xlzv2"] Oct 31 00:15:08.816865 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.816833 5222 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 31 00:15:08.818515 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.818493 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1bc3d212-b67c-4368-90a3-8e080a4f1452-device-dir\") pod \"aws-ebs-csi-driver-node-ndd7n\" (UID: \"1bc3d212-b67c-4368-90a3-8e080a4f1452\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ndd7n" Oct 31 00:15:08.818572 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.818528 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzs4x\" (UniqueName: \"kubernetes.io/projected/9801ff00-9e9a-4141-8212-ca32bf0251b3-kube-api-access-xzs4x\") pod \"multus-b6vpd\" (UID: \"9801ff00-9e9a-4141-8212-ca32bf0251b3\") " pod="openshift-multus/multus-b6vpd" Oct 31 00:15:08.818572 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.818546 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/83d351d1-c6e7-43a6-a0fb-eaecefa8c5b2-config-volume\") pod \"dns-default-lrh4r\" (UID: \"83d351d1-c6e7-43a6-a0fb-eaecefa8c5b2\") " pod="openshift-dns/dns-default-lrh4r" Oct 31 00:15:08.818659 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.818563 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ea9a5160-4696-4b75-8a10-0fbc54b2ad04-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-rsf6d\" (UID: \"ea9a5160-4696-4b75-8a10-0fbc54b2ad04\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rsf6d" Oct 31 00:15:08.818659 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.818629 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1bc3d212-b67c-4368-90a3-8e080a4f1452-registration-dir\") pod \"aws-ebs-csi-driver-node-ndd7n\" (UID: \"1bc3d212-b67c-4368-90a3-8e080a4f1452\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ndd7n" Oct 31 00:15:08.818750 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.818680 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9801ff00-9e9a-4141-8212-ca32bf0251b3-cnibin\") pod \"multus-b6vpd\" (UID: \"9801ff00-9e9a-4141-8212-ca32bf0251b3\") " pod="openshift-multus/multus-b6vpd" Oct 31 00:15:08.818800 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.818789 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d22409bc-1bcf-46b1-9789-6f7298566a7f-data-volume\") pod \"insights-runtime-extractor-nq56p\" (UID: \"d22409bc-1bcf-46b1-9789-6f7298566a7f\") " pod="openshift-insights/insights-runtime-extractor-nq56p" Oct 31 00:15:08.818849 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.818825 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6a21f264-1e80-4eab-8c6e-27ca9ca19f0d-cni-binary-copy\") pod \"multus-additional-cni-plugins-jsctc\" (UID: \"6a21f264-1e80-4eab-8c6e-27ca9ca19f0d\") " pod="openshift-multus/multus-additional-cni-plugins-jsctc" Oct 31 00:15:08.818880 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.818848 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0eb4888f-40a8-4aba-85d8-3c845382bd02-sys\") pod \"tuned-vsfrc\" (UID: \"0eb4888f-40a8-4aba-85d8-3c845382bd02\") " pod="openshift-cluster-node-tuning-operator/tuned-vsfrc" Oct 31 00:15:08.818955 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.818935 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/ea9a5160-4696-4b75-8a10-0fbc54b2ad04-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-rsf6d\" (UID: \"ea9a5160-4696-4b75-8a10-0fbc54b2ad04\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rsf6d" Oct 31 00:15:08.818955 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.818938 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d22409bc-1bcf-46b1-9789-6f7298566a7f-data-volume\") pod \"insights-runtime-extractor-nq56p\" (UID: \"d22409bc-1bcf-46b1-9789-6f7298566a7f\") " pod="openshift-insights/insights-runtime-extractor-nq56p" Oct 31 00:15:08.819021 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.818980 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ms4cs\" (UniqueName: \"kubernetes.io/projected/0cbf0042-931c-4135-9330-f9e798b7a567-kube-api-access-ms4cs\") pod \"network-check-target-ml8n5\" (UID: \"0cbf0042-931c-4135-9330-f9e798b7a567\") " pod="openshift-network-diagnostics/network-check-target-ml8n5" Oct 31 00:15:08.819055 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.819008 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6a21f264-1e80-4eab-8c6e-27ca9ca19f0d-system-cni-dir\") pod \"multus-additional-cni-plugins-jsctc\" (UID: \"6a21f264-1e80-4eab-8c6e-27ca9ca19f0d\") " pod="openshift-multus/multus-additional-cni-plugins-jsctc" Oct 31 00:15:08.819055 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.819042 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0eb4888f-40a8-4aba-85d8-3c845382bd02-etc-tuned\") pod \"tuned-vsfrc\" (UID: \"0eb4888f-40a8-4aba-85d8-3c845382bd02\") " pod="openshift-cluster-node-tuning-operator/tuned-vsfrc" Oct 31 00:15:08.819119 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.819082 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0f8c3c03-a1fc-4376-98c6-feeeefa08068-host-cni-bin\") pod \"ovnkube-node-h9hsf\" (UID: \"0f8c3c03-a1fc-4376-98c6-feeeefa08068\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:15:08.819148 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.819140 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0eb4888f-40a8-4aba-85d8-3c845382bd02-etc-tuned\") pod \"tuned-vsfrc\" (UID: \"0eb4888f-40a8-4aba-85d8-3c845382bd02\") " pod="openshift-cluster-node-tuning-operator/tuned-vsfrc" Oct 31 00:15:08.819185 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.819157 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wrw4\" (UniqueName: \"kubernetes.io/projected/0f8c3c03-a1fc-4376-98c6-feeeefa08068-kube-api-access-8wrw4\") pod \"ovnkube-node-h9hsf\" (UID: \"0f8c3c03-a1fc-4376-98c6-feeeefa08068\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:15:08.819222 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.819189 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ea9a5160-4696-4b75-8a10-0fbc54b2ad04-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-rsf6d\" (UID: \"ea9a5160-4696-4b75-8a10-0fbc54b2ad04\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rsf6d" Oct 31 00:15:08.819260 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.819223 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9801ff00-9e9a-4141-8212-ca32bf0251b3-multus-daemon-config\") pod \"multus-b6vpd\" (UID: \"9801ff00-9e9a-4141-8212-ca32bf0251b3\") " pod="openshift-multus/multus-b6vpd" Oct 31 00:15:08.819260 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.819250 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6a21f264-1e80-4eab-8c6e-27ca9ca19f0d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jsctc\" (UID: \"6a21f264-1e80-4eab-8c6e-27ca9ca19f0d\") " pod="openshift-multus/multus-additional-cni-plugins-jsctc" Oct 31 00:15:08.819321 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.819276 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0f8c3c03-a1fc-4376-98c6-feeeefa08068-run-systemd\") pod \"ovnkube-node-h9hsf\" (UID: \"0f8c3c03-a1fc-4376-98c6-feeeefa08068\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:15:08.819321 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.819301 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/978a9050-dddb-4869-85d8-84a001554c5b-iptables-alerter-script\") pod \"iptables-alerter-4nwn7\" (UID: \"978a9050-dddb-4869-85d8-84a001554c5b\") " pod="openshift-network-operator/iptables-alerter-4nwn7" Oct 31 00:15:08.819417 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.819325 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cspld\" (UniqueName: \"kubernetes.io/projected/c61956a7-ba51-4063-bc25-aaa418168bc2-kube-api-access-cspld\") pod \"downloads-6dd789c8b7-nbn2h\" (UID: \"c61956a7-ba51-4063-bc25-aaa418168bc2\") " pod="openshift-console/downloads-6dd789c8b7-nbn2h" Oct 31 00:15:08.819417 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.819402 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/26aba8fc-e2da-430f-b0d8-6ee45468441d-config-volume\") pod \"alertmanager-main-0\" (UID: \"26aba8fc-e2da-430f-b0d8-6ee45468441d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:15:08.819473 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.819422 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0f8c3c03-a1fc-4376-98c6-feeeefa08068-host-run-ovn-kubernetes\") pod \"ovnkube-node-h9hsf\" (UID: \"0f8c3c03-a1fc-4376-98c6-feeeefa08068\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:15:08.819473 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.819438 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9801ff00-9e9a-4141-8212-ca32bf0251b3-host-run-netns\") pod \"multus-b6vpd\" (UID: \"9801ff00-9e9a-4141-8212-ca32bf0251b3\") " pod="openshift-multus/multus-b6vpd" Oct 31 00:15:08.819473 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.819457 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/26aba8fc-e2da-430f-b0d8-6ee45468441d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"26aba8fc-e2da-430f-b0d8-6ee45468441d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:15:08.819571 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.819514 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/eadb6436-5e5b-4116-8ed7-26aa935fa749-serviceca\") pod \"node-ca-xlzv2\" (UID: \"eadb6436-5e5b-4116-8ed7-26aa935fa749\") " pod="openshift-image-registry/node-ca-xlzv2" Oct 31 00:15:08.819571 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.819531 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrkcn\" (UniqueName: \"kubernetes.io/projected/0eb4888f-40a8-4aba-85d8-3c845382bd02-kube-api-access-nrkcn\") pod \"tuned-vsfrc\" (UID: \"0eb4888f-40a8-4aba-85d8-3c845382bd02\") " pod="openshift-cluster-node-tuning-operator/tuned-vsfrc" Oct 31 00:15:08.819571 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.819548 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0f8c3c03-a1fc-4376-98c6-feeeefa08068-ovnkube-script-lib\") pod \"ovnkube-node-h9hsf\" (UID: \"0f8c3c03-a1fc-4376-98c6-feeeefa08068\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:15:08.819571 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.819567 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5c58f866-ef80-42d5-aab0-0cdefbc7a158-registry-tls\") pod \"image-registry-6644876fb8-9bszd\" (UID: \"5c58f866-ef80-42d5-aab0-0cdefbc7a158\") " pod="openshift-image-registry/image-registry-6644876fb8-9bszd" Oct 31 00:15:08.819753 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.819606 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nz8r2\" (UniqueName: \"kubernetes.io/projected/eadb6436-5e5b-4116-8ed7-26aa935fa749-kube-api-access-nz8r2\") pod \"node-ca-xlzv2\" (UID: \"eadb6436-5e5b-4116-8ed7-26aa935fa749\") " pod="openshift-image-registry/node-ca-xlzv2" Oct 31 00:15:08.819753 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.819649 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1bc3d212-b67c-4368-90a3-8e080a4f1452-socket-dir\") pod \"aws-ebs-csi-driver-node-ndd7n\" (UID: \"1bc3d212-b67c-4368-90a3-8e080a4f1452\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ndd7n" Oct 31 00:15:08.819753 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.819667 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1bc3d212-b67c-4368-90a3-8e080a4f1452-etc-selinux\") pod \"aws-ebs-csi-driver-node-ndd7n\" (UID: \"1bc3d212-b67c-4368-90a3-8e080a4f1452\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ndd7n" Oct 31 00:15:08.819753 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.819683 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9801ff00-9e9a-4141-8212-ca32bf0251b3-multus-cni-dir\") pod \"multus-b6vpd\" (UID: \"9801ff00-9e9a-4141-8212-ca32bf0251b3\") " pod="openshift-multus/multus-b6vpd" Oct 31 00:15:08.819753 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.819704 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d22409bc-1bcf-46b1-9789-6f7298566a7f-crio-socket\") pod \"insights-runtime-extractor-nq56p\" (UID: \"d22409bc-1bcf-46b1-9789-6f7298566a7f\") " pod="openshift-insights/insights-runtime-extractor-nq56p" Oct 31 00:15:08.819753 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.819736 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djjgz\" (UniqueName: \"kubernetes.io/projected/a7d0a920-91fe-4c4a-b9bc-67550c324e7f-kube-api-access-djjgz\") pod \"node-resolver-gkm2c\" (UID: \"a7d0a920-91fe-4c4a-b9bc-67550c324e7f\") " pod="openshift-dns/node-resolver-gkm2c" Oct 31 00:15:08.819931 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.819767 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0f8c3c03-a1fc-4376-98c6-feeeefa08068-host-run-netns\") pod \"ovnkube-node-h9hsf\" (UID: \"0f8c3c03-a1fc-4376-98c6-feeeefa08068\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:15:08.819931 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.819856 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1b88d3b9-589f-4f4e-b8a6-872f4fb5536b-node-exporter-tls\") pod \"node-exporter-dbcsc\" (UID: \"1b88d3b9-589f-4f4e-b8a6-872f4fb5536b\") " pod="openshift-monitoring/node-exporter-dbcsc" Oct 31 00:15:08.819931 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.819924 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7hn4\" (UniqueName: \"kubernetes.io/projected/ea9a5160-4696-4b75-8a10-0fbc54b2ad04-kube-api-access-v7hn4\") pod \"kube-state-metrics-5dc8d7d89b-rsf6d\" (UID: \"ea9a5160-4696-4b75-8a10-0fbc54b2ad04\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rsf6d" Oct 31 00:15:08.820027 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.819945 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wm7wn\" (UniqueName: \"kubernetes.io/projected/1bc3d212-b67c-4368-90a3-8e080a4f1452-kube-api-access-wm7wn\") pod \"aws-ebs-csi-driver-node-ndd7n\" (UID: \"1bc3d212-b67c-4368-90a3-8e080a4f1452\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ndd7n" Oct 31 00:15:08.820027 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.819961 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwf5l\" (UniqueName: \"kubernetes.io/projected/652defc0-301f-4679-8e49-6729381259dd-kube-api-access-jwf5l\") pod \"openshift-state-metrics-7bd7489957-xd6zl\" (UID: \"652defc0-301f-4679-8e49-6729381259dd\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xd6zl" Oct 31 00:15:08.820027 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.819977 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/26aba8fc-e2da-430f-b0d8-6ee45468441d-config-out\") pod \"alertmanager-main-0\" (UID: \"26aba8fc-e2da-430f-b0d8-6ee45468441d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:15:08.820027 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.819992 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0eb4888f-40a8-4aba-85d8-3c845382bd02-run\") pod \"tuned-vsfrc\" (UID: \"0eb4888f-40a8-4aba-85d8-3c845382bd02\") " pod="openshift-cluster-node-tuning-operator/tuned-vsfrc" Oct 31 00:15:08.820027 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.820006 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1b88d3b9-589f-4f4e-b8a6-872f4fb5536b-sys\") pod \"node-exporter-dbcsc\" (UID: \"1b88d3b9-589f-4f4e-b8a6-872f4fb5536b\") " pod="openshift-monitoring/node-exporter-dbcsc" Oct 31 00:15:08.820175 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.820027 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1b88d3b9-589f-4f4e-b8a6-872f4fb5536b-node-exporter-accelerators-collector-config\") pod \"node-exporter-dbcsc\" (UID: \"1b88d3b9-589f-4f4e-b8a6-872f4fb5536b\") " pod="openshift-monitoring/node-exporter-dbcsc" Oct 31 00:15:08.820175 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.820042 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26aba8fc-e2da-430f-b0d8-6ee45468441d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"26aba8fc-e2da-430f-b0d8-6ee45468441d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:15:08.820175 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.820058 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0f8c3c03-a1fc-4376-98c6-feeeefa08068-node-log\") pod \"ovnkube-node-h9hsf\" (UID: \"0f8c3c03-a1fc-4376-98c6-feeeefa08068\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:15:08.820175 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.820089 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1b88d3b9-589f-4f4e-b8a6-872f4fb5536b-metrics-client-ca\") pod \"node-exporter-dbcsc\" (UID: \"1b88d3b9-589f-4f4e-b8a6-872f4fb5536b\") " pod="openshift-monitoring/node-exporter-dbcsc" Oct 31 00:15:08.820175 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.820101 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/26aba8fc-e2da-430f-b0d8-6ee45468441d-config-out\") pod \"alertmanager-main-0\" (UID: \"26aba8fc-e2da-430f-b0d8-6ee45468441d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:15:08.820175 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.820105 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1b88d3b9-589f-4f4e-b8a6-872f4fb5536b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dbcsc\" (UID: \"1b88d3b9-589f-4f4e-b8a6-872f4fb5536b\") " pod="openshift-monitoring/node-exporter-dbcsc" Oct 31 00:15:08.820550 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.820138 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5c58f866-ef80-42d5-aab0-0cdefbc7a158-bound-sa-token\") pod \"image-registry-6644876fb8-9bszd\" (UID: \"5c58f866-ef80-42d5-aab0-0cdefbc7a158\") " pod="openshift-image-registry/image-registry-6644876fb8-9bszd" Oct 31 00:15:08.820684 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.820556 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d22409bc-1bcf-46b1-9789-6f7298566a7f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nq56p\" (UID: \"d22409bc-1bcf-46b1-9789-6f7298566a7f\") " pod="openshift-insights/insights-runtime-extractor-nq56p" Oct 31 00:15:08.820684 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.820574 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/26aba8fc-e2da-430f-b0d8-6ee45468441d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"26aba8fc-e2da-430f-b0d8-6ee45468441d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:15:08.820684 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.820592 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0f8c3c03-a1fc-4376-98c6-feeeefa08068-ovn-node-metrics-cert\") pod \"ovnkube-node-h9hsf\" (UID: \"0f8c3c03-a1fc-4376-98c6-feeeefa08068\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:15:08.820684 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.820608 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1bc3d212-b67c-4368-90a3-8e080a4f1452-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ndd7n\" (UID: \"1bc3d212-b67c-4368-90a3-8e080a4f1452\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ndd7n" Oct 31 00:15:08.820684 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.820624 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0eb4888f-40a8-4aba-85d8-3c845382bd02-etc-modprobe-d\") pod \"tuned-vsfrc\" (UID: \"0eb4888f-40a8-4aba-85d8-3c845382bd02\") " pod="openshift-cluster-node-tuning-operator/tuned-vsfrc" Oct 31 00:15:08.820684 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.820643 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9801ff00-9e9a-4141-8212-ca32bf0251b3-multus-socket-dir-parent\") pod \"multus-b6vpd\" (UID: \"9801ff00-9e9a-4141-8212-ca32bf0251b3\") " pod="openshift-multus/multus-b6vpd" Oct 31 00:15:08.820865 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.820697 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5c58f866-ef80-42d5-aab0-0cdefbc7a158-registry-certificates\") pod \"image-registry-6644876fb8-9bszd\" (UID: \"5c58f866-ef80-42d5-aab0-0cdefbc7a158\") " pod="openshift-image-registry/image-registry-6644876fb8-9bszd" Oct 31 00:15:08.820865 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.820721 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0eb4888f-40a8-4aba-85d8-3c845382bd02-etc-systemd\") pod \"tuned-vsfrc\" (UID: \"0eb4888f-40a8-4aba-85d8-3c845382bd02\") " pod="openshift-cluster-node-tuning-operator/tuned-vsfrc" Oct 31 00:15:08.820865 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.820751 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f8c3c03-a1fc-4376-98c6-feeeefa08068-var-lib-openvswitch\") pod \"ovnkube-node-h9hsf\" (UID: \"0f8c3c03-a1fc-4376-98c6-feeeefa08068\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:15:08.820865 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.820769 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0f8c3c03-a1fc-4376-98c6-feeeefa08068-run-ovn\") pod \"ovnkube-node-h9hsf\" (UID: \"0f8c3c03-a1fc-4376-98c6-feeeefa08068\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:15:08.820865 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.820786 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0f8c3c03-a1fc-4376-98c6-feeeefa08068-ovnkube-config\") pod \"ovnkube-node-h9hsf\" (UID: \"0f8c3c03-a1fc-4376-98c6-feeeefa08068\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:15:08.820865 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.820803 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/97d11160-2114-4fbf-95c2-fa7b4c9d2260-cert\") pod \"ingress-canary-x5s75\" (UID: \"97d11160-2114-4fbf-95c2-fa7b4c9d2260\") " pod="openshift-ingress-canary/ingress-canary-x5s75" Oct 31 00:15:08.820865 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.820845 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xc9bh\" (UniqueName: \"kubernetes.io/projected/a077dea4-b5a3-4335-bb6a-7618e794eb8f-kube-api-access-xc9bh\") pod \"network-metrics-daemon-rjfn2\" (UID: \"a077dea4-b5a3-4335-bb6a-7618e794eb8f\") " pod="openshift-multus/network-metrics-daemon-rjfn2" Oct 31 00:15:08.821052 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.820933 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2ed1e615-cbca-482c-89cc-0fe7d8e7e597-konnectivity-ca\") pod \"konnectivity-agent-lvp2h\" (UID: \"2ed1e615-cbca-482c-89cc-0fe7d8e7e597\") " pod="kube-system/konnectivity-agent-lvp2h" Oct 31 00:15:08.821052 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.820974 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9801ff00-9e9a-4141-8212-ca32bf0251b3-hostroot\") pod \"multus-b6vpd\" (UID: \"9801ff00-9e9a-4141-8212-ca32bf0251b3\") " pod="openshift-multus/multus-b6vpd" Oct 31 00:15:08.821052 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.821003 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9801ff00-9e9a-4141-8212-ca32bf0251b3-multus-conf-dir\") pod \"multus-b6vpd\" (UID: \"9801ff00-9e9a-4141-8212-ca32bf0251b3\") " pod="openshift-multus/multus-b6vpd" Oct 31 00:15:08.821052 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.821032 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9801ff00-9e9a-4141-8212-ca32bf0251b3-host-run-multus-certs\") pod \"multus-b6vpd\" (UID: \"9801ff00-9e9a-4141-8212-ca32bf0251b3\") " pod="openshift-multus/multus-b6vpd" Oct 31 00:15:08.821183 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.821064 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/652defc0-301f-4679-8e49-6729381259dd-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-xd6zl\" (UID: \"652defc0-301f-4679-8e49-6729381259dd\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xd6zl" Oct 31 00:15:08.821183 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.821096 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/26aba8fc-e2da-430f-b0d8-6ee45468441d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"26aba8fc-e2da-430f-b0d8-6ee45468441d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:15:08.821183 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.821157 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1b88d3b9-589f-4f4e-b8a6-872f4fb5536b-root\") pod \"node-exporter-dbcsc\" (UID: \"1b88d3b9-589f-4f4e-b8a6-872f4fb5536b\") " pod="openshift-monitoring/node-exporter-dbcsc" Oct 31 00:15:08.821183 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.821182 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d02ae6b5-4860-45b3-8d16-4a49531b77d4-original-pull-secret\") pod \"global-pull-secret-syncer-fwgtx\" (UID: \"d02ae6b5-4860-45b3-8d16-4a49531b77d4\") " pod="kube-system/global-pull-secret-syncer-fwgtx" Oct 31 00:15:08.821353 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.821198 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/26aba8fc-e2da-430f-b0d8-6ee45468441d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"26aba8fc-e2da-430f-b0d8-6ee45468441d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:15:08.821353 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.821258 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6a21f264-1e80-4eab-8c6e-27ca9ca19f0d-cnibin\") pod \"multus-additional-cni-plugins-jsctc\" (UID: \"6a21f264-1e80-4eab-8c6e-27ca9ca19f0d\") " pod="openshift-multus/multus-additional-cni-plugins-jsctc" Oct 31 00:15:08.821353 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.821296 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8j6tz\" (UniqueName: \"kubernetes.io/projected/6a21f264-1e80-4eab-8c6e-27ca9ca19f0d-kube-api-access-8j6tz\") pod \"multus-additional-cni-plugins-jsctc\" (UID: \"6a21f264-1e80-4eab-8c6e-27ca9ca19f0d\") " pod="openshift-multus/multus-additional-cni-plugins-jsctc" Oct 31 00:15:08.821353 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.821330 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0eb4888f-40a8-4aba-85d8-3c845382bd02-etc-sysctl-d\") pod \"tuned-vsfrc\" (UID: \"0eb4888f-40a8-4aba-85d8-3c845382bd02\") " pod="openshift-cluster-node-tuning-operator/tuned-vsfrc" Oct 31 00:15:08.821537 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.821356 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0f8c3c03-a1fc-4376-98c6-feeeefa08068-env-overrides\") pod \"ovnkube-node-h9hsf\" (UID: \"0f8c3c03-a1fc-4376-98c6-feeeefa08068\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:15:08.821537 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.821410 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1b88d3b9-589f-4f4e-b8a6-872f4fb5536b-node-exporter-wtmp\") pod \"node-exporter-dbcsc\" (UID: \"1b88d3b9-589f-4f4e-b8a6-872f4fb5536b\") " pod="openshift-monitoring/node-exporter-dbcsc" Oct 31 00:15:08.821537 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.821443 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0eb4888f-40a8-4aba-85d8-3c845382bd02-host\") pod \"tuned-vsfrc\" (UID: \"0eb4888f-40a8-4aba-85d8-3c845382bd02\") " pod="openshift-cluster-node-tuning-operator/tuned-vsfrc" Oct 31 00:15:08.821537 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.821477 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/978a9050-dddb-4869-85d8-84a001554c5b-host-slash\") pod \"iptables-alerter-4nwn7\" (UID: \"978a9050-dddb-4869-85d8-84a001554c5b\") " pod="openshift-network-operator/iptables-alerter-4nwn7" Oct 31 00:15:08.821537 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.821524 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/ea9a5160-4696-4b75-8a10-0fbc54b2ad04-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-rsf6d\" (UID: \"ea9a5160-4696-4b75-8a10-0fbc54b2ad04\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rsf6d" Oct 31 00:15:08.821690 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.821579 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/ea9a5160-4696-4b75-8a10-0fbc54b2ad04-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-rsf6d\" (UID: \"ea9a5160-4696-4b75-8a10-0fbc54b2ad04\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rsf6d" Oct 31 00:15:08.821690 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.821591 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d22409bc-1bcf-46b1-9789-6f7298566a7f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nq56p\" (UID: \"d22409bc-1bcf-46b1-9789-6f7298566a7f\") " pod="openshift-insights/insights-runtime-extractor-nq56p" Oct 31 00:15:08.821690 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.821621 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/26aba8fc-e2da-430f-b0d8-6ee45468441d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"26aba8fc-e2da-430f-b0d8-6ee45468441d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:15:08.821690 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.821645 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0eb4888f-40a8-4aba-85d8-3c845382bd02-tmp\") pod \"tuned-vsfrc\" (UID: \"0eb4888f-40a8-4aba-85d8-3c845382bd02\") " pod="openshift-cluster-node-tuning-operator/tuned-vsfrc" Oct 31 00:15:08.821690 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.821662 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0f8c3c03-a1fc-4376-98c6-feeeefa08068-systemd-units\") pod \"ovnkube-node-h9hsf\" (UID: \"0f8c3c03-a1fc-4376-98c6-feeeefa08068\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:15:08.821690 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.821678 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eadb6436-5e5b-4116-8ed7-26aa935fa749-host\") pod \"node-ca-xlzv2\" (UID: \"eadb6436-5e5b-4116-8ed7-26aa935fa749\") " pod="openshift-image-registry/node-ca-xlzv2" Oct 31 00:15:08.822012 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.821699 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9801ff00-9e9a-4141-8212-ca32bf0251b3-host-var-lib-kubelet\") pod \"multus-b6vpd\" (UID: \"9801ff00-9e9a-4141-8212-ca32bf0251b3\") " pod="openshift-multus/multus-b6vpd" Oct 31 00:15:08.822012 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.821780 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/652defc0-301f-4679-8e49-6729381259dd-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-xd6zl\" (UID: \"652defc0-301f-4679-8e49-6729381259dd\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xd6zl" Oct 31 00:15:08.822012 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.821806 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0eb4888f-40a8-4aba-85d8-3c845382bd02-tmp\") pod \"tuned-vsfrc\" (UID: \"0eb4888f-40a8-4aba-85d8-3c845382bd02\") " pod="openshift-cluster-node-tuning-operator/tuned-vsfrc" Oct 31 00:15:08.822012 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.821814 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d02ae6b5-4860-45b3-8d16-4a49531b77d4-dbus\") pod \"global-pull-secret-syncer-fwgtx\" (UID: \"d02ae6b5-4860-45b3-8d16-4a49531b77d4\") " pod="kube-system/global-pull-secret-syncer-fwgtx" Oct 31 00:15:08.822012 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.821894 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/26aba8fc-e2da-430f-b0d8-6ee45468441d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"26aba8fc-e2da-430f-b0d8-6ee45468441d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:15:08.822012 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.821929 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0f8c3c03-a1fc-4376-98c6-feeeefa08068-host-cni-netd\") pod \"ovnkube-node-h9hsf\" (UID: \"0f8c3c03-a1fc-4376-98c6-feeeefa08068\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:15:08.822012 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.821961 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pc97\" (UniqueName: \"kubernetes.io/projected/97d11160-2114-4fbf-95c2-fa7b4c9d2260-kube-api-access-6pc97\") pod \"ingress-canary-x5s75\" (UID: \"97d11160-2114-4fbf-95c2-fa7b4c9d2260\") " pod="openshift-ingress-canary/ingress-canary-x5s75" Oct 31 00:15:08.822255 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.822050 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2ed1e615-cbca-482c-89cc-0fe7d8e7e597-agent-certs\") pod \"konnectivity-agent-lvp2h\" (UID: \"2ed1e615-cbca-482c-89cc-0fe7d8e7e597\") " pod="kube-system/konnectivity-agent-lvp2h" Oct 31 00:15:08.822255 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.822067 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9801ff00-9e9a-4141-8212-ca32bf0251b3-system-cni-dir\") pod \"multus-b6vpd\" (UID: \"9801ff00-9e9a-4141-8212-ca32bf0251b3\") " pod="openshift-multus/multus-b6vpd" Oct 31 00:15:08.822255 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.822085 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/26aba8fc-e2da-430f-b0d8-6ee45468441d-web-config\") pod \"alertmanager-main-0\" (UID: \"26aba8fc-e2da-430f-b0d8-6ee45468441d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:15:08.822255 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.822110 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0f8c3c03-a1fc-4376-98c6-feeeefa08068-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-h9hsf\" (UID: \"0f8c3c03-a1fc-4376-98c6-feeeefa08068\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:15:08.822255 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.822135 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0eb4888f-40a8-4aba-85d8-3c845382bd02-etc-sysctl-conf\") pod \"tuned-vsfrc\" (UID: \"0eb4888f-40a8-4aba-85d8-3c845382bd02\") " pod="openshift-cluster-node-tuning-operator/tuned-vsfrc" Oct 31 00:15:08.822255 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.822233 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f8c3c03-a1fc-4376-98c6-feeeefa08068-run-openvswitch\") pod \"ovnkube-node-h9hsf\" (UID: \"0f8c3c03-a1fc-4376-98c6-feeeefa08068\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:15:08.822514 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.822277 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1b88d3b9-589f-4f4e-b8a6-872f4fb5536b-node-exporter-textfile\") pod \"node-exporter-dbcsc\" (UID: \"1b88d3b9-589f-4f4e-b8a6-872f4fb5536b\") " pod="openshift-monitoring/node-exporter-dbcsc" Oct 31 00:15:08.822514 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.822310 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a077dea4-b5a3-4335-bb6a-7618e794eb8f-metrics-certs\") pod \"network-metrics-daemon-rjfn2\" (UID: \"a077dea4-b5a3-4335-bb6a-7618e794eb8f\") " pod="openshift-multus/network-metrics-daemon-rjfn2" Oct 31 00:15:08.822514 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.822334 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9801ff00-9e9a-4141-8212-ca32bf0251b3-cni-binary-copy\") pod \"multus-b6vpd\" (UID: \"9801ff00-9e9a-4141-8212-ca32bf0251b3\") " pod="openshift-multus/multus-b6vpd" Oct 31 00:15:08.822514 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.822350 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9801ff00-9e9a-4141-8212-ca32bf0251b3-host-run-k8s-cni-cncf-io\") pod \"multus-b6vpd\" (UID: \"9801ff00-9e9a-4141-8212-ca32bf0251b3\") " pod="openshift-multus/multus-b6vpd" Oct 31 00:15:08.822514 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.822357 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1b88d3b9-589f-4f4e-b8a6-872f4fb5536b-node-exporter-textfile\") pod \"node-exporter-dbcsc\" (UID: \"1b88d3b9-589f-4f4e-b8a6-872f4fb5536b\") " pod="openshift-monitoring/node-exporter-dbcsc" Oct 31 00:15:08.822514 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.822395 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c61956a7-ba51-4063-bc25-aaa418168bc2-tmp\") pod \"downloads-6dd789c8b7-nbn2h\" (UID: \"c61956a7-ba51-4063-bc25-aaa418168bc2\") " pod="openshift-console/downloads-6dd789c8b7-nbn2h" Oct 31 00:15:08.822514 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.822493 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/26aba8fc-e2da-430f-b0d8-6ee45468441d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"26aba8fc-e2da-430f-b0d8-6ee45468441d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:15:08.822514 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.822500 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c61956a7-ba51-4063-bc25-aaa418168bc2-tmp\") pod \"downloads-6dd789c8b7-nbn2h\" (UID: \"c61956a7-ba51-4063-bc25-aaa418168bc2\") " pod="openshift-console/downloads-6dd789c8b7-nbn2h" Oct 31 00:15:08.822514 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.822509 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6a21f264-1e80-4eab-8c6e-27ca9ca19f0d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jsctc\" (UID: \"6a21f264-1e80-4eab-8c6e-27ca9ca19f0d\") " pod="openshift-multus/multus-additional-cni-plugins-jsctc" Oct 31 00:15:08.822827 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.822524 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0eb4888f-40a8-4aba-85d8-3c845382bd02-etc-sysconfig\") pod \"tuned-vsfrc\" (UID: \"0eb4888f-40a8-4aba-85d8-3c845382bd02\") " pod="openshift-cluster-node-tuning-operator/tuned-vsfrc" Oct 31 00:15:08.822827 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.822561 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a7d0a920-91fe-4c4a-b9bc-67550c324e7f-hosts-file\") pod \"node-resolver-gkm2c\" (UID: \"a7d0a920-91fe-4c4a-b9bc-67550c324e7f\") " pod="openshift-dns/node-resolver-gkm2c" Oct 31 00:15:08.822827 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.822593 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0eb4888f-40a8-4aba-85d8-3c845382bd02-lib-modules\") pod \"tuned-vsfrc\" (UID: \"0eb4888f-40a8-4aba-85d8-3c845382bd02\") " pod="openshift-cluster-node-tuning-operator/tuned-vsfrc" Oct 31 00:15:08.822827 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.822595 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/26aba8fc-e2da-430f-b0d8-6ee45468441d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"26aba8fc-e2da-430f-b0d8-6ee45468441d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:15:08.822827 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.822629 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0f8c3c03-a1fc-4376-98c6-feeeefa08068-log-socket\") pod \"ovnkube-node-h9hsf\" (UID: \"0f8c3c03-a1fc-4376-98c6-feeeefa08068\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:15:08.822827 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.822675 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqw94\" (UniqueName: \"kubernetes.io/projected/1b88d3b9-589f-4f4e-b8a6-872f4fb5536b-kube-api-access-dqw94\") pod \"node-exporter-dbcsc\" (UID: \"1b88d3b9-589f-4f4e-b8a6-872f4fb5536b\") " pod="openshift-monitoring/node-exporter-dbcsc" Oct 31 00:15:08.822827 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.822710 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d02ae6b5-4860-45b3-8d16-4a49531b77d4-kubelet-config\") pod \"global-pull-secret-syncer-fwgtx\" (UID: \"d02ae6b5-4860-45b3-8d16-4a49531b77d4\") " pod="kube-system/global-pull-secret-syncer-fwgtx" Oct 31 00:15:08.822827 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.822743 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5c58f866-ef80-42d5-aab0-0cdefbc7a158-installation-pull-secrets\") pod \"image-registry-6644876fb8-9bszd\" (UID: \"5c58f866-ef80-42d5-aab0-0cdefbc7a158\") " pod="openshift-image-registry/image-registry-6644876fb8-9bszd" Oct 31 00:15:08.822827 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.822773 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9801ff00-9e9a-4141-8212-ca32bf0251b3-host-var-lib-cni-multus\") pod \"multus-b6vpd\" (UID: \"9801ff00-9e9a-4141-8212-ca32bf0251b3\") " pod="openshift-multus/multus-b6vpd" Oct 31 00:15:08.822827 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.822801 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srgbh\" (UniqueName: \"kubernetes.io/projected/d22409bc-1bcf-46b1-9789-6f7298566a7f-kube-api-access-srgbh\") pod \"insights-runtime-extractor-nq56p\" (UID: \"d22409bc-1bcf-46b1-9789-6f7298566a7f\") " pod="openshift-insights/insights-runtime-extractor-nq56p" Oct 31 00:15:08.822827 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.822830 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5hld\" (UniqueName: \"kubernetes.io/projected/5c58f866-ef80-42d5-aab0-0cdefbc7a158-kube-api-access-n5hld\") pod \"image-registry-6644876fb8-9bszd\" (UID: \"5c58f866-ef80-42d5-aab0-0cdefbc7a158\") " pod="openshift-image-registry/image-registry-6644876fb8-9bszd" Oct 31 00:15:08.823123 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.822956 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f8c3c03-a1fc-4376-98c6-feeeefa08068-etc-openvswitch\") pod \"ovnkube-node-h9hsf\" (UID: \"0f8c3c03-a1fc-4376-98c6-feeeefa08068\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:15:08.823123 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.822974 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9801ff00-9e9a-4141-8212-ca32bf0251b3-etc-kubernetes\") pod \"multus-b6vpd\" (UID: \"9801ff00-9e9a-4141-8212-ca32bf0251b3\") " pod="openshift-multus/multus-b6vpd" Oct 31 00:15:08.823123 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.823019 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83d351d1-c6e7-43a6-a0fb-eaecefa8c5b2-tmp-dir\") pod \"dns-default-lrh4r\" (UID: \"83d351d1-c6e7-43a6-a0fb-eaecefa8c5b2\") " pod="openshift-dns/dns-default-lrh4r" Oct 31 00:15:08.823123 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.823036 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9801ff00-9e9a-4141-8212-ca32bf0251b3-host-var-lib-cni-bin\") pod \"multus-b6vpd\" (UID: \"9801ff00-9e9a-4141-8212-ca32bf0251b3\") " pod="openshift-multus/multus-b6vpd" Oct 31 00:15:08.823123 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.823050 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1bc3d212-b67c-4368-90a3-8e080a4f1452-sys-fs\") pod \"aws-ebs-csi-driver-node-ndd7n\" (UID: \"1bc3d212-b67c-4368-90a3-8e080a4f1452\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ndd7n" Oct 31 00:15:08.823123 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.823072 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5c58f866-ef80-42d5-aab0-0cdefbc7a158-trusted-ca\") pod \"image-registry-6644876fb8-9bszd\" (UID: \"5c58f866-ef80-42d5-aab0-0cdefbc7a158\") " pod="openshift-image-registry/image-registry-6644876fb8-9bszd" Oct 31 00:15:08.823123 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.823105 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83d351d1-c6e7-43a6-a0fb-eaecefa8c5b2-tmp-dir\") pod \"dns-default-lrh4r\" (UID: \"83d351d1-c6e7-43a6-a0fb-eaecefa8c5b2\") " pod="openshift-dns/dns-default-lrh4r" Oct 31 00:15:08.823123 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.823107 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/26aba8fc-e2da-430f-b0d8-6ee45468441d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"26aba8fc-e2da-430f-b0d8-6ee45468441d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:15:08.823408 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.823137 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4v246\" (UniqueName: \"kubernetes.io/projected/26aba8fc-e2da-430f-b0d8-6ee45468441d-kube-api-access-4v246\") pod \"alertmanager-main-0\" (UID: \"26aba8fc-e2da-430f-b0d8-6ee45468441d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:15:08.823408 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.823169 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0f8c3c03-a1fc-4376-98c6-feeeefa08068-host-slash\") pod \"ovnkube-node-h9hsf\" (UID: \"0f8c3c03-a1fc-4376-98c6-feeeefa08068\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:15:08.823408 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.823210 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5c58f866-ef80-42d5-aab0-0cdefbc7a158-image-registry-private-configuration\") pod \"image-registry-6644876fb8-9bszd\" (UID: \"5c58f866-ef80-42d5-aab0-0cdefbc7a158\") " pod="openshift-image-registry/image-registry-6644876fb8-9bszd" Oct 31 00:15:08.823408 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.823314 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5c58f866-ef80-42d5-aab0-0cdefbc7a158-ca-trust-extracted\") pod \"image-registry-6644876fb8-9bszd\" (UID: \"5c58f866-ef80-42d5-aab0-0cdefbc7a158\") " pod="openshift-image-registry/image-registry-6644876fb8-9bszd" Oct 31 00:15:08.823408 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.823384 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5c58f866-ef80-42d5-aab0-0cdefbc7a158-ca-trust-extracted\") pod \"image-registry-6644876fb8-9bszd\" (UID: \"5c58f866-ef80-42d5-aab0-0cdefbc7a158\") " pod="openshift-image-registry/image-registry-6644876fb8-9bszd" Oct 31 00:15:08.823548 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.823420 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6a21f264-1e80-4eab-8c6e-27ca9ca19f0d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jsctc\" (UID: \"6a21f264-1e80-4eab-8c6e-27ca9ca19f0d\") " pod="openshift-multus/multus-additional-cni-plugins-jsctc" Oct 31 00:15:08.823548 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.823441 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0f8c3c03-a1fc-4376-98c6-feeeefa08068-host-kubelet\") pod \"ovnkube-node-h9hsf\" (UID: \"0f8c3c03-a1fc-4376-98c6-feeeefa08068\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:15:08.823548 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.823459 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0eb4888f-40a8-4aba-85d8-3c845382bd02-etc-kubernetes\") pod \"tuned-vsfrc\" (UID: \"0eb4888f-40a8-4aba-85d8-3c845382bd02\") " pod="openshift-cluster-node-tuning-operator/tuned-vsfrc" Oct 31 00:15:08.823548 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.823483 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lsrt\" (UniqueName: \"kubernetes.io/projected/83d351d1-c6e7-43a6-a0fb-eaecefa8c5b2-kube-api-access-5lsrt\") pod \"dns-default-lrh4r\" (UID: \"83d351d1-c6e7-43a6-a0fb-eaecefa8c5b2\") " pod="openshift-dns/dns-default-lrh4r" Oct 31 00:15:08.823548 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.823509 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ea9a5160-4696-4b75-8a10-0fbc54b2ad04-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-rsf6d\" (UID: \"ea9a5160-4696-4b75-8a10-0fbc54b2ad04\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rsf6d" Oct 31 00:15:08.823548 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.823529 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbpb7\" (UniqueName: \"kubernetes.io/projected/978a9050-dddb-4869-85d8-84a001554c5b-kube-api-access-gbpb7\") pod \"iptables-alerter-4nwn7\" (UID: \"978a9050-dddb-4869-85d8-84a001554c5b\") " pod="openshift-network-operator/iptables-alerter-4nwn7" Oct 31 00:15:08.823548 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.823546 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/652defc0-301f-4679-8e49-6729381259dd-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-xd6zl\" (UID: \"652defc0-301f-4679-8e49-6729381259dd\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xd6zl" Oct 31 00:15:08.823840 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.823572 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7d0a920-91fe-4c4a-b9bc-67550c324e7f-tmp-dir\") pod \"node-resolver-gkm2c\" (UID: \"a7d0a920-91fe-4c4a-b9bc-67550c324e7f\") " pod="openshift-dns/node-resolver-gkm2c" Oct 31 00:15:08.823840 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.823597 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0eb4888f-40a8-4aba-85d8-3c845382bd02-var-lib-kubelet\") pod \"tuned-vsfrc\" (UID: \"0eb4888f-40a8-4aba-85d8-3c845382bd02\") " pod="openshift-cluster-node-tuning-operator/tuned-vsfrc" Oct 31 00:15:08.823840 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.823625 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/83d351d1-c6e7-43a6-a0fb-eaecefa8c5b2-metrics-tls\") pod \"dns-default-lrh4r\" (UID: \"83d351d1-c6e7-43a6-a0fb-eaecefa8c5b2\") " pod="openshift-dns/dns-default-lrh4r" Oct 31 00:15:08.823840 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.823675 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7d0a920-91fe-4c4a-b9bc-67550c324e7f-tmp-dir\") pod \"node-resolver-gkm2c\" (UID: \"a7d0a920-91fe-4c4a-b9bc-67550c324e7f\") " pod="openshift-dns/node-resolver-gkm2c" Oct 31 00:15:08.845197 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.845161 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 31 00:15:08.845197 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.845199 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 31 00:15:08.850532 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.849877 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-7hvcc\"" Oct 31 00:15:08.850532 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.849995 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-plmbm\"" Oct 31 00:15:08.850532 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.850034 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 31 00:15:08.850532 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.850097 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-h9nxs\"" Oct 31 00:15:08.850532 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.850434 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-bvv2r\"" Oct 31 00:15:08.851590 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.851021 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 31 00:15:08.851590 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.851263 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2ed1e615-cbca-482c-89cc-0fe7d8e7e597-konnectivity-ca\") pod \"konnectivity-agent-lvp2h\" (UID: \"2ed1e615-cbca-482c-89cc-0fe7d8e7e597\") " pod="kube-system/konnectivity-agent-lvp2h" Oct 31 00:15:08.851590 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.851384 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-5jgbw\"" Oct 31 00:15:08.851590 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.851427 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 31 00:15:08.851590 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.851463 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-lg2hb\"" Oct 31 00:15:08.851850 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.851571 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 31 00:15:08.851850 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.851641 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-bcxbn\"" Oct 31 00:15:08.851850 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.851673 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 31 00:15:08.851850 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.851690 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 31 00:15:08.851850 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.851704 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 31 00:15:08.851850 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.851674 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 31 00:15:08.851850 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.851735 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 31 00:15:08.851850 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.851748 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-qdpqj\"" Oct 31 00:15:08.851850 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.851811 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 31 00:15:08.852538 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.852521 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 31 00:15:08.852637 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.852545 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 31 00:15:08.852637 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.852590 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-v2pb4\"" Oct 31 00:15:08.852637 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.852623 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 31 00:15:08.852865 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.852852 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 31 00:15:08.852907 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.852852 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9801ff00-9e9a-4141-8212-ca32bf0251b3-cni-binary-copy\") pod \"multus-b6vpd\" (UID: \"9801ff00-9e9a-4141-8212-ca32bf0251b3\") " pod="openshift-multus/multus-b6vpd" Oct 31 00:15:08.852956 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.852912 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 31 00:15:08.852956 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.852943 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 31 00:15:08.852956 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.852951 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 31 00:15:08.853056 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.852956 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 31 00:15:08.853056 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.852975 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 31 00:15:08.853056 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.852982 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 31 00:15:08.854269 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.854255 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6a21f264-1e80-4eab-8c6e-27ca9ca19f0d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jsctc\" (UID: \"6a21f264-1e80-4eab-8c6e-27ca9ca19f0d\") " pod="openshift-multus/multus-additional-cni-plugins-jsctc" Oct 31 00:15:08.859170 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.859147 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6a21f264-1e80-4eab-8c6e-27ca9ca19f0d-cni-binary-copy\") pod \"multus-additional-cni-plugins-jsctc\" (UID: \"6a21f264-1e80-4eab-8c6e-27ca9ca19f0d\") " pod="openshift-multus/multus-additional-cni-plugins-jsctc" Oct 31 00:15:08.859510 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.859487 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9801ff00-9e9a-4141-8212-ca32bf0251b3-multus-daemon-config\") pod \"multus-b6vpd\" (UID: \"9801ff00-9e9a-4141-8212-ca32bf0251b3\") " pod="openshift-multus/multus-b6vpd" Oct 31 00:15:08.859563 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.859551 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/978a9050-dddb-4869-85d8-84a001554c5b-iptables-alerter-script\") pod \"iptables-alerter-4nwn7\" (UID: \"978a9050-dddb-4869-85d8-84a001554c5b\") " pod="openshift-network-operator/iptables-alerter-4nwn7" Oct 31 00:15:08.859609 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.859553 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6a21f264-1e80-4eab-8c6e-27ca9ca19f0d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jsctc\" (UID: \"6a21f264-1e80-4eab-8c6e-27ca9ca19f0d\") " pod="openshift-multus/multus-additional-cni-plugins-jsctc" Oct 31 00:15:08.861109 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.861091 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/eadb6436-5e5b-4116-8ed7-26aa935fa749-serviceca\") pod \"node-ca-xlzv2\" (UID: \"eadb6436-5e5b-4116-8ed7-26aa935fa749\") " pod="openshift-image-registry/node-ca-xlzv2" Oct 31 00:15:08.861297 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.861280 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5c58f866-ef80-42d5-aab0-0cdefbc7a158-registry-certificates\") pod \"image-registry-6644876fb8-9bszd\" (UID: \"5c58f866-ef80-42d5-aab0-0cdefbc7a158\") " pod="openshift-image-registry/image-registry-6644876fb8-9bszd" Oct 31 00:15:08.861557 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.861541 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d02ae6b5-4860-45b3-8d16-4a49531b77d4-original-pull-secret\") pod \"global-pull-secret-syncer-fwgtx\" (UID: \"d02ae6b5-4860-45b3-8d16-4a49531b77d4\") " pod="kube-system/global-pull-secret-syncer-fwgtx" Oct 31 00:15:08.862957 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.862941 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2ed1e615-cbca-482c-89cc-0fe7d8e7e597-agent-certs\") pod \"konnectivity-agent-lvp2h\" (UID: \"2ed1e615-cbca-482c-89cc-0fe7d8e7e597\") " pod="kube-system/konnectivity-agent-lvp2h" Oct 31 00:15:08.911001 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.910966 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrkcn\" (UniqueName: \"kubernetes.io/projected/0eb4888f-40a8-4aba-85d8-3c845382bd02-kube-api-access-nrkcn\") pod \"tuned-vsfrc\" (UID: \"0eb4888f-40a8-4aba-85d8-3c845382bd02\") " pod="openshift-cluster-node-tuning-operator/tuned-vsfrc" Oct 31 00:15:08.912835 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.912812 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5c58f866-ef80-42d5-aab0-0cdefbc7a158-bound-sa-token\") pod \"image-registry-6644876fb8-9bszd\" (UID: \"5c58f866-ef80-42d5-aab0-0cdefbc7a158\") " pod="openshift-image-registry/image-registry-6644876fb8-9bszd" Oct 31 00:15:08.913386 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.913347 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djjgz\" (UniqueName: \"kubernetes.io/projected/a7d0a920-91fe-4c4a-b9bc-67550c324e7f-kube-api-access-djjgz\") pod \"node-resolver-gkm2c\" (UID: \"a7d0a920-91fe-4c4a-b9bc-67550c324e7f\") " pod="openshift-dns/node-resolver-gkm2c" Oct 31 00:15:08.915581 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.915564 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzs4x\" (UniqueName: \"kubernetes.io/projected/9801ff00-9e9a-4141-8212-ca32bf0251b3-kube-api-access-xzs4x\") pod \"multus-b6vpd\" (UID: \"9801ff00-9e9a-4141-8212-ca32bf0251b3\") " pod="openshift-multus/multus-b6vpd" Oct 31 00:15:08.918382 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.918349 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wm7wn\" (UniqueName: \"kubernetes.io/projected/1bc3d212-b67c-4368-90a3-8e080a4f1452-kube-api-access-wm7wn\") pod \"aws-ebs-csi-driver-node-ndd7n\" (UID: \"1bc3d212-b67c-4368-90a3-8e080a4f1452\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ndd7n" Oct 31 00:15:08.918761 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.918745 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5hld\" (UniqueName: \"kubernetes.io/projected/5c58f866-ef80-42d5-aab0-0cdefbc7a158-kube-api-access-n5hld\") pod \"image-registry-6644876fb8-9bszd\" (UID: \"5c58f866-ef80-42d5-aab0-0cdefbc7a158\") " pod="openshift-image-registry/image-registry-6644876fb8-9bszd" Oct 31 00:15:08.922570 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.922552 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8j6tz\" (UniqueName: \"kubernetes.io/projected/6a21f264-1e80-4eab-8c6e-27ca9ca19f0d-kube-api-access-8j6tz\") pod \"multus-additional-cni-plugins-jsctc\" (UID: \"6a21f264-1e80-4eab-8c6e-27ca9ca19f0d\") " pod="openshift-multus/multus-additional-cni-plugins-jsctc" Oct 31 00:15:08.923955 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.923937 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9801ff00-9e9a-4141-8212-ca32bf0251b3-host-var-lib-cni-bin\") pod \"multus-b6vpd\" (UID: \"9801ff00-9e9a-4141-8212-ca32bf0251b3\") " pod="openshift-multus/multus-b6vpd" Oct 31 00:15:08.923994 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.923966 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1bc3d212-b67c-4368-90a3-8e080a4f1452-sys-fs\") pod \"aws-ebs-csi-driver-node-ndd7n\" (UID: \"1bc3d212-b67c-4368-90a3-8e080a4f1452\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ndd7n" Oct 31 00:15:08.923994 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.923982 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0f8c3c03-a1fc-4376-98c6-feeeefa08068-host-slash\") pod \"ovnkube-node-h9hsf\" (UID: \"0f8c3c03-a1fc-4376-98c6-feeeefa08068\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:15:08.924062 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.923984 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9801ff00-9e9a-4141-8212-ca32bf0251b3-host-var-lib-cni-bin\") pod \"multus-b6vpd\" (UID: \"9801ff00-9e9a-4141-8212-ca32bf0251b3\") " pod="openshift-multus/multus-b6vpd" Oct 31 00:15:08.924062 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.924027 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0f8c3c03-a1fc-4376-98c6-feeeefa08068-host-kubelet\") pod \"ovnkube-node-h9hsf\" (UID: \"0f8c3c03-a1fc-4376-98c6-feeeefa08068\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:15:08.924062 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.924037 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0f8c3c03-a1fc-4376-98c6-feeeefa08068-host-slash\") pod \"ovnkube-node-h9hsf\" (UID: \"0f8c3c03-a1fc-4376-98c6-feeeefa08068\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:15:08.924062 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.923999 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0f8c3c03-a1fc-4376-98c6-feeeefa08068-host-kubelet\") pod \"ovnkube-node-h9hsf\" (UID: \"0f8c3c03-a1fc-4376-98c6-feeeefa08068\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:15:08.924170 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.924094 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1bc3d212-b67c-4368-90a3-8e080a4f1452-sys-fs\") pod \"aws-ebs-csi-driver-node-ndd7n\" (UID: \"1bc3d212-b67c-4368-90a3-8e080a4f1452\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ndd7n" Oct 31 00:15:08.924170 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.924103 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0eb4888f-40a8-4aba-85d8-3c845382bd02-etc-kubernetes\") pod \"tuned-vsfrc\" (UID: \"0eb4888f-40a8-4aba-85d8-3c845382bd02\") " pod="openshift-cluster-node-tuning-operator/tuned-vsfrc" Oct 31 00:15:08.924170 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.924133 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0eb4888f-40a8-4aba-85d8-3c845382bd02-var-lib-kubelet\") pod \"tuned-vsfrc\" (UID: \"0eb4888f-40a8-4aba-85d8-3c845382bd02\") " pod="openshift-cluster-node-tuning-operator/tuned-vsfrc" Oct 31 00:15:08.924170 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.924140 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0eb4888f-40a8-4aba-85d8-3c845382bd02-etc-kubernetes\") pod \"tuned-vsfrc\" (UID: \"0eb4888f-40a8-4aba-85d8-3c845382bd02\") " pod="openshift-cluster-node-tuning-operator/tuned-vsfrc" Oct 31 00:15:08.924170 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.924149 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1bc3d212-b67c-4368-90a3-8e080a4f1452-device-dir\") pod \"aws-ebs-csi-driver-node-ndd7n\" (UID: \"1bc3d212-b67c-4368-90a3-8e080a4f1452\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ndd7n" Oct 31 00:15:08.924170 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.924167 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1bc3d212-b67c-4368-90a3-8e080a4f1452-registration-dir\") pod \"aws-ebs-csi-driver-node-ndd7n\" (UID: \"1bc3d212-b67c-4368-90a3-8e080a4f1452\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ndd7n" Oct 31 00:15:08.924421 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.924176 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0eb4888f-40a8-4aba-85d8-3c845382bd02-var-lib-kubelet\") pod \"tuned-vsfrc\" (UID: \"0eb4888f-40a8-4aba-85d8-3c845382bd02\") " pod="openshift-cluster-node-tuning-operator/tuned-vsfrc" Oct 31 00:15:08.924421 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.924182 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9801ff00-9e9a-4141-8212-ca32bf0251b3-cnibin\") pod \"multus-b6vpd\" (UID: \"9801ff00-9e9a-4141-8212-ca32bf0251b3\") " pod="openshift-multus/multus-b6vpd" Oct 31 00:15:08.924421 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.924199 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0eb4888f-40a8-4aba-85d8-3c845382bd02-sys\") pod \"tuned-vsfrc\" (UID: \"0eb4888f-40a8-4aba-85d8-3c845382bd02\") " pod="openshift-cluster-node-tuning-operator/tuned-vsfrc" Oct 31 00:15:08.924421 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.924211 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1bc3d212-b67c-4368-90a3-8e080a4f1452-device-dir\") pod \"aws-ebs-csi-driver-node-ndd7n\" (UID: \"1bc3d212-b67c-4368-90a3-8e080a4f1452\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ndd7n" Oct 31 00:15:08.924421 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.924215 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6a21f264-1e80-4eab-8c6e-27ca9ca19f0d-system-cni-dir\") pod \"multus-additional-cni-plugins-jsctc\" (UID: \"6a21f264-1e80-4eab-8c6e-27ca9ca19f0d\") " pod="openshift-multus/multus-additional-cni-plugins-jsctc" Oct 31 00:15:08.924421 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.924243 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6a21f264-1e80-4eab-8c6e-27ca9ca19f0d-system-cni-dir\") pod \"multus-additional-cni-plugins-jsctc\" (UID: \"6a21f264-1e80-4eab-8c6e-27ca9ca19f0d\") " pod="openshift-multus/multus-additional-cni-plugins-jsctc" Oct 31 00:15:08.924421 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.924249 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9801ff00-9e9a-4141-8212-ca32bf0251b3-cnibin\") pod \"multus-b6vpd\" (UID: \"9801ff00-9e9a-4141-8212-ca32bf0251b3\") " pod="openshift-multus/multus-b6vpd" Oct 31 00:15:08.924421 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.924252 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0f8c3c03-a1fc-4376-98c6-feeeefa08068-host-cni-bin\") pod \"ovnkube-node-h9hsf\" (UID: \"0f8c3c03-a1fc-4376-98c6-feeeefa08068\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:15:08.924421 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.924246 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1bc3d212-b67c-4368-90a3-8e080a4f1452-registration-dir\") pod \"aws-ebs-csi-driver-node-ndd7n\" (UID: \"1bc3d212-b67c-4368-90a3-8e080a4f1452\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ndd7n" Oct 31 00:15:08.924421 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.924289 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0f8c3c03-a1fc-4376-98c6-feeeefa08068-run-systemd\") pod \"ovnkube-node-h9hsf\" (UID: \"0f8c3c03-a1fc-4376-98c6-feeeefa08068\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:15:08.924421 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.924302 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0eb4888f-40a8-4aba-85d8-3c845382bd02-sys\") pod \"tuned-vsfrc\" (UID: \"0eb4888f-40a8-4aba-85d8-3c845382bd02\") " pod="openshift-cluster-node-tuning-operator/tuned-vsfrc" Oct 31 00:15:08.924421 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.924311 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0f8c3c03-a1fc-4376-98c6-feeeefa08068-host-run-ovn-kubernetes\") pod \"ovnkube-node-h9hsf\" (UID: \"0f8c3c03-a1fc-4376-98c6-feeeefa08068\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:15:08.924421 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.924328 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0f8c3c03-a1fc-4376-98c6-feeeefa08068-host-run-ovn-kubernetes\") pod \"ovnkube-node-h9hsf\" (UID: \"0f8c3c03-a1fc-4376-98c6-feeeefa08068\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:15:08.924421 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.924343 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0f8c3c03-a1fc-4376-98c6-feeeefa08068-host-cni-bin\") pod \"ovnkube-node-h9hsf\" (UID: \"0f8c3c03-a1fc-4376-98c6-feeeefa08068\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:15:08.924421 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.924344 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0f8c3c03-a1fc-4376-98c6-feeeefa08068-run-systemd\") pod \"ovnkube-node-h9hsf\" (UID: \"0f8c3c03-a1fc-4376-98c6-feeeefa08068\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:15:08.924421 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.924357 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9801ff00-9e9a-4141-8212-ca32bf0251b3-host-run-netns\") pod \"multus-b6vpd\" (UID: \"9801ff00-9e9a-4141-8212-ca32bf0251b3\") " pod="openshift-multus/multus-b6vpd" Oct 31 00:15:08.924421 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.924413 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9801ff00-9e9a-4141-8212-ca32bf0251b3-host-run-netns\") pod \"multus-b6vpd\" (UID: \"9801ff00-9e9a-4141-8212-ca32bf0251b3\") " pod="openshift-multus/multus-b6vpd" Oct 31 00:15:08.924421 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.924424 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1bc3d212-b67c-4368-90a3-8e080a4f1452-socket-dir\") pod \"aws-ebs-csi-driver-node-ndd7n\" (UID: \"1bc3d212-b67c-4368-90a3-8e080a4f1452\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ndd7n" Oct 31 00:15:08.925222 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.924452 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1bc3d212-b67c-4368-90a3-8e080a4f1452-etc-selinux\") pod \"aws-ebs-csi-driver-node-ndd7n\" (UID: \"1bc3d212-b67c-4368-90a3-8e080a4f1452\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ndd7n" Oct 31 00:15:08.925222 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.924469 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9801ff00-9e9a-4141-8212-ca32bf0251b3-multus-cni-dir\") pod \"multus-b6vpd\" (UID: \"9801ff00-9e9a-4141-8212-ca32bf0251b3\") " pod="openshift-multus/multus-b6vpd" Oct 31 00:15:08.925222 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.924482 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1bc3d212-b67c-4368-90a3-8e080a4f1452-socket-dir\") pod \"aws-ebs-csi-driver-node-ndd7n\" (UID: \"1bc3d212-b67c-4368-90a3-8e080a4f1452\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ndd7n" Oct 31 00:15:08.925222 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.924485 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d22409bc-1bcf-46b1-9789-6f7298566a7f-crio-socket\") pod \"insights-runtime-extractor-nq56p\" (UID: \"d22409bc-1bcf-46b1-9789-6f7298566a7f\") " pod="openshift-insights/insights-runtime-extractor-nq56p" Oct 31 00:15:08.925222 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.924505 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0f8c3c03-a1fc-4376-98c6-feeeefa08068-host-run-netns\") pod \"ovnkube-node-h9hsf\" (UID: \"0f8c3c03-a1fc-4376-98c6-feeeefa08068\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:15:08.925222 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.924525 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9801ff00-9e9a-4141-8212-ca32bf0251b3-multus-cni-dir\") pod \"multus-b6vpd\" (UID: \"9801ff00-9e9a-4141-8212-ca32bf0251b3\") " pod="openshift-multus/multus-b6vpd" Oct 31 00:15:08.925222 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.924531 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0eb4888f-40a8-4aba-85d8-3c845382bd02-run\") pod \"tuned-vsfrc\" (UID: \"0eb4888f-40a8-4aba-85d8-3c845382bd02\") " pod="openshift-cluster-node-tuning-operator/tuned-vsfrc" Oct 31 00:15:08.925222 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.924543 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0f8c3c03-a1fc-4376-98c6-feeeefa08068-host-run-netns\") pod \"ovnkube-node-h9hsf\" (UID: \"0f8c3c03-a1fc-4376-98c6-feeeefa08068\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:15:08.925222 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.924562 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d22409bc-1bcf-46b1-9789-6f7298566a7f-crio-socket\") pod \"insights-runtime-extractor-nq56p\" (UID: \"d22409bc-1bcf-46b1-9789-6f7298566a7f\") " pod="openshift-insights/insights-runtime-extractor-nq56p" Oct 31 00:15:08.925222 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.924561 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1bc3d212-b67c-4368-90a3-8e080a4f1452-etc-selinux\") pod \"aws-ebs-csi-driver-node-ndd7n\" (UID: \"1bc3d212-b67c-4368-90a3-8e080a4f1452\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ndd7n" Oct 31 00:15:08.925222 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.924578 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1b88d3b9-589f-4f4e-b8a6-872f4fb5536b-sys\") pod \"node-exporter-dbcsc\" (UID: \"1b88d3b9-589f-4f4e-b8a6-872f4fb5536b\") " pod="openshift-monitoring/node-exporter-dbcsc" Oct 31 00:15:08.925222 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.924588 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0eb4888f-40a8-4aba-85d8-3c845382bd02-run\") pod \"tuned-vsfrc\" (UID: \"0eb4888f-40a8-4aba-85d8-3c845382bd02\") " pod="openshift-cluster-node-tuning-operator/tuned-vsfrc" Oct 31 00:15:08.925222 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.924607 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0f8c3c03-a1fc-4376-98c6-feeeefa08068-node-log\") pod \"ovnkube-node-h9hsf\" (UID: \"0f8c3c03-a1fc-4376-98c6-feeeefa08068\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:15:08.925222 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.924642 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1bc3d212-b67c-4368-90a3-8e080a4f1452-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ndd7n\" (UID: \"1bc3d212-b67c-4368-90a3-8e080a4f1452\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ndd7n" Oct 31 00:15:08.925222 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.924668 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0eb4888f-40a8-4aba-85d8-3c845382bd02-etc-modprobe-d\") pod \"tuned-vsfrc\" (UID: \"0eb4888f-40a8-4aba-85d8-3c845382bd02\") " pod="openshift-cluster-node-tuning-operator/tuned-vsfrc" Oct 31 00:15:08.925222 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.924671 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1b88d3b9-589f-4f4e-b8a6-872f4fb5536b-sys\") pod \"node-exporter-dbcsc\" (UID: \"1b88d3b9-589f-4f4e-b8a6-872f4fb5536b\") " pod="openshift-monitoring/node-exporter-dbcsc" Oct 31 00:15:08.925222 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.924684 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0f8c3c03-a1fc-4376-98c6-feeeefa08068-node-log\") pod \"ovnkube-node-h9hsf\" (UID: \"0f8c3c03-a1fc-4376-98c6-feeeefa08068\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:15:08.925222 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.924699 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9801ff00-9e9a-4141-8212-ca32bf0251b3-multus-socket-dir-parent\") pod \"multus-b6vpd\" (UID: \"9801ff00-9e9a-4141-8212-ca32bf0251b3\") " pod="openshift-multus/multus-b6vpd" Oct 31 00:15:08.925222 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.924729 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1bc3d212-b67c-4368-90a3-8e080a4f1452-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ndd7n\" (UID: \"1bc3d212-b67c-4368-90a3-8e080a4f1452\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ndd7n" Oct 31 00:15:08.925222 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.924734 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0eb4888f-40a8-4aba-85d8-3c845382bd02-etc-systemd\") pod \"tuned-vsfrc\" (UID: \"0eb4888f-40a8-4aba-85d8-3c845382bd02\") " pod="openshift-cluster-node-tuning-operator/tuned-vsfrc" Oct 31 00:15:08.925222 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.924742 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9801ff00-9e9a-4141-8212-ca32bf0251b3-multus-socket-dir-parent\") pod \"multus-b6vpd\" (UID: \"9801ff00-9e9a-4141-8212-ca32bf0251b3\") " pod="openshift-multus/multus-b6vpd" Oct 31 00:15:08.925222 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.924763 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f8c3c03-a1fc-4376-98c6-feeeefa08068-var-lib-openvswitch\") pod \"ovnkube-node-h9hsf\" (UID: \"0f8c3c03-a1fc-4376-98c6-feeeefa08068\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:15:08.925222 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.924769 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0eb4888f-40a8-4aba-85d8-3c845382bd02-etc-systemd\") pod \"tuned-vsfrc\" (UID: \"0eb4888f-40a8-4aba-85d8-3c845382bd02\") " pod="openshift-cluster-node-tuning-operator/tuned-vsfrc" Oct 31 00:15:08.925222 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.924779 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0f8c3c03-a1fc-4376-98c6-feeeefa08068-run-ovn\") pod \"ovnkube-node-h9hsf\" (UID: \"0f8c3c03-a1fc-4376-98c6-feeeefa08068\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:15:08.925222 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.924779 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0eb4888f-40a8-4aba-85d8-3c845382bd02-etc-modprobe-d\") pod \"tuned-vsfrc\" (UID: \"0eb4888f-40a8-4aba-85d8-3c845382bd02\") " pod="openshift-cluster-node-tuning-operator/tuned-vsfrc" Oct 31 00:15:08.925222 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.924795 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f8c3c03-a1fc-4376-98c6-feeeefa08068-var-lib-openvswitch\") pod \"ovnkube-node-h9hsf\" (UID: \"0f8c3c03-a1fc-4376-98c6-feeeefa08068\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:15:08.925222 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.924802 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9801ff00-9e9a-4141-8212-ca32bf0251b3-hostroot\") pod \"multus-b6vpd\" (UID: \"9801ff00-9e9a-4141-8212-ca32bf0251b3\") " pod="openshift-multus/multus-b6vpd" Oct 31 00:15:08.925222 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.924821 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9801ff00-9e9a-4141-8212-ca32bf0251b3-hostroot\") pod \"multus-b6vpd\" (UID: \"9801ff00-9e9a-4141-8212-ca32bf0251b3\") " pod="openshift-multus/multus-b6vpd" Oct 31 00:15:08.925222 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.924821 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0f8c3c03-a1fc-4376-98c6-feeeefa08068-run-ovn\") pod \"ovnkube-node-h9hsf\" (UID: \"0f8c3c03-a1fc-4376-98c6-feeeefa08068\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:15:08.925222 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.924824 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9801ff00-9e9a-4141-8212-ca32bf0251b3-multus-conf-dir\") pod \"multus-b6vpd\" (UID: \"9801ff00-9e9a-4141-8212-ca32bf0251b3\") " pod="openshift-multus/multus-b6vpd" Oct 31 00:15:08.925222 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.924843 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9801ff00-9e9a-4141-8212-ca32bf0251b3-multus-conf-dir\") pod \"multus-b6vpd\" (UID: \"9801ff00-9e9a-4141-8212-ca32bf0251b3\") " pod="openshift-multus/multus-b6vpd" Oct 31 00:15:08.925222 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.924854 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9801ff00-9e9a-4141-8212-ca32bf0251b3-host-run-multus-certs\") pod \"multus-b6vpd\" (UID: \"9801ff00-9e9a-4141-8212-ca32bf0251b3\") " pod="openshift-multus/multus-b6vpd" Oct 31 00:15:08.925222 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.924895 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1b88d3b9-589f-4f4e-b8a6-872f4fb5536b-root\") pod \"node-exporter-dbcsc\" (UID: \"1b88d3b9-589f-4f4e-b8a6-872f4fb5536b\") " pod="openshift-monitoring/node-exporter-dbcsc" Oct 31 00:15:08.925222 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.924907 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9801ff00-9e9a-4141-8212-ca32bf0251b3-host-run-multus-certs\") pod \"multus-b6vpd\" (UID: \"9801ff00-9e9a-4141-8212-ca32bf0251b3\") " pod="openshift-multus/multus-b6vpd" Oct 31 00:15:08.925222 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.924933 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6a21f264-1e80-4eab-8c6e-27ca9ca19f0d-cnibin\") pod \"multus-additional-cni-plugins-jsctc\" (UID: \"6a21f264-1e80-4eab-8c6e-27ca9ca19f0d\") " pod="openshift-multus/multus-additional-cni-plugins-jsctc" Oct 31 00:15:08.925222 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.924963 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0eb4888f-40a8-4aba-85d8-3c845382bd02-etc-sysctl-d\") pod \"tuned-vsfrc\" (UID: \"0eb4888f-40a8-4aba-85d8-3c845382bd02\") " pod="openshift-cluster-node-tuning-operator/tuned-vsfrc" Oct 31 00:15:08.925222 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.924992 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1b88d3b9-589f-4f4e-b8a6-872f4fb5536b-node-exporter-wtmp\") pod \"node-exporter-dbcsc\" (UID: \"1b88d3b9-589f-4f4e-b8a6-872f4fb5536b\") " pod="openshift-monitoring/node-exporter-dbcsc" Oct 31 00:15:08.925222 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.925005 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6a21f264-1e80-4eab-8c6e-27ca9ca19f0d-cnibin\") pod \"multus-additional-cni-plugins-jsctc\" (UID: \"6a21f264-1e80-4eab-8c6e-27ca9ca19f0d\") " pod="openshift-multus/multus-additional-cni-plugins-jsctc" Oct 31 00:15:08.925222 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.924994 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1b88d3b9-589f-4f4e-b8a6-872f4fb5536b-root\") pod \"node-exporter-dbcsc\" (UID: \"1b88d3b9-589f-4f4e-b8a6-872f4fb5536b\") " pod="openshift-monitoring/node-exporter-dbcsc" Oct 31 00:15:08.925222 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.925017 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0eb4888f-40a8-4aba-85d8-3c845382bd02-host\") pod \"tuned-vsfrc\" (UID: \"0eb4888f-40a8-4aba-85d8-3c845382bd02\") " pod="openshift-cluster-node-tuning-operator/tuned-vsfrc" Oct 31 00:15:08.925222 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.925050 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1b88d3b9-589f-4f4e-b8a6-872f4fb5536b-node-exporter-wtmp\") pod \"node-exporter-dbcsc\" (UID: \"1b88d3b9-589f-4f4e-b8a6-872f4fb5536b\") " pod="openshift-monitoring/node-exporter-dbcsc" Oct 31 00:15:08.925222 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.925073 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/978a9050-dddb-4869-85d8-84a001554c5b-host-slash\") pod \"iptables-alerter-4nwn7\" (UID: \"978a9050-dddb-4869-85d8-84a001554c5b\") " pod="openshift-network-operator/iptables-alerter-4nwn7" Oct 31 00:15:08.925222 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.925081 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0eb4888f-40a8-4aba-85d8-3c845382bd02-etc-sysctl-d\") pod \"tuned-vsfrc\" (UID: \"0eb4888f-40a8-4aba-85d8-3c845382bd02\") " pod="openshift-cluster-node-tuning-operator/tuned-vsfrc" Oct 31 00:15:08.925222 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.925105 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0f8c3c03-a1fc-4376-98c6-feeeefa08068-systemd-units\") pod \"ovnkube-node-h9hsf\" (UID: \"0f8c3c03-a1fc-4376-98c6-feeeefa08068\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:15:08.925222 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.925084 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0eb4888f-40a8-4aba-85d8-3c845382bd02-host\") pod \"tuned-vsfrc\" (UID: \"0eb4888f-40a8-4aba-85d8-3c845382bd02\") " pod="openshift-cluster-node-tuning-operator/tuned-vsfrc" Oct 31 00:15:08.925222 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.925131 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eadb6436-5e5b-4116-8ed7-26aa935fa749-host\") pod \"node-ca-xlzv2\" (UID: \"eadb6436-5e5b-4116-8ed7-26aa935fa749\") " pod="openshift-image-registry/node-ca-xlzv2" Oct 31 00:15:08.925222 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.925156 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9801ff00-9e9a-4141-8212-ca32bf0251b3-host-var-lib-kubelet\") pod \"multus-b6vpd\" (UID: \"9801ff00-9e9a-4141-8212-ca32bf0251b3\") " pod="openshift-multus/multus-b6vpd" Oct 31 00:15:08.925222 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.925167 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0f8c3c03-a1fc-4376-98c6-feeeefa08068-systemd-units\") pod \"ovnkube-node-h9hsf\" (UID: \"0f8c3c03-a1fc-4376-98c6-feeeefa08068\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:15:08.925222 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.925186 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d02ae6b5-4860-45b3-8d16-4a49531b77d4-dbus\") pod \"global-pull-secret-syncer-fwgtx\" (UID: \"d02ae6b5-4860-45b3-8d16-4a49531b77d4\") " pod="kube-system/global-pull-secret-syncer-fwgtx" Oct 31 00:15:08.925222 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.925187 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eadb6436-5e5b-4116-8ed7-26aa935fa749-host\") pod \"node-ca-xlzv2\" (UID: \"eadb6436-5e5b-4116-8ed7-26aa935fa749\") " pod="openshift-image-registry/node-ca-xlzv2" Oct 31 00:15:08.925222 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.925188 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/978a9050-dddb-4869-85d8-84a001554c5b-host-slash\") pod \"iptables-alerter-4nwn7\" (UID: \"978a9050-dddb-4869-85d8-84a001554c5b\") " pod="openshift-network-operator/iptables-alerter-4nwn7" Oct 31 00:15:08.925222 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.925203 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0f8c3c03-a1fc-4376-98c6-feeeefa08068-host-cni-netd\") pod \"ovnkube-node-h9hsf\" (UID: \"0f8c3c03-a1fc-4376-98c6-feeeefa08068\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:15:08.925222 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.925210 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9801ff00-9e9a-4141-8212-ca32bf0251b3-host-var-lib-kubelet\") pod \"multus-b6vpd\" (UID: \"9801ff00-9e9a-4141-8212-ca32bf0251b3\") " pod="openshift-multus/multus-b6vpd" Oct 31 00:15:08.925222 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.925223 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9801ff00-9e9a-4141-8212-ca32bf0251b3-system-cni-dir\") pod \"multus-b6vpd\" (UID: \"9801ff00-9e9a-4141-8212-ca32bf0251b3\") " pod="openshift-multus/multus-b6vpd" Oct 31 00:15:08.925222 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.925237 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d02ae6b5-4860-45b3-8d16-4a49531b77d4-dbus\") pod \"global-pull-secret-syncer-fwgtx\" (UID: \"d02ae6b5-4860-45b3-8d16-4a49531b77d4\") " pod="kube-system/global-pull-secret-syncer-fwgtx" Oct 31 00:15:08.925222 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.925239 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0f8c3c03-a1fc-4376-98c6-feeeefa08068-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-h9hsf\" (UID: \"0f8c3c03-a1fc-4376-98c6-feeeefa08068\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:15:08.925222 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.925245 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0f8c3c03-a1fc-4376-98c6-feeeefa08068-host-cni-netd\") pod \"ovnkube-node-h9hsf\" (UID: \"0f8c3c03-a1fc-4376-98c6-feeeefa08068\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:15:08.925222 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.925257 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0eb4888f-40a8-4aba-85d8-3c845382bd02-etc-sysctl-conf\") pod \"tuned-vsfrc\" (UID: \"0eb4888f-40a8-4aba-85d8-3c845382bd02\") " pod="openshift-cluster-node-tuning-operator/tuned-vsfrc" Oct 31 00:15:08.926941 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.925274 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f8c3c03-a1fc-4376-98c6-feeeefa08068-run-openvswitch\") pod \"ovnkube-node-h9hsf\" (UID: \"0f8c3c03-a1fc-4376-98c6-feeeefa08068\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:15:08.926941 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.925280 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9801ff00-9e9a-4141-8212-ca32bf0251b3-system-cni-dir\") pod \"multus-b6vpd\" (UID: \"9801ff00-9e9a-4141-8212-ca32bf0251b3\") " pod="openshift-multus/multus-b6vpd" Oct 31 00:15:08.926941 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.925278 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0f8c3c03-a1fc-4376-98c6-feeeefa08068-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-h9hsf\" (UID: \"0f8c3c03-a1fc-4376-98c6-feeeefa08068\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:15:08.926941 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.925291 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9801ff00-9e9a-4141-8212-ca32bf0251b3-host-run-k8s-cni-cncf-io\") pod \"multus-b6vpd\" (UID: \"9801ff00-9e9a-4141-8212-ca32bf0251b3\") " pod="openshift-multus/multus-b6vpd" Oct 31 00:15:08.926941 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.925319 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9801ff00-9e9a-4141-8212-ca32bf0251b3-host-run-k8s-cni-cncf-io\") pod \"multus-b6vpd\" (UID: \"9801ff00-9e9a-4141-8212-ca32bf0251b3\") " pod="openshift-multus/multus-b6vpd" Oct 31 00:15:08.926941 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.925330 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6a21f264-1e80-4eab-8c6e-27ca9ca19f0d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jsctc\" (UID: \"6a21f264-1e80-4eab-8c6e-27ca9ca19f0d\") " pod="openshift-multus/multus-additional-cni-plugins-jsctc" Oct 31 00:15:08.926941 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.925345 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f8c3c03-a1fc-4376-98c6-feeeefa08068-run-openvswitch\") pod \"ovnkube-node-h9hsf\" (UID: \"0f8c3c03-a1fc-4376-98c6-feeeefa08068\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:15:08.926941 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.925348 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0eb4888f-40a8-4aba-85d8-3c845382bd02-etc-sysconfig\") pod \"tuned-vsfrc\" (UID: \"0eb4888f-40a8-4aba-85d8-3c845382bd02\") " pod="openshift-cluster-node-tuning-operator/tuned-vsfrc" Oct 31 00:15:08.926941 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.925396 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0eb4888f-40a8-4aba-85d8-3c845382bd02-etc-sysctl-conf\") pod \"tuned-vsfrc\" (UID: \"0eb4888f-40a8-4aba-85d8-3c845382bd02\") " pod="openshift-cluster-node-tuning-operator/tuned-vsfrc" Oct 31 00:15:08.926941 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.925409 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0eb4888f-40a8-4aba-85d8-3c845382bd02-etc-sysconfig\") pod \"tuned-vsfrc\" (UID: \"0eb4888f-40a8-4aba-85d8-3c845382bd02\") " pod="openshift-cluster-node-tuning-operator/tuned-vsfrc" Oct 31 00:15:08.926941 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.925414 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a7d0a920-91fe-4c4a-b9bc-67550c324e7f-hosts-file\") pod \"node-resolver-gkm2c\" (UID: \"a7d0a920-91fe-4c4a-b9bc-67550c324e7f\") " pod="openshift-dns/node-resolver-gkm2c" Oct 31 00:15:08.926941 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.925442 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0eb4888f-40a8-4aba-85d8-3c845382bd02-lib-modules\") pod \"tuned-vsfrc\" (UID: \"0eb4888f-40a8-4aba-85d8-3c845382bd02\") " pod="openshift-cluster-node-tuning-operator/tuned-vsfrc" Oct 31 00:15:08.926941 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.925451 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a7d0a920-91fe-4c4a-b9bc-67550c324e7f-hosts-file\") pod \"node-resolver-gkm2c\" (UID: \"a7d0a920-91fe-4c4a-b9bc-67550c324e7f\") " pod="openshift-dns/node-resolver-gkm2c" Oct 31 00:15:08.926941 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.925468 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0f8c3c03-a1fc-4376-98c6-feeeefa08068-log-socket\") pod \"ovnkube-node-h9hsf\" (UID: \"0f8c3c03-a1fc-4376-98c6-feeeefa08068\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:15:08.926941 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.925413 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6a21f264-1e80-4eab-8c6e-27ca9ca19f0d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jsctc\" (UID: \"6a21f264-1e80-4eab-8c6e-27ca9ca19f0d\") " pod="openshift-multus/multus-additional-cni-plugins-jsctc" Oct 31 00:15:08.926941 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.925504 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0f8c3c03-a1fc-4376-98c6-feeeefa08068-log-socket\") pod \"ovnkube-node-h9hsf\" (UID: \"0f8c3c03-a1fc-4376-98c6-feeeefa08068\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:15:08.926941 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.925525 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d02ae6b5-4860-45b3-8d16-4a49531b77d4-kubelet-config\") pod \"global-pull-secret-syncer-fwgtx\" (UID: \"d02ae6b5-4860-45b3-8d16-4a49531b77d4\") " pod="kube-system/global-pull-secret-syncer-fwgtx" Oct 31 00:15:08.926941 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.925530 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0eb4888f-40a8-4aba-85d8-3c845382bd02-lib-modules\") pod \"tuned-vsfrc\" (UID: \"0eb4888f-40a8-4aba-85d8-3c845382bd02\") " pod="openshift-cluster-node-tuning-operator/tuned-vsfrc" Oct 31 00:15:08.926941 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.925542 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9801ff00-9e9a-4141-8212-ca32bf0251b3-host-var-lib-cni-multus\") pod \"multus-b6vpd\" (UID: \"9801ff00-9e9a-4141-8212-ca32bf0251b3\") " pod="openshift-multus/multus-b6vpd" Oct 31 00:15:08.926941 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.925561 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f8c3c03-a1fc-4376-98c6-feeeefa08068-etc-openvswitch\") pod \"ovnkube-node-h9hsf\" (UID: \"0f8c3c03-a1fc-4376-98c6-feeeefa08068\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:15:08.926941 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.925566 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d02ae6b5-4860-45b3-8d16-4a49531b77d4-kubelet-config\") pod \"global-pull-secret-syncer-fwgtx\" (UID: \"d02ae6b5-4860-45b3-8d16-4a49531b77d4\") " pod="kube-system/global-pull-secret-syncer-fwgtx" Oct 31 00:15:08.926941 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.925575 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9801ff00-9e9a-4141-8212-ca32bf0251b3-etc-kubernetes\") pod \"multus-b6vpd\" (UID: \"9801ff00-9e9a-4141-8212-ca32bf0251b3\") " pod="openshift-multus/multus-b6vpd" Oct 31 00:15:08.926941 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.925592 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f8c3c03-a1fc-4376-98c6-feeeefa08068-etc-openvswitch\") pod \"ovnkube-node-h9hsf\" (UID: \"0f8c3c03-a1fc-4376-98c6-feeeefa08068\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:15:08.926941 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.925600 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9801ff00-9e9a-4141-8212-ca32bf0251b3-host-var-lib-cni-multus\") pod \"multus-b6vpd\" (UID: \"9801ff00-9e9a-4141-8212-ca32bf0251b3\") " pod="openshift-multus/multus-b6vpd" Oct 31 00:15:08.926941 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.925610 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9801ff00-9e9a-4141-8212-ca32bf0251b3-etc-kubernetes\") pod \"multus-b6vpd\" (UID: \"9801ff00-9e9a-4141-8212-ca32bf0251b3\") " pod="openshift-multus/multus-b6vpd" Oct 31 00:15:08.927588 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.927233 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lsrt\" (UniqueName: \"kubernetes.io/projected/83d351d1-c6e7-43a6-a0fb-eaecefa8c5b2-kube-api-access-5lsrt\") pod \"dns-default-lrh4r\" (UID: \"83d351d1-c6e7-43a6-a0fb-eaecefa8c5b2\") " pod="openshift-dns/dns-default-lrh4r" Oct 31 00:15:08.929398 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.929349 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xc9bh\" (UniqueName: \"kubernetes.io/projected/a077dea4-b5a3-4335-bb6a-7618e794eb8f-kube-api-access-xc9bh\") pod \"network-metrics-daemon-rjfn2\" (UID: \"a077dea4-b5a3-4335-bb6a-7618e794eb8f\") " pod="openshift-multus/network-metrics-daemon-rjfn2" Oct 31 00:15:08.930004 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.929982 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ms4cs\" (UniqueName: \"kubernetes.io/projected/0cbf0042-931c-4135-9330-f9e798b7a567-kube-api-access-ms4cs\") pod \"network-check-target-ml8n5\" (UID: \"0cbf0042-931c-4135-9330-f9e798b7a567\") " pod="openshift-network-diagnostics/network-check-target-ml8n5" Oct 31 00:15:08.930363 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.930346 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nz8r2\" (UniqueName: \"kubernetes.io/projected/eadb6436-5e5b-4116-8ed7-26aa935fa749-kube-api-access-nz8r2\") pod \"node-ca-xlzv2\" (UID: \"eadb6436-5e5b-4116-8ed7-26aa935fa749\") " pod="openshift-image-registry/node-ca-xlzv2" Oct 31 00:15:08.943613 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.943579 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 31 00:15:08.943613 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.943604 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 31 00:15:08.943613 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.943579 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 31 00:15:08.943892 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.943621 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 31 00:15:08.943892 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.943636 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 31 00:15:08.943892 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.943692 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 31 00:15:08.943892 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.943679 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zk5sj\"" Oct 31 00:15:08.943892 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.943655 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 31 00:15:08.943892 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.943813 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-mkn2t\"" Oct 31 00:15:08.943892 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.943821 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 31 00:15:08.943892 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.943828 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 31 00:15:08.943892 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.943838 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 31 00:15:08.943892 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.943851 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-pk6r6\"" Oct 31 00:15:08.943892 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.943841 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 31 00:15:08.943892 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.943863 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-hcz6l\"" Oct 31 00:15:08.943892 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.943877 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 31 00:15:08.943892 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.943863 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 31 00:15:08.943892 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.943853 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 31 00:15:08.944280 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.943907 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-xfr46\"" Oct 31 00:15:08.944280 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.943940 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 31 00:15:08.944280 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.943980 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 31 00:15:08.944280 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.944046 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 31 00:15:08.944280 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.944089 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-ks82c\"" Oct 31 00:15:08.944280 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.944143 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-ds5xr\"" Oct 31 00:15:08.944280 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.944242 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-74dxc\"" Oct 31 00:15:08.944486 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.944295 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 31 00:15:08.944486 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.944390 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 31 00:15:08.944486 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.944452 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 31 00:15:08.944567 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.944511 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-qszwm\"" Oct 31 00:15:08.944596 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.944569 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 31 00:15:08.944623 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.944603 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-xx7tk\"" Oct 31 00:15:08.944650 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.944635 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-gf7wd\"" Oct 31 00:15:08.944687 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.944674 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-ltd79\"" Oct 31 00:15:08.944733 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.944724 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 31 00:15:08.946533 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.946516 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 31 00:15:08.946569 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.946557 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 31 00:15:08.946608 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.946590 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 31 00:15:08.946686 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.946673 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 31 00:15:08.946724 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.946714 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 31 00:15:08.946878 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.946868 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 31 00:15:08.947233 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.947222 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 31 00:15:08.947272 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.947228 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 00:15:08.947272 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.947247 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 31 00:15:08.947329 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.947278 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 31 00:15:08.947425 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.947414 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 31 00:15:08.947727 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.947716 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 31 00:15:08.947798 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.947725 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 31 00:15:08.947798 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.947756 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 31 00:15:08.947996 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.947983 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 31 00:15:08.948064 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.948054 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 31 00:15:08.948526 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.948514 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 31 00:15:08.948726 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.948707 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwf5l\" (UniqueName: \"kubernetes.io/projected/652defc0-301f-4679-8e49-6729381259dd-kube-api-access-jwf5l\") pod \"openshift-state-metrics-7bd7489957-xd6zl\" (UID: \"652defc0-301f-4679-8e49-6729381259dd\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xd6zl" Oct 31 00:15:08.948929 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.948914 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/83d351d1-c6e7-43a6-a0fb-eaecefa8c5b2-config-volume\") pod \"dns-default-lrh4r\" (UID: \"83d351d1-c6e7-43a6-a0fb-eaecefa8c5b2\") " pod="openshift-dns/dns-default-lrh4r" Oct 31 00:15:08.948967 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.948917 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbpb7\" (UniqueName: \"kubernetes.io/projected/978a9050-dddb-4869-85d8-84a001554c5b-kube-api-access-gbpb7\") pod \"iptables-alerter-4nwn7\" (UID: \"978a9050-dddb-4869-85d8-84a001554c5b\") " pod="openshift-network-operator/iptables-alerter-4nwn7" Oct 31 00:15:08.949055 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.949041 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pc97\" (UniqueName: \"kubernetes.io/projected/97d11160-2114-4fbf-95c2-fa7b4c9d2260-kube-api-access-6pc97\") pod \"ingress-canary-x5s75\" (UID: \"97d11160-2114-4fbf-95c2-fa7b4c9d2260\") " pod="openshift-ingress-canary/ingress-canary-x5s75" Oct 31 00:15:08.949963 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.949947 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ea9a5160-4696-4b75-8a10-0fbc54b2ad04-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-rsf6d\" (UID: \"ea9a5160-4696-4b75-8a10-0fbc54b2ad04\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rsf6d" Oct 31 00:15:08.949963 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.949956 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/26aba8fc-e2da-430f-b0d8-6ee45468441d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"26aba8fc-e2da-430f-b0d8-6ee45468441d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:15:08.950026 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.949955 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/26aba8fc-e2da-430f-b0d8-6ee45468441d-config-volume\") pod \"alertmanager-main-0\" (UID: \"26aba8fc-e2da-430f-b0d8-6ee45468441d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:15:08.950077 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.950063 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5c58f866-ef80-42d5-aab0-0cdefbc7a158-registry-tls\") pod \"image-registry-6644876fb8-9bszd\" (UID: \"5c58f866-ef80-42d5-aab0-0cdefbc7a158\") " pod="openshift-image-registry/image-registry-6644876fb8-9bszd" Oct 31 00:15:08.950124 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.950111 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1b88d3b9-589f-4f4e-b8a6-872f4fb5536b-node-exporter-tls\") pod \"node-exporter-dbcsc\" (UID: \"1b88d3b9-589f-4f4e-b8a6-872f4fb5536b\") " pod="openshift-monitoring/node-exporter-dbcsc" Oct 31 00:15:08.951393 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.951200 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0f8c3c03-a1fc-4376-98c6-feeeefa08068-ovn-node-metrics-cert\") pod \"ovnkube-node-h9hsf\" (UID: \"0f8c3c03-a1fc-4376-98c6-feeeefa08068\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:15:08.951393 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.951226 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1b88d3b9-589f-4f4e-b8a6-872f4fb5536b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dbcsc\" (UID: \"1b88d3b9-589f-4f4e-b8a6-872f4fb5536b\") " pod="openshift-monitoring/node-exporter-dbcsc" Oct 31 00:15:08.951393 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.951245 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/26aba8fc-e2da-430f-b0d8-6ee45468441d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"26aba8fc-e2da-430f-b0d8-6ee45468441d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:15:08.951393 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.951250 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/26aba8fc-e2da-430f-b0d8-6ee45468441d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"26aba8fc-e2da-430f-b0d8-6ee45468441d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:15:08.951393 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.951308 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0f8c3c03-a1fc-4376-98c6-feeeefa08068-ovnkube-config\") pod \"ovnkube-node-h9hsf\" (UID: \"0f8c3c03-a1fc-4376-98c6-feeeefa08068\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:15:08.951393 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.951328 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d22409bc-1bcf-46b1-9789-6f7298566a7f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nq56p\" (UID: \"d22409bc-1bcf-46b1-9789-6f7298566a7f\") " pod="openshift-insights/insights-runtime-extractor-nq56p" Oct 31 00:15:08.951393 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.951349 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/97d11160-2114-4fbf-95c2-fa7b4c9d2260-cert\") pod \"ingress-canary-x5s75\" (UID: \"97d11160-2114-4fbf-95c2-fa7b4c9d2260\") " pod="openshift-ingress-canary/ingress-canary-x5s75" Oct 31 00:15:08.951393 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.951388 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1b88d3b9-589f-4f4e-b8a6-872f4fb5536b-node-exporter-accelerators-collector-config\") pod \"node-exporter-dbcsc\" (UID: \"1b88d3b9-589f-4f4e-b8a6-872f4fb5536b\") " pod="openshift-monitoring/node-exporter-dbcsc" Oct 31 00:15:08.951748 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.951395 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/652defc0-301f-4679-8e49-6729381259dd-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-xd6zl\" (UID: \"652defc0-301f-4679-8e49-6729381259dd\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xd6zl" Oct 31 00:15:08.951748 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.951426 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/26aba8fc-e2da-430f-b0d8-6ee45468441d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"26aba8fc-e2da-430f-b0d8-6ee45468441d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:15:08.951748 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.951581 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0f8c3c03-a1fc-4376-98c6-feeeefa08068-env-overrides\") pod \"ovnkube-node-h9hsf\" (UID: \"0f8c3c03-a1fc-4376-98c6-feeeefa08068\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:15:08.951842 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.951820 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7hn4\" (UniqueName: \"kubernetes.io/projected/ea9a5160-4696-4b75-8a10-0fbc54b2ad04-kube-api-access-v7hn4\") pod \"kube-state-metrics-5dc8d7d89b-rsf6d\" (UID: \"ea9a5160-4696-4b75-8a10-0fbc54b2ad04\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rsf6d" Oct 31 00:15:08.951984 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.951966 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d22409bc-1bcf-46b1-9789-6f7298566a7f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nq56p\" (UID: \"d22409bc-1bcf-46b1-9789-6f7298566a7f\") " pod="openshift-insights/insights-runtime-extractor-nq56p" Oct 31 00:15:08.951984 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.951975 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/652defc0-301f-4679-8e49-6729381259dd-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-xd6zl\" (UID: \"652defc0-301f-4679-8e49-6729381259dd\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xd6zl" Oct 31 00:15:08.952087 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.952075 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/26aba8fc-e2da-430f-b0d8-6ee45468441d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"26aba8fc-e2da-430f-b0d8-6ee45468441d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:15:08.953097 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.953079 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a077dea4-b5a3-4335-bb6a-7618e794eb8f-metrics-certs\") pod \"network-metrics-daemon-rjfn2\" (UID: \"a077dea4-b5a3-4335-bb6a-7618e794eb8f\") " pod="openshift-multus/network-metrics-daemon-rjfn2" Oct 31 00:15:08.953097 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.953084 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5c58f866-ef80-42d5-aab0-0cdefbc7a158-installation-pull-secrets\") pod \"image-registry-6644876fb8-9bszd\" (UID: \"5c58f866-ef80-42d5-aab0-0cdefbc7a158\") " pod="openshift-image-registry/image-registry-6644876fb8-9bszd" Oct 31 00:15:08.953097 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.953082 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/26aba8fc-e2da-430f-b0d8-6ee45468441d-web-config\") pod \"alertmanager-main-0\" (UID: \"26aba8fc-e2da-430f-b0d8-6ee45468441d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:15:08.953444 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.953429 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqw94\" (UniqueName: \"kubernetes.io/projected/1b88d3b9-589f-4f4e-b8a6-872f4fb5536b-kube-api-access-dqw94\") pod \"node-exporter-dbcsc\" (UID: \"1b88d3b9-589f-4f4e-b8a6-872f4fb5536b\") " pod="openshift-monitoring/node-exporter-dbcsc" Oct 31 00:15:08.953579 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.953564 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/26aba8fc-e2da-430f-b0d8-6ee45468441d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"26aba8fc-e2da-430f-b0d8-6ee45468441d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:15:08.953706 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.953691 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ea9a5160-4696-4b75-8a10-0fbc54b2ad04-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-rsf6d\" (UID: \"ea9a5160-4696-4b75-8a10-0fbc54b2ad04\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rsf6d" Oct 31 00:15:08.953832 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.953707 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5c58f866-ef80-42d5-aab0-0cdefbc7a158-image-registry-private-configuration\") pod \"image-registry-6644876fb8-9bszd\" (UID: \"5c58f866-ef80-42d5-aab0-0cdefbc7a158\") " pod="openshift-image-registry/image-registry-6644876fb8-9bszd" Oct 31 00:15:08.954069 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.954050 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cspld\" (UniqueName: \"kubernetes.io/projected/c61956a7-ba51-4063-bc25-aaa418168bc2-kube-api-access-cspld\") pod \"downloads-6dd789c8b7-nbn2h\" (UID: \"c61956a7-ba51-4063-bc25-aaa418168bc2\") " pod="openshift-console/downloads-6dd789c8b7-nbn2h" Oct 31 00:15:08.954239 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.954224 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/83d351d1-c6e7-43a6-a0fb-eaecefa8c5b2-metrics-tls\") pod \"dns-default-lrh4r\" (UID: \"83d351d1-c6e7-43a6-a0fb-eaecefa8c5b2\") " pod="openshift-dns/dns-default-lrh4r" Oct 31 00:15:08.954626 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.954609 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4v246\" (UniqueName: \"kubernetes.io/projected/26aba8fc-e2da-430f-b0d8-6ee45468441d-kube-api-access-4v246\") pod \"alertmanager-main-0\" (UID: \"26aba8fc-e2da-430f-b0d8-6ee45468441d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:15:08.955801 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.955776 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wrw4\" (UniqueName: \"kubernetes.io/projected/0f8c3c03-a1fc-4376-98c6-feeeefa08068-kube-api-access-8wrw4\") pod \"ovnkube-node-h9hsf\" (UID: \"0f8c3c03-a1fc-4376-98c6-feeeefa08068\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:15:08.955889 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.955846 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srgbh\" (UniqueName: \"kubernetes.io/projected/d22409bc-1bcf-46b1-9789-6f7298566a7f-kube-api-access-srgbh\") pod \"insights-runtime-extractor-nq56p\" (UID: \"d22409bc-1bcf-46b1-9789-6f7298566a7f\") " pod="openshift-insights/insights-runtime-extractor-nq56p" Oct 31 00:15:08.995450 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.995420 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 31 00:15:08.995625 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.995465 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 31 00:15:08.995625 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.995490 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 31 00:15:08.995625 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.995475 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 31 00:15:08.995625 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.995579 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 31 00:15:08.999108 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.999084 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ea9a5160-4696-4b75-8a10-0fbc54b2ad04-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-rsf6d\" (UID: \"ea9a5160-4696-4b75-8a10-0fbc54b2ad04\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rsf6d" Oct 31 00:15:08.999283 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:08.999265 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/ea9a5160-4696-4b75-8a10-0fbc54b2ad04-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-rsf6d\" (UID: \"ea9a5160-4696-4b75-8a10-0fbc54b2ad04\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rsf6d" Oct 31 00:15:09.000200 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:09.000184 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0f8c3c03-a1fc-4376-98c6-feeeefa08068-ovnkube-script-lib\") pod \"ovnkube-node-h9hsf\" (UID: \"0f8c3c03-a1fc-4376-98c6-feeeefa08068\") " pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:15:09.000416 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:09.000397 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1b88d3b9-589f-4f4e-b8a6-872f4fb5536b-metrics-client-ca\") pod \"node-exporter-dbcsc\" (UID: \"1b88d3b9-589f-4f4e-b8a6-872f4fb5536b\") " pod="openshift-monitoring/node-exporter-dbcsc" Oct 31 00:15:09.000519 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:09.000506 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26aba8fc-e2da-430f-b0d8-6ee45468441d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"26aba8fc-e2da-430f-b0d8-6ee45468441d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:15:09.002425 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:09.002410 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/26aba8fc-e2da-430f-b0d8-6ee45468441d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"26aba8fc-e2da-430f-b0d8-6ee45468441d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:15:09.003649 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:09.003632 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5c58f866-ef80-42d5-aab0-0cdefbc7a158-trusted-ca\") pod \"image-registry-6644876fb8-9bszd\" (UID: \"5c58f866-ef80-42d5-aab0-0cdefbc7a158\") " pod="openshift-image-registry/image-registry-6644876fb8-9bszd" Oct 31 00:15:09.003805 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:09.003792 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/652defc0-301f-4679-8e49-6729381259dd-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-xd6zl\" (UID: \"652defc0-301f-4679-8e49-6729381259dd\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xd6zl" Oct 31 00:15:09.097569 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:09.097549 5222 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 00:15:09.224010 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:09.223982 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-68dfc74784-dx6d6"] Oct 31 00:15:09.228303 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:09.228282 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-68dfc74784-dx6d6" Oct 31 00:15:09.284881 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:09.284837 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Oct 31 00:15:09.284881 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:09.284857 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 31 00:15:09.285175 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:09.285160 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Oct 31 00:15:09.285656 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:09.285635 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Oct 31 00:15:09.285769 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:09.285704 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-8bcst\"" Oct 31 00:15:09.285769 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:09.285730 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-22qsvkmif98fq\"" Oct 31 00:15:09.285922 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:09.285903 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-68dfc74784-dx6d6"] Oct 31 00:15:09.329053 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:09.329023 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2cc453ef-8580-4745-b57b-03f7516056a7-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-68dfc74784-dx6d6\" (UID: \"2cc453ef-8580-4745-b57b-03f7516056a7\") " pod="openshift-monitoring/metrics-server-68dfc74784-dx6d6" Oct 31 00:15:09.329257 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:09.329079 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/2cc453ef-8580-4745-b57b-03f7516056a7-client-ca-bundle\") pod \"metrics-server-68dfc74784-dx6d6\" (UID: \"2cc453ef-8580-4745-b57b-03f7516056a7\") " pod="openshift-monitoring/metrics-server-68dfc74784-dx6d6" Oct 31 00:15:09.329257 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:09.329109 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/2cc453ef-8580-4745-b57b-03f7516056a7-secret-metrics-server-tls\") pod \"metrics-server-68dfc74784-dx6d6\" (UID: \"2cc453ef-8580-4745-b57b-03f7516056a7\") " pod="openshift-monitoring/metrics-server-68dfc74784-dx6d6" Oct 31 00:15:09.329257 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:09.329211 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/2cc453ef-8580-4745-b57b-03f7516056a7-metrics-server-audit-profiles\") pod \"metrics-server-68dfc74784-dx6d6\" (UID: \"2cc453ef-8580-4745-b57b-03f7516056a7\") " pod="openshift-monitoring/metrics-server-68dfc74784-dx6d6" Oct 31 00:15:09.329440 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:09.329252 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/2cc453ef-8580-4745-b57b-03f7516056a7-secret-metrics-server-client-certs\") pod \"metrics-server-68dfc74784-dx6d6\" (UID: \"2cc453ef-8580-4745-b57b-03f7516056a7\") " pod="openshift-monitoring/metrics-server-68dfc74784-dx6d6" Oct 31 00:15:09.329440 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:09.329288 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/2cc453ef-8580-4745-b57b-03f7516056a7-audit-log\") pod \"metrics-server-68dfc74784-dx6d6\" (UID: \"2cc453ef-8580-4745-b57b-03f7516056a7\") " pod="openshift-monitoring/metrics-server-68dfc74784-dx6d6" Oct 31 00:15:09.329440 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:09.329330 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptxzw\" (UniqueName: \"kubernetes.io/projected/2cc453ef-8580-4745-b57b-03f7516056a7-kube-api-access-ptxzw\") pod \"metrics-server-68dfc74784-dx6d6\" (UID: \"2cc453ef-8580-4745-b57b-03f7516056a7\") " pod="openshift-monitoring/metrics-server-68dfc74784-dx6d6" Oct 31 00:15:09.430617 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:09.430543 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/2cc453ef-8580-4745-b57b-03f7516056a7-client-ca-bundle\") pod \"metrics-server-68dfc74784-dx6d6\" (UID: \"2cc453ef-8580-4745-b57b-03f7516056a7\") " pod="openshift-monitoring/metrics-server-68dfc74784-dx6d6" Oct 31 00:15:09.430617 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:09.430575 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/2cc453ef-8580-4745-b57b-03f7516056a7-secret-metrics-server-tls\") pod \"metrics-server-68dfc74784-dx6d6\" (UID: \"2cc453ef-8580-4745-b57b-03f7516056a7\") " pod="openshift-monitoring/metrics-server-68dfc74784-dx6d6" Oct 31 00:15:09.430617 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:09.430594 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/2cc453ef-8580-4745-b57b-03f7516056a7-metrics-server-audit-profiles\") pod \"metrics-server-68dfc74784-dx6d6\" (UID: \"2cc453ef-8580-4745-b57b-03f7516056a7\") " pod="openshift-monitoring/metrics-server-68dfc74784-dx6d6" Oct 31 00:15:09.430617 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:09.430612 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/2cc453ef-8580-4745-b57b-03f7516056a7-secret-metrics-server-client-certs\") pod \"metrics-server-68dfc74784-dx6d6\" (UID: \"2cc453ef-8580-4745-b57b-03f7516056a7\") " pod="openshift-monitoring/metrics-server-68dfc74784-dx6d6" Oct 31 00:15:09.430943 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:09.430730 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/2cc453ef-8580-4745-b57b-03f7516056a7-audit-log\") pod \"metrics-server-68dfc74784-dx6d6\" (UID: \"2cc453ef-8580-4745-b57b-03f7516056a7\") " pod="openshift-monitoring/metrics-server-68dfc74784-dx6d6" Oct 31 00:15:09.430943 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:09.430760 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptxzw\" (UniqueName: \"kubernetes.io/projected/2cc453ef-8580-4745-b57b-03f7516056a7-kube-api-access-ptxzw\") pod \"metrics-server-68dfc74784-dx6d6\" (UID: \"2cc453ef-8580-4745-b57b-03f7516056a7\") " pod="openshift-monitoring/metrics-server-68dfc74784-dx6d6" Oct 31 00:15:09.430943 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:09.430795 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2cc453ef-8580-4745-b57b-03f7516056a7-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-68dfc74784-dx6d6\" (UID: \"2cc453ef-8580-4745-b57b-03f7516056a7\") " pod="openshift-monitoring/metrics-server-68dfc74784-dx6d6" Oct 31 00:15:09.431073 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:09.431003 5222 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 31 00:15:09.431203 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:09.431178 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/2cc453ef-8580-4745-b57b-03f7516056a7-audit-log\") pod \"metrics-server-68dfc74784-dx6d6\" (UID: \"2cc453ef-8580-4745-b57b-03f7516056a7\") " pod="openshift-monitoring/metrics-server-68dfc74784-dx6d6" Oct 31 00:15:09.431598 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:09.431572 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2cc453ef-8580-4745-b57b-03f7516056a7-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-68dfc74784-dx6d6\" (UID: \"2cc453ef-8580-4745-b57b-03f7516056a7\") " pod="openshift-monitoring/metrics-server-68dfc74784-dx6d6" Oct 31 00:15:09.431994 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:09.431969 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/2cc453ef-8580-4745-b57b-03f7516056a7-metrics-server-audit-profiles\") pod \"metrics-server-68dfc74784-dx6d6\" (UID: \"2cc453ef-8580-4745-b57b-03f7516056a7\") " pod="openshift-monitoring/metrics-server-68dfc74784-dx6d6" Oct 31 00:15:09.433492 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:09.433472 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/2cc453ef-8580-4745-b57b-03f7516056a7-secret-metrics-server-tls\") pod \"metrics-server-68dfc74784-dx6d6\" (UID: \"2cc453ef-8580-4745-b57b-03f7516056a7\") " pod="openshift-monitoring/metrics-server-68dfc74784-dx6d6" Oct 31 00:15:09.433588 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:09.433508 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/2cc453ef-8580-4745-b57b-03f7516056a7-client-ca-bundle\") pod \"metrics-server-68dfc74784-dx6d6\" (UID: \"2cc453ef-8580-4745-b57b-03f7516056a7\") " pod="openshift-monitoring/metrics-server-68dfc74784-dx6d6" Oct 31 00:15:09.433588 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:09.433557 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/2cc453ef-8580-4745-b57b-03f7516056a7-secret-metrics-server-client-certs\") pod \"metrics-server-68dfc74784-dx6d6\" (UID: \"2cc453ef-8580-4745-b57b-03f7516056a7\") " pod="openshift-monitoring/metrics-server-68dfc74784-dx6d6" Oct 31 00:15:09.494383 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:09.494339 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptxzw\" (UniqueName: \"kubernetes.io/projected/2cc453ef-8580-4745-b57b-03f7516056a7-kube-api-access-ptxzw\") pod \"metrics-server-68dfc74784-dx6d6\" (UID: \"2cc453ef-8580-4745-b57b-03f7516056a7\") " pod="openshift-monitoring/metrics-server-68dfc74784-dx6d6" Oct 31 00:15:09.540722 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:09.540686 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-68dfc74784-dx6d6" Oct 31 00:15:09.716200 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:09.716116 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dbcsc" event={"ID":"1b88d3b9-589f-4f4e-b8a6-872f4fb5536b","Type":"ContainerStarted","Data":"ebe94d6dd57fc1a4fba8270bb0489726cbf1cd6f241c2ee0b0664ab5d849a725"} Oct 31 00:15:09.716200 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:09.716173 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dbcsc" event={"ID":"1b88d3b9-589f-4f4e-b8a6-872f4fb5536b","Type":"ContainerStarted","Data":"c3fe07d575eba4859046d134ae1b6b28991538e35d73f92d9f798473fd301bd2"} Oct 31 00:15:09.934199 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:09.934159 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-68dfc74784-dx6d6"] Oct 31 00:15:09.950400 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:09.950329 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cc453ef_8580_4745_b57b_03f7516056a7.slice/crio-cd2f140364234b787aa93c38aba58214d9b11eaff48bf8dd4ef083048aab7f8d WatchSource:0}: Error finding container cd2f140364234b787aa93c38aba58214d9b11eaff48bf8dd4ef083048aab7f8d: Status 404 returned error can't find the container with id cd2f140364234b787aa93c38aba58214d9b11eaff48bf8dd4ef083048aab7f8d Oct 31 00:15:10.381644 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:10.381612 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:15:10.727591 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:10.727507 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rsf6d" event={"ID":"ea9a5160-4696-4b75-8a10-0fbc54b2ad04","Type":"ContainerStarted","Data":"ebd35ff1a42409bba6b4bd7111957c31ed343a145fdfd74b1d71d4ad4c61a962"} Oct 31 00:15:10.727591 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:10.727574 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rsf6d" event={"ID":"ea9a5160-4696-4b75-8a10-0fbc54b2ad04","Type":"ContainerStarted","Data":"bbf8b0f1ee3c67130ab0c653c4451c296bf571c595252a8155d78b9bc70be009"} Oct 31 00:15:10.728068 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:10.727607 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rsf6d" event={"ID":"ea9a5160-4696-4b75-8a10-0fbc54b2ad04","Type":"ContainerStarted","Data":"90f8b549a4f7ec5b2166df3bad11ccfcd60421136f7faa0733eed1c0dfab194a"} Oct 31 00:15:10.728871 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:10.728847 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-68dfc74784-dx6d6" event={"ID":"2cc453ef-8580-4745-b57b-03f7516056a7","Type":"ContainerStarted","Data":"cd2f140364234b787aa93c38aba58214d9b11eaff48bf8dd4ef083048aab7f8d"} Oct 31 00:15:11.734793 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:11.734755 5222 generic.go:356] "Generic (PLEG): container finished" podID="26aba8fc-e2da-430f-b0d8-6ee45468441d" containerID="bc1fedc7ac40dc097d38d3e6b2fed32c26a3c8fa7ba633e52174d0cc3819d1ca" exitCode=0 Oct 31 00:15:11.735242 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:11.734835 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"26aba8fc-e2da-430f-b0d8-6ee45468441d","Type":"ContainerDied","Data":"bc1fedc7ac40dc097d38d3e6b2fed32c26a3c8fa7ba633e52174d0cc3819d1ca"} Oct 31 00:15:11.870902 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:11.870864 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-lrh4r" Oct 31 00:15:12.741027 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:12.740923 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-68dfc74784-dx6d6" event={"ID":"2cc453ef-8580-4745-b57b-03f7516056a7","Type":"ContainerStarted","Data":"2220e892daef2e0479fe0047086d8136daaa1d6b45b29665b9e04f8e8f4235f1"} Oct 31 00:15:12.861672 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:12.861614 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-68dfc74784-dx6d6" podStartSLOduration=1.385973533 podStartE2EDuration="3.861597945s" totalImagesPullingTime="2.475624412s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:15:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:15:12.860956055 +0000 UTC m=+5.332125804" watchObservedRunningTime="2026-10-31 00:15:12.861597945 +0000 UTC m=+5.332767698" Oct 31 00:15:13.209486 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:13.209455 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-ml8n5" Oct 31 00:15:14.751007 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:14.750969 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"26aba8fc-e2da-430f-b0d8-6ee45468441d","Type":"ContainerStarted","Data":"e5feaf6f21a314406a7e43b9f37f4d13ceb9dc02ffa03cdeceec2b53d27424ad"} Oct 31 00:15:14.751390 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:14.751060 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"26aba8fc-e2da-430f-b0d8-6ee45468441d","Type":"ContainerStarted","Data":"e7ff2a63615b3ad0d56475374f903d09361d6f69bd02735970a97950a843878e"} Oct 31 00:15:14.751390 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:14.751086 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"26aba8fc-e2da-430f-b0d8-6ee45468441d","Type":"ContainerStarted","Data":"2863d9cb4d6566e2e47954688ca8308131271a1c3f3c08a461bf1a65a9361bd8"} Oct 31 00:15:14.751390 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:14.751110 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"26aba8fc-e2da-430f-b0d8-6ee45468441d","Type":"ContainerStarted","Data":"a5d1a9b4427b9e1b72476e20b21f459edc4d5fff321004f6a5000e973a958c49"} Oct 31 00:15:14.751390 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:14.751131 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"26aba8fc-e2da-430f-b0d8-6ee45468441d","Type":"ContainerStarted","Data":"18c9379d2fb95e32c95525a0b00a4e8decc49e9c5c0cc5b40f836cf911d2924a"} Oct 31 00:15:16.670211 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:16.670178 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:15:16.670589 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:16.670428 5222 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 00:15:16.762254 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:16.762221 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"26aba8fc-e2da-430f-b0d8-6ee45468441d","Type":"ContainerStarted","Data":"36fbb56dcdbbd7f6efde984010aa55ff0a639fc0dcbae836fe4e466181549f13"} Oct 31 00:15:18.831846 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:18.831809 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6644876fb8-9bszd" Oct 31 00:15:21.388036 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:21.388005 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-59c7d45ccf-r8m6l"] Oct 31 00:15:21.392112 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:21.392093 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59c7d45ccf-r8m6l" Oct 31 00:15:21.440586 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:21.440542 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8c8bf8e8-c2a1-4c3e-ae23-9446233d0924-console-config\") pod \"console-59c7d45ccf-r8m6l\" (UID: \"8c8bf8e8-c2a1-4c3e-ae23-9446233d0924\") " pod="openshift-console/console-59c7d45ccf-r8m6l" Oct 31 00:15:21.440586 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:21.440585 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8c8bf8e8-c2a1-4c3e-ae23-9446233d0924-tmp\") pod \"console-59c7d45ccf-r8m6l\" (UID: \"8c8bf8e8-c2a1-4c3e-ae23-9446233d0924\") " pod="openshift-console/console-59c7d45ccf-r8m6l" Oct 31 00:15:21.440824 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:21.440808 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grw4v\" (UniqueName: \"kubernetes.io/projected/8c8bf8e8-c2a1-4c3e-ae23-9446233d0924-kube-api-access-grw4v\") pod \"console-59c7d45ccf-r8m6l\" (UID: \"8c8bf8e8-c2a1-4c3e-ae23-9446233d0924\") " pod="openshift-console/console-59c7d45ccf-r8m6l" Oct 31 00:15:21.440869 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:21.440833 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8c8bf8e8-c2a1-4c3e-ae23-9446233d0924-service-ca\") pod \"console-59c7d45ccf-r8m6l\" (UID: \"8c8bf8e8-c2a1-4c3e-ae23-9446233d0924\") " pod="openshift-console/console-59c7d45ccf-r8m6l" Oct 31 00:15:21.440969 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:21.440934 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8c8bf8e8-c2a1-4c3e-ae23-9446233d0924-oauth-serving-cert\") pod \"console-59c7d45ccf-r8m6l\" (UID: \"8c8bf8e8-c2a1-4c3e-ae23-9446233d0924\") " pod="openshift-console/console-59c7d45ccf-r8m6l" Oct 31 00:15:21.441079 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:21.441058 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8c8bf8e8-c2a1-4c3e-ae23-9446233d0924-console-oauth-config\") pod \"console-59c7d45ccf-r8m6l\" (UID: \"8c8bf8e8-c2a1-4c3e-ae23-9446233d0924\") " pod="openshift-console/console-59c7d45ccf-r8m6l" Oct 31 00:15:21.441116 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:21.441087 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c8bf8e8-c2a1-4c3e-ae23-9446233d0924-trusted-ca-bundle\") pod \"console-59c7d45ccf-r8m6l\" (UID: \"8c8bf8e8-c2a1-4c3e-ae23-9446233d0924\") " pod="openshift-console/console-59c7d45ccf-r8m6l" Oct 31 00:15:21.441153 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:21.441113 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8c8bf8e8-c2a1-4c3e-ae23-9446233d0924-console-serving-cert\") pod \"console-59c7d45ccf-r8m6l\" (UID: \"8c8bf8e8-c2a1-4c3e-ae23-9446233d0924\") " pod="openshift-console/console-59c7d45ccf-r8m6l" Oct 31 00:15:21.500197 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:21.500166 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Oct 31 00:15:21.500197 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:21.500167 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Oct 31 00:15:21.500197 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:21.500167 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Oct 31 00:15:21.500523 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:21.500405 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-vpkbr\"" Oct 31 00:15:21.500523 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:21.500404 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Oct 31 00:15:21.500523 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:21.500440 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Oct 31 00:15:21.542333 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:21.542297 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8c8bf8e8-c2a1-4c3e-ae23-9446233d0924-oauth-serving-cert\") pod \"console-59c7d45ccf-r8m6l\" (UID: \"8c8bf8e8-c2a1-4c3e-ae23-9446233d0924\") " pod="openshift-console/console-59c7d45ccf-r8m6l" Oct 31 00:15:21.542532 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:21.542350 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8c8bf8e8-c2a1-4c3e-ae23-9446233d0924-console-oauth-config\") pod \"console-59c7d45ccf-r8m6l\" (UID: \"8c8bf8e8-c2a1-4c3e-ae23-9446233d0924\") " pod="openshift-console/console-59c7d45ccf-r8m6l" Oct 31 00:15:21.542532 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:21.542390 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c8bf8e8-c2a1-4c3e-ae23-9446233d0924-trusted-ca-bundle\") pod \"console-59c7d45ccf-r8m6l\" (UID: \"8c8bf8e8-c2a1-4c3e-ae23-9446233d0924\") " pod="openshift-console/console-59c7d45ccf-r8m6l" Oct 31 00:15:21.542532 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:21.542428 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8c8bf8e8-c2a1-4c3e-ae23-9446233d0924-console-serving-cert\") pod \"console-59c7d45ccf-r8m6l\" (UID: \"8c8bf8e8-c2a1-4c3e-ae23-9446233d0924\") " pod="openshift-console/console-59c7d45ccf-r8m6l" Oct 31 00:15:21.542532 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:21.542468 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8c8bf8e8-c2a1-4c3e-ae23-9446233d0924-console-config\") pod \"console-59c7d45ccf-r8m6l\" (UID: \"8c8bf8e8-c2a1-4c3e-ae23-9446233d0924\") " pod="openshift-console/console-59c7d45ccf-r8m6l" Oct 31 00:15:21.542532 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:21.542491 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8c8bf8e8-c2a1-4c3e-ae23-9446233d0924-tmp\") pod \"console-59c7d45ccf-r8m6l\" (UID: \"8c8bf8e8-c2a1-4c3e-ae23-9446233d0924\") " pod="openshift-console/console-59c7d45ccf-r8m6l" Oct 31 00:15:21.542532 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:21.542532 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grw4v\" (UniqueName: \"kubernetes.io/projected/8c8bf8e8-c2a1-4c3e-ae23-9446233d0924-kube-api-access-grw4v\") pod \"console-59c7d45ccf-r8m6l\" (UID: \"8c8bf8e8-c2a1-4c3e-ae23-9446233d0924\") " pod="openshift-console/console-59c7d45ccf-r8m6l" Oct 31 00:15:21.542854 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:21.542831 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8c8bf8e8-c2a1-4c3e-ae23-9446233d0924-service-ca\") pod \"console-59c7d45ccf-r8m6l\" (UID: \"8c8bf8e8-c2a1-4c3e-ae23-9446233d0924\") " pod="openshift-console/console-59c7d45ccf-r8m6l" Oct 31 00:15:21.542994 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:21.542963 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8c8bf8e8-c2a1-4c3e-ae23-9446233d0924-tmp\") pod \"console-59c7d45ccf-r8m6l\" (UID: \"8c8bf8e8-c2a1-4c3e-ae23-9446233d0924\") " pod="openshift-console/console-59c7d45ccf-r8m6l" Oct 31 00:15:21.543239 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:21.543194 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8c8bf8e8-c2a1-4c3e-ae23-9446233d0924-oauth-serving-cert\") pod \"console-59c7d45ccf-r8m6l\" (UID: \"8c8bf8e8-c2a1-4c3e-ae23-9446233d0924\") " pod="openshift-console/console-59c7d45ccf-r8m6l" Oct 31 00:15:21.543492 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:21.543472 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8c8bf8e8-c2a1-4c3e-ae23-9446233d0924-console-config\") pod \"console-59c7d45ccf-r8m6l\" (UID: \"8c8bf8e8-c2a1-4c3e-ae23-9446233d0924\") " pod="openshift-console/console-59c7d45ccf-r8m6l" Oct 31 00:15:21.543961 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:21.543932 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8c8bf8e8-c2a1-4c3e-ae23-9446233d0924-service-ca\") pod \"console-59c7d45ccf-r8m6l\" (UID: \"8c8bf8e8-c2a1-4c3e-ae23-9446233d0924\") " pod="openshift-console/console-59c7d45ccf-r8m6l" Oct 31 00:15:21.545389 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:21.545354 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8c8bf8e8-c2a1-4c3e-ae23-9446233d0924-console-oauth-config\") pod \"console-59c7d45ccf-r8m6l\" (UID: \"8c8bf8e8-c2a1-4c3e-ae23-9446233d0924\") " pod="openshift-console/console-59c7d45ccf-r8m6l" Oct 31 00:15:21.545486 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:21.545435 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8c8bf8e8-c2a1-4c3e-ae23-9446233d0924-console-serving-cert\") pod \"console-59c7d45ccf-r8m6l\" (UID: \"8c8bf8e8-c2a1-4c3e-ae23-9446233d0924\") " pod="openshift-console/console-59c7d45ccf-r8m6l" Oct 31 00:15:21.605244 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:21.605210 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Oct 31 00:15:21.607712 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:21.607645 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-59c7d45ccf-r8m6l"] Oct 31 00:15:21.610438 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:21.610407 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grw4v\" (UniqueName: \"kubernetes.io/projected/8c8bf8e8-c2a1-4c3e-ae23-9446233d0924-kube-api-access-grw4v\") pod \"console-59c7d45ccf-r8m6l\" (UID: \"8c8bf8e8-c2a1-4c3e-ae23-9446233d0924\") " pod="openshift-console/console-59c7d45ccf-r8m6l" Oct 31 00:15:21.613382 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:21.613338 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c8bf8e8-c2a1-4c3e-ae23-9446233d0924-trusted-ca-bundle\") pod \"console-59c7d45ccf-r8m6l\" (UID: \"8c8bf8e8-c2a1-4c3e-ae23-9446233d0924\") " pod="openshift-console/console-59c7d45ccf-r8m6l" Oct 31 00:15:21.707245 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:21.707163 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59c7d45ccf-r8m6l" Oct 31 00:15:22.084321 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:15:22.084288 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c8bf8e8_c2a1_4c3e_ae23_9446233d0924.slice/crio-f214561a97a576ec851ee7043c67238108c0a061bbdfdb02531b7969325c2a41 WatchSource:0}: Error finding container f214561a97a576ec851ee7043c67238108c0a061bbdfdb02531b7969325c2a41: Status 404 returned error can't find the container with id f214561a97a576ec851ee7043c67238108c0a061bbdfdb02531b7969325c2a41 Oct 31 00:15:22.084884 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:22.084853 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-59c7d45ccf-r8m6l"] Oct 31 00:15:22.785979 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:22.785934 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59c7d45ccf-r8m6l" event={"ID":"8c8bf8e8-c2a1-4c3e-ae23-9446233d0924","Type":"ContainerStarted","Data":"f214561a97a576ec851ee7043c67238108c0a061bbdfdb02531b7969325c2a41"} Oct 31 00:15:22.807804 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:22.807765 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-68dfc74784-dx6d6" Oct 31 00:15:23.706720 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:23.706632 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-68dfc74784-dx6d6" Oct 31 00:15:25.800931 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:25.800841 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59c7d45ccf-r8m6l" event={"ID":"8c8bf8e8-c2a1-4c3e-ae23-9446233d0924","Type":"ContainerStarted","Data":"41ba21539915ea1331b33805c4c4e714870b479dfb8c0e57d8d152ece64d4850"} Oct 31 00:15:25.925458 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:25.925400 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-59c7d45ccf-r8m6l" podStartSLOduration=1.478041578 podStartE2EDuration="4.925359573s" totalImagesPullingTime="3.447317995s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:15:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:15:25.925017435 +0000 UTC m=+18.396187184" watchObservedRunningTime="2026-10-31 00:15:25.925359573 +0000 UTC m=+18.396529313" Oct 31 00:15:29.819727 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:29.819684 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-nbn2h" event={"ID":"c61956a7-ba51-4063-bc25-aaa418168bc2","Type":"ContainerStarted","Data":"46a55c923612add46ecf95a6c65cfd5f5e779c7dc11fef11f71ecce9566bb439"} Oct 31 00:15:29.820189 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:29.820175 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-nbn2h" Oct 31 00:15:29.836561 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:29.836529 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-nbn2h" Oct 31 00:15:30.793457 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:30.793419 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-h9hsf" Oct 31 00:15:31.707903 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:31.707860 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-59c7d45ccf-r8m6l" Oct 31 00:15:31.708362 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:31.707942 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-59c7d45ccf-r8m6l" Oct 31 00:15:31.713692 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:31.713664 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-59c7d45ccf-r8m6l" Oct 31 00:15:31.832424 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:31.832396 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-59c7d45ccf-r8m6l" Oct 31 00:15:35.894643 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:15:35.894602 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 00:15:37.214469 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:37.214436 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-lvp2h" Oct 31 00:15:42.813203 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:42.813172 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-68dfc74784-dx6d6" Oct 31 00:15:42.817053 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:15:42.817029 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-68dfc74784-dx6d6" Oct 31 00:16:05.890151 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:16:05.890112 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 00:16:29.992560 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:29.992525 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 00:16:29.992974 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:29.992859 5222 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="26aba8fc-e2da-430f-b0d8-6ee45468441d" containerName="alertmanager" containerID="cri-o://18c9379d2fb95e32c95525a0b00a4e8decc49e9c5c0cc5b40f836cf911d2924a" gracePeriod=120 Oct 31 00:16:29.992974 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:29.992892 5222 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="26aba8fc-e2da-430f-b0d8-6ee45468441d" containerName="kube-rbac-proxy-web" containerID="cri-o://2863d9cb4d6566e2e47954688ca8308131271a1c3f3c08a461bf1a65a9361bd8" gracePeriod=120 Oct 31 00:16:29.992974 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:29.992885 5222 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="26aba8fc-e2da-430f-b0d8-6ee45468441d" containerName="kube-rbac-proxy-metric" containerID="cri-o://e5feaf6f21a314406a7e43b9f37f4d13ceb9dc02ffa03cdeceec2b53d27424ad" gracePeriod=120 Oct 31 00:16:29.992974 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:29.992939 5222 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="26aba8fc-e2da-430f-b0d8-6ee45468441d" containerName="prom-label-proxy" containerID="cri-o://36fbb56dcdbbd7f6efde984010aa55ff0a639fc0dcbae836fe4e466181549f13" gracePeriod=120 Oct 31 00:16:29.993166 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:29.992963 5222 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="26aba8fc-e2da-430f-b0d8-6ee45468441d" containerName="kube-rbac-proxy" containerID="cri-o://e7ff2a63615b3ad0d56475374f903d09361d6f69bd02735970a97950a843878e" gracePeriod=120 Oct 31 00:16:29.993166 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:29.992975 5222 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="26aba8fc-e2da-430f-b0d8-6ee45468441d" containerName="config-reloader" containerID="cri-o://a5d1a9b4427b9e1b72476e20b21f459edc4d5fff321004f6a5000e973a958c49" gracePeriod=120 Oct 31 00:16:31.041597 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:31.041561 5222 generic.go:356] "Generic (PLEG): container finished" podID="26aba8fc-e2da-430f-b0d8-6ee45468441d" containerID="36fbb56dcdbbd7f6efde984010aa55ff0a639fc0dcbae836fe4e466181549f13" exitCode=0 Oct 31 00:16:31.041597 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:31.041587 5222 generic.go:356] "Generic (PLEG): container finished" podID="26aba8fc-e2da-430f-b0d8-6ee45468441d" containerID="e5feaf6f21a314406a7e43b9f37f4d13ceb9dc02ffa03cdeceec2b53d27424ad" exitCode=0 Oct 31 00:16:31.041597 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:31.041594 5222 generic.go:356] "Generic (PLEG): container finished" podID="26aba8fc-e2da-430f-b0d8-6ee45468441d" containerID="e7ff2a63615b3ad0d56475374f903d09361d6f69bd02735970a97950a843878e" exitCode=0 Oct 31 00:16:31.041597 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:31.041601 5222 generic.go:356] "Generic (PLEG): container finished" podID="26aba8fc-e2da-430f-b0d8-6ee45468441d" containerID="a5d1a9b4427b9e1b72476e20b21f459edc4d5fff321004f6a5000e973a958c49" exitCode=0 Oct 31 00:16:31.041597 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:31.041607 5222 generic.go:356] "Generic (PLEG): container finished" podID="26aba8fc-e2da-430f-b0d8-6ee45468441d" containerID="18c9379d2fb95e32c95525a0b00a4e8decc49e9c5c0cc5b40f836cf911d2924a" exitCode=0 Oct 31 00:16:31.042170 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:31.041635 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"26aba8fc-e2da-430f-b0d8-6ee45468441d","Type":"ContainerDied","Data":"36fbb56dcdbbd7f6efde984010aa55ff0a639fc0dcbae836fe4e466181549f13"} Oct 31 00:16:31.042170 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:31.041717 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"26aba8fc-e2da-430f-b0d8-6ee45468441d","Type":"ContainerDied","Data":"e5feaf6f21a314406a7e43b9f37f4d13ceb9dc02ffa03cdeceec2b53d27424ad"} Oct 31 00:16:31.042170 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:31.041767 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"26aba8fc-e2da-430f-b0d8-6ee45468441d","Type":"ContainerDied","Data":"e7ff2a63615b3ad0d56475374f903d09361d6f69bd02735970a97950a843878e"} Oct 31 00:16:31.042170 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:31.041807 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"26aba8fc-e2da-430f-b0d8-6ee45468441d","Type":"ContainerDied","Data":"a5d1a9b4427b9e1b72476e20b21f459edc4d5fff321004f6a5000e973a958c49"} Oct 31 00:16:31.042170 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:31.041843 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"26aba8fc-e2da-430f-b0d8-6ee45468441d","Type":"ContainerDied","Data":"18c9379d2fb95e32c95525a0b00a4e8decc49e9c5c0cc5b40f836cf911d2924a"} Oct 31 00:16:31.252095 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:31.252070 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:16:31.418923 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:31.418835 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26aba8fc-e2da-430f-b0d8-6ee45468441d-tls-assets\" (UniqueName: \"kubernetes.io/projected/26aba8fc-e2da-430f-b0d8-6ee45468441d-tls-assets\") pod \"26aba8fc-e2da-430f-b0d8-6ee45468441d\" (UID: \"26aba8fc-e2da-430f-b0d8-6ee45468441d\") " Oct 31 00:16:31.418923 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:31.418888 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26aba8fc-e2da-430f-b0d8-6ee45468441d-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26aba8fc-e2da-430f-b0d8-6ee45468441d-alertmanager-trusted-ca-bundle\") pod \"26aba8fc-e2da-430f-b0d8-6ee45468441d\" (UID: \"26aba8fc-e2da-430f-b0d8-6ee45468441d\") " Oct 31 00:16:31.418923 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:31.418920 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26aba8fc-e2da-430f-b0d8-6ee45468441d-config-volume\" (UniqueName: \"kubernetes.io/secret/26aba8fc-e2da-430f-b0d8-6ee45468441d-config-volume\") pod \"26aba8fc-e2da-430f-b0d8-6ee45468441d\" (UID: \"26aba8fc-e2da-430f-b0d8-6ee45468441d\") " Oct 31 00:16:31.419214 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:31.418940 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26aba8fc-e2da-430f-b0d8-6ee45468441d-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/26aba8fc-e2da-430f-b0d8-6ee45468441d-secret-alertmanager-kube-rbac-proxy\") pod \"26aba8fc-e2da-430f-b0d8-6ee45468441d\" (UID: \"26aba8fc-e2da-430f-b0d8-6ee45468441d\") " Oct 31 00:16:31.419214 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:31.418969 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26aba8fc-e2da-430f-b0d8-6ee45468441d-config-out\" (UniqueName: \"kubernetes.io/empty-dir/26aba8fc-e2da-430f-b0d8-6ee45468441d-config-out\") pod \"26aba8fc-e2da-430f-b0d8-6ee45468441d\" (UID: \"26aba8fc-e2da-430f-b0d8-6ee45468441d\") " Oct 31 00:16:31.419214 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:31.419003 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26aba8fc-e2da-430f-b0d8-6ee45468441d-web-config\" (UniqueName: \"kubernetes.io/secret/26aba8fc-e2da-430f-b0d8-6ee45468441d-web-config\") pod \"26aba8fc-e2da-430f-b0d8-6ee45468441d\" (UID: \"26aba8fc-e2da-430f-b0d8-6ee45468441d\") " Oct 31 00:16:31.419214 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:31.419047 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26aba8fc-e2da-430f-b0d8-6ee45468441d-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/26aba8fc-e2da-430f-b0d8-6ee45468441d-metrics-client-ca\") pod \"26aba8fc-e2da-430f-b0d8-6ee45468441d\" (UID: \"26aba8fc-e2da-430f-b0d8-6ee45468441d\") " Oct 31 00:16:31.419214 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:31.419084 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26aba8fc-e2da-430f-b0d8-6ee45468441d-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/26aba8fc-e2da-430f-b0d8-6ee45468441d-secret-alertmanager-kube-rbac-proxy-web\") pod \"26aba8fc-e2da-430f-b0d8-6ee45468441d\" (UID: \"26aba8fc-e2da-430f-b0d8-6ee45468441d\") " Oct 31 00:16:31.419214 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:31.419117 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26aba8fc-e2da-430f-b0d8-6ee45468441d-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/26aba8fc-e2da-430f-b0d8-6ee45468441d-cluster-tls-config\") pod \"26aba8fc-e2da-430f-b0d8-6ee45468441d\" (UID: \"26aba8fc-e2da-430f-b0d8-6ee45468441d\") " Oct 31 00:16:31.419214 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:31.419146 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26aba8fc-e2da-430f-b0d8-6ee45468441d-kube-api-access-4v246\" (UniqueName: \"kubernetes.io/projected/26aba8fc-e2da-430f-b0d8-6ee45468441d-kube-api-access-4v246\") pod \"26aba8fc-e2da-430f-b0d8-6ee45468441d\" (UID: \"26aba8fc-e2da-430f-b0d8-6ee45468441d\") " Oct 31 00:16:31.419214 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:31.419174 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26aba8fc-e2da-430f-b0d8-6ee45468441d-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/26aba8fc-e2da-430f-b0d8-6ee45468441d-alertmanager-main-db\") pod \"26aba8fc-e2da-430f-b0d8-6ee45468441d\" (UID: \"26aba8fc-e2da-430f-b0d8-6ee45468441d\") " Oct 31 00:16:31.419214 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:31.419211 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26aba8fc-e2da-430f-b0d8-6ee45468441d-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/26aba8fc-e2da-430f-b0d8-6ee45468441d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"26aba8fc-e2da-430f-b0d8-6ee45468441d\" (UID: \"26aba8fc-e2da-430f-b0d8-6ee45468441d\") " Oct 31 00:16:31.419685 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:31.419267 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26aba8fc-e2da-430f-b0d8-6ee45468441d-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/26aba8fc-e2da-430f-b0d8-6ee45468441d-secret-alertmanager-main-tls\") pod \"26aba8fc-e2da-430f-b0d8-6ee45468441d\" (UID: \"26aba8fc-e2da-430f-b0d8-6ee45468441d\") " Oct 31 00:16:31.419685 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:31.419452 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26aba8fc-e2da-430f-b0d8-6ee45468441d-alertmanager-trusted-ca-bundle" pod "26aba8fc-e2da-430f-b0d8-6ee45468441d" (UID: "26aba8fc-e2da-430f-b0d8-6ee45468441d"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:16:31.419685 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:31.419569 5222 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26aba8fc-e2da-430f-b0d8-6ee45468441d-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:31.420330 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:31.419957 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26aba8fc-e2da-430f-b0d8-6ee45468441d-alertmanager-main-db" pod "26aba8fc-e2da-430f-b0d8-6ee45468441d" (UID: "26aba8fc-e2da-430f-b0d8-6ee45468441d"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:16:31.420330 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:31.420255 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26aba8fc-e2da-430f-b0d8-6ee45468441d-metrics-client-ca" pod "26aba8fc-e2da-430f-b0d8-6ee45468441d" (UID: "26aba8fc-e2da-430f-b0d8-6ee45468441d"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:16:31.422941 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:31.422893 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26aba8fc-e2da-430f-b0d8-6ee45468441d-config-out" pod "26aba8fc-e2da-430f-b0d8-6ee45468441d" (UID: "26aba8fc-e2da-430f-b0d8-6ee45468441d"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:16:31.423158 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:31.423123 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26aba8fc-e2da-430f-b0d8-6ee45468441d-config-volume" pod "26aba8fc-e2da-430f-b0d8-6ee45468441d" (UID: "26aba8fc-e2da-430f-b0d8-6ee45468441d"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:16:31.423544 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:31.423515 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26aba8fc-e2da-430f-b0d8-6ee45468441d-secret-alertmanager-main-tls" pod "26aba8fc-e2da-430f-b0d8-6ee45468441d" (UID: "26aba8fc-e2da-430f-b0d8-6ee45468441d"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:16:31.423647 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:31.423600 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26aba8fc-e2da-430f-b0d8-6ee45468441d-kube-api-access-4v246" pod "26aba8fc-e2da-430f-b0d8-6ee45468441d" (UID: "26aba8fc-e2da-430f-b0d8-6ee45468441d"). InnerVolumeSpecName "kube-api-access-4v246". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:16:31.423647 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:31.423618 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26aba8fc-e2da-430f-b0d8-6ee45468441d-secret-alertmanager-kube-rbac-proxy-metric" pod "26aba8fc-e2da-430f-b0d8-6ee45468441d" (UID: "26aba8fc-e2da-430f-b0d8-6ee45468441d"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:16:31.423779 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:31.423754 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26aba8fc-e2da-430f-b0d8-6ee45468441d-tls-assets" pod "26aba8fc-e2da-430f-b0d8-6ee45468441d" (UID: "26aba8fc-e2da-430f-b0d8-6ee45468441d"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:16:31.423840 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:31.423792 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26aba8fc-e2da-430f-b0d8-6ee45468441d-secret-alertmanager-kube-rbac-proxy-web" pod "26aba8fc-e2da-430f-b0d8-6ee45468441d" (UID: "26aba8fc-e2da-430f-b0d8-6ee45468441d"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:16:31.423918 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:31.423901 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26aba8fc-e2da-430f-b0d8-6ee45468441d-secret-alertmanager-kube-rbac-proxy" pod "26aba8fc-e2da-430f-b0d8-6ee45468441d" (UID: "26aba8fc-e2da-430f-b0d8-6ee45468441d"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:16:31.427260 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:31.427239 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26aba8fc-e2da-430f-b0d8-6ee45468441d-cluster-tls-config" pod "26aba8fc-e2da-430f-b0d8-6ee45468441d" (UID: "26aba8fc-e2da-430f-b0d8-6ee45468441d"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:16:31.434990 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:31.434963 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26aba8fc-e2da-430f-b0d8-6ee45468441d-web-config" pod "26aba8fc-e2da-430f-b0d8-6ee45468441d" (UID: "26aba8fc-e2da-430f-b0d8-6ee45468441d"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:16:31.520079 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:31.520037 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4v246\" (UniqueName: \"kubernetes.io/projected/26aba8fc-e2da-430f-b0d8-6ee45468441d-kube-api-access-4v246\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:31.520079 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:31.520066 5222 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/26aba8fc-e2da-430f-b0d8-6ee45468441d-alertmanager-main-db\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:31.520079 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:31.520079 5222 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/26aba8fc-e2da-430f-b0d8-6ee45468441d-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:31.520079 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:31.520089 5222 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/26aba8fc-e2da-430f-b0d8-6ee45468441d-secret-alertmanager-main-tls\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:31.520355 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:31.520101 5222 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/26aba8fc-e2da-430f-b0d8-6ee45468441d-tls-assets\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:31.520355 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:31.520110 5222 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/26aba8fc-e2da-430f-b0d8-6ee45468441d-config-volume\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:31.520355 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:31.520118 5222 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/26aba8fc-e2da-430f-b0d8-6ee45468441d-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:31.520355 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:31.520127 5222 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/26aba8fc-e2da-430f-b0d8-6ee45468441d-config-out\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:31.520355 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:31.520137 5222 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/26aba8fc-e2da-430f-b0d8-6ee45468441d-web-config\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:31.520355 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:31.520146 5222 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/26aba8fc-e2da-430f-b0d8-6ee45468441d-metrics-client-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:31.520355 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:31.520156 5222 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/26aba8fc-e2da-430f-b0d8-6ee45468441d-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:31.520355 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:31.520164 5222 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/26aba8fc-e2da-430f-b0d8-6ee45468441d-cluster-tls-config\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:32.052680 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:32.052643 5222 generic.go:356] "Generic (PLEG): container finished" podID="26aba8fc-e2da-430f-b0d8-6ee45468441d" containerID="2863d9cb4d6566e2e47954688ca8308131271a1c3f3c08a461bf1a65a9361bd8" exitCode=0 Oct 31 00:16:32.053103 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:32.052726 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:16:32.053103 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:32.052729 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"26aba8fc-e2da-430f-b0d8-6ee45468441d","Type":"ContainerDied","Data":"2863d9cb4d6566e2e47954688ca8308131271a1c3f3c08a461bf1a65a9361bd8"} Oct 31 00:16:32.053103 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:32.052827 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"26aba8fc-e2da-430f-b0d8-6ee45468441d","Type":"ContainerDied","Data":"e352133d4a0ba734426022a4680011129d2529bcf53892b04212d2b156322dd4"} Oct 31 00:16:32.053103 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:32.052871 5222 scope.go:122] "RemoveContainer" containerID="36fbb56dcdbbd7f6efde984010aa55ff0a639fc0dcbae836fe4e466181549f13" Oct 31 00:16:32.060746 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:32.060718 5222 scope.go:122] "RemoveContainer" containerID="e5feaf6f21a314406a7e43b9f37f4d13ceb9dc02ffa03cdeceec2b53d27424ad" Oct 31 00:16:32.068102 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:32.068076 5222 scope.go:122] "RemoveContainer" containerID="e7ff2a63615b3ad0d56475374f903d09361d6f69bd02735970a97950a843878e" Oct 31 00:16:32.075477 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:32.075453 5222 scope.go:122] "RemoveContainer" containerID="2863d9cb4d6566e2e47954688ca8308131271a1c3f3c08a461bf1a65a9361bd8" Oct 31 00:16:32.082521 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:32.082502 5222 scope.go:122] "RemoveContainer" containerID="a5d1a9b4427b9e1b72476e20b21f459edc4d5fff321004f6a5000e973a958c49" Oct 31 00:16:32.089931 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:32.089903 5222 scope.go:122] "RemoveContainer" containerID="18c9379d2fb95e32c95525a0b00a4e8decc49e9c5c0cc5b40f836cf911d2924a" Oct 31 00:16:32.098777 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:32.098753 5222 scope.go:122] "RemoveContainer" containerID="bc1fedc7ac40dc097d38d3e6b2fed32c26a3c8fa7ba633e52174d0cc3819d1ca" Oct 31 00:16:32.106216 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:32.106192 5222 scope.go:122] "RemoveContainer" containerID="36fbb56dcdbbd7f6efde984010aa55ff0a639fc0dcbae836fe4e466181549f13" Oct 31 00:16:32.106593 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:16:32.106570 5222 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36fbb56dcdbbd7f6efde984010aa55ff0a639fc0dcbae836fe4e466181549f13\": container with ID starting with 36fbb56dcdbbd7f6efde984010aa55ff0a639fc0dcbae836fe4e466181549f13 not found: ID does not exist" containerID="36fbb56dcdbbd7f6efde984010aa55ff0a639fc0dcbae836fe4e466181549f13" Oct 31 00:16:32.106679 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:32.106604 5222 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36fbb56dcdbbd7f6efde984010aa55ff0a639fc0dcbae836fe4e466181549f13"} err="failed to get container status \"36fbb56dcdbbd7f6efde984010aa55ff0a639fc0dcbae836fe4e466181549f13\": rpc error: code = NotFound desc = could not find container \"36fbb56dcdbbd7f6efde984010aa55ff0a639fc0dcbae836fe4e466181549f13\": container with ID starting with 36fbb56dcdbbd7f6efde984010aa55ff0a639fc0dcbae836fe4e466181549f13 not found: ID does not exist" Oct 31 00:16:32.106679 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:32.106642 5222 scope.go:122] "RemoveContainer" containerID="e5feaf6f21a314406a7e43b9f37f4d13ceb9dc02ffa03cdeceec2b53d27424ad" Oct 31 00:16:32.106943 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:16:32.106922 5222 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5feaf6f21a314406a7e43b9f37f4d13ceb9dc02ffa03cdeceec2b53d27424ad\": container with ID starting with e5feaf6f21a314406a7e43b9f37f4d13ceb9dc02ffa03cdeceec2b53d27424ad not found: ID does not exist" containerID="e5feaf6f21a314406a7e43b9f37f4d13ceb9dc02ffa03cdeceec2b53d27424ad" Oct 31 00:16:32.106998 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:32.106962 5222 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5feaf6f21a314406a7e43b9f37f4d13ceb9dc02ffa03cdeceec2b53d27424ad"} err="failed to get container status \"e5feaf6f21a314406a7e43b9f37f4d13ceb9dc02ffa03cdeceec2b53d27424ad\": rpc error: code = NotFound desc = could not find container \"e5feaf6f21a314406a7e43b9f37f4d13ceb9dc02ffa03cdeceec2b53d27424ad\": container with ID starting with e5feaf6f21a314406a7e43b9f37f4d13ceb9dc02ffa03cdeceec2b53d27424ad not found: ID does not exist" Oct 31 00:16:32.106998 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:32.106977 5222 scope.go:122] "RemoveContainer" containerID="e7ff2a63615b3ad0d56475374f903d09361d6f69bd02735970a97950a843878e" Oct 31 00:16:32.107278 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:16:32.107262 5222 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7ff2a63615b3ad0d56475374f903d09361d6f69bd02735970a97950a843878e\": container with ID starting with e7ff2a63615b3ad0d56475374f903d09361d6f69bd02735970a97950a843878e not found: ID does not exist" containerID="e7ff2a63615b3ad0d56475374f903d09361d6f69bd02735970a97950a843878e" Oct 31 00:16:32.107343 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:32.107294 5222 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7ff2a63615b3ad0d56475374f903d09361d6f69bd02735970a97950a843878e"} err="failed to get container status \"e7ff2a63615b3ad0d56475374f903d09361d6f69bd02735970a97950a843878e\": rpc error: code = NotFound desc = could not find container \"e7ff2a63615b3ad0d56475374f903d09361d6f69bd02735970a97950a843878e\": container with ID starting with e7ff2a63615b3ad0d56475374f903d09361d6f69bd02735970a97950a843878e not found: ID does not exist" Oct 31 00:16:32.107343 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:32.107308 5222 scope.go:122] "RemoveContainer" containerID="2863d9cb4d6566e2e47954688ca8308131271a1c3f3c08a461bf1a65a9361bd8" Oct 31 00:16:32.107597 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:16:32.107580 5222 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2863d9cb4d6566e2e47954688ca8308131271a1c3f3c08a461bf1a65a9361bd8\": container with ID starting with 2863d9cb4d6566e2e47954688ca8308131271a1c3f3c08a461bf1a65a9361bd8 not found: ID does not exist" containerID="2863d9cb4d6566e2e47954688ca8308131271a1c3f3c08a461bf1a65a9361bd8" Oct 31 00:16:32.107640 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:32.107603 5222 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2863d9cb4d6566e2e47954688ca8308131271a1c3f3c08a461bf1a65a9361bd8"} err="failed to get container status \"2863d9cb4d6566e2e47954688ca8308131271a1c3f3c08a461bf1a65a9361bd8\": rpc error: code = NotFound desc = could not find container \"2863d9cb4d6566e2e47954688ca8308131271a1c3f3c08a461bf1a65a9361bd8\": container with ID starting with 2863d9cb4d6566e2e47954688ca8308131271a1c3f3c08a461bf1a65a9361bd8 not found: ID does not exist" Oct 31 00:16:32.107640 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:32.107616 5222 scope.go:122] "RemoveContainer" containerID="a5d1a9b4427b9e1b72476e20b21f459edc4d5fff321004f6a5000e973a958c49" Oct 31 00:16:32.107838 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:16:32.107822 5222 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5d1a9b4427b9e1b72476e20b21f459edc4d5fff321004f6a5000e973a958c49\": container with ID starting with a5d1a9b4427b9e1b72476e20b21f459edc4d5fff321004f6a5000e973a958c49 not found: ID does not exist" containerID="a5d1a9b4427b9e1b72476e20b21f459edc4d5fff321004f6a5000e973a958c49" Oct 31 00:16:32.107877 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:32.107841 5222 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5d1a9b4427b9e1b72476e20b21f459edc4d5fff321004f6a5000e973a958c49"} err="failed to get container status \"a5d1a9b4427b9e1b72476e20b21f459edc4d5fff321004f6a5000e973a958c49\": rpc error: code = NotFound desc = could not find container \"a5d1a9b4427b9e1b72476e20b21f459edc4d5fff321004f6a5000e973a958c49\": container with ID starting with a5d1a9b4427b9e1b72476e20b21f459edc4d5fff321004f6a5000e973a958c49 not found: ID does not exist" Oct 31 00:16:32.107877 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:32.107853 5222 scope.go:122] "RemoveContainer" containerID="18c9379d2fb95e32c95525a0b00a4e8decc49e9c5c0cc5b40f836cf911d2924a" Oct 31 00:16:32.108080 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:16:32.108063 5222 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18c9379d2fb95e32c95525a0b00a4e8decc49e9c5c0cc5b40f836cf911d2924a\": container with ID starting with 18c9379d2fb95e32c95525a0b00a4e8decc49e9c5c0cc5b40f836cf911d2924a not found: ID does not exist" containerID="18c9379d2fb95e32c95525a0b00a4e8decc49e9c5c0cc5b40f836cf911d2924a" Oct 31 00:16:32.108119 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:32.108085 5222 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18c9379d2fb95e32c95525a0b00a4e8decc49e9c5c0cc5b40f836cf911d2924a"} err="failed to get container status \"18c9379d2fb95e32c95525a0b00a4e8decc49e9c5c0cc5b40f836cf911d2924a\": rpc error: code = NotFound desc = could not find container \"18c9379d2fb95e32c95525a0b00a4e8decc49e9c5c0cc5b40f836cf911d2924a\": container with ID starting with 18c9379d2fb95e32c95525a0b00a4e8decc49e9c5c0cc5b40f836cf911d2924a not found: ID does not exist" Oct 31 00:16:32.108119 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:32.108098 5222 scope.go:122] "RemoveContainer" containerID="bc1fedc7ac40dc097d38d3e6b2fed32c26a3c8fa7ba633e52174d0cc3819d1ca" Oct 31 00:16:32.108295 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:16:32.108281 5222 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc1fedc7ac40dc097d38d3e6b2fed32c26a3c8fa7ba633e52174d0cc3819d1ca\": container with ID starting with bc1fedc7ac40dc097d38d3e6b2fed32c26a3c8fa7ba633e52174d0cc3819d1ca not found: ID does not exist" containerID="bc1fedc7ac40dc097d38d3e6b2fed32c26a3c8fa7ba633e52174d0cc3819d1ca" Oct 31 00:16:32.108332 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:32.108298 5222 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc1fedc7ac40dc097d38d3e6b2fed32c26a3c8fa7ba633e52174d0cc3819d1ca"} err="failed to get container status \"bc1fedc7ac40dc097d38d3e6b2fed32c26a3c8fa7ba633e52174d0cc3819d1ca\": rpc error: code = NotFound desc = could not find container \"bc1fedc7ac40dc097d38d3e6b2fed32c26a3c8fa7ba633e52174d0cc3819d1ca\": container with ID starting with bc1fedc7ac40dc097d38d3e6b2fed32c26a3c8fa7ba633e52174d0cc3819d1ca not found: ID does not exist" Oct 31 00:16:32.275760 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:32.275722 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 00:16:32.327211 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:32.327134 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 00:16:32.327619 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:32.327602 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 00:16:32.327842 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:32.327831 5222 cpu_manager.go:414] "containerMap: removing container" podUID="26aba8fc-e2da-430f-b0d8-6ee45468441d" containerName="prom-label-proxy" Oct 31 00:16:32.327879 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:32.327844 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26aba8fc-e2da-430f-b0d8-6ee45468441d" containerName="prom-label-proxy" Oct 31 00:16:32.327879 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:32.327858 5222 cpu_manager.go:414] "containerMap: removing container" podUID="26aba8fc-e2da-430f-b0d8-6ee45468441d" containerName="config-reloader" Oct 31 00:16:32.327879 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:32.327863 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26aba8fc-e2da-430f-b0d8-6ee45468441d" containerName="config-reloader" Oct 31 00:16:32.327879 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:32.327875 5222 cpu_manager.go:414] "containerMap: removing container" podUID="26aba8fc-e2da-430f-b0d8-6ee45468441d" containerName="kube-rbac-proxy-web" Oct 31 00:16:32.327879 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:32.327879 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26aba8fc-e2da-430f-b0d8-6ee45468441d" containerName="kube-rbac-proxy-web" Oct 31 00:16:32.328021 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:32.327889 5222 cpu_manager.go:414] "containerMap: removing container" podUID="26aba8fc-e2da-430f-b0d8-6ee45468441d" containerName="init-config-reloader" Oct 31 00:16:32.328021 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:32.327894 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26aba8fc-e2da-430f-b0d8-6ee45468441d" containerName="init-config-reloader" Oct 31 00:16:32.328021 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:32.327908 5222 cpu_manager.go:414] "containerMap: removing container" podUID="26aba8fc-e2da-430f-b0d8-6ee45468441d" containerName="kube-rbac-proxy" Oct 31 00:16:32.328021 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:32.327912 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26aba8fc-e2da-430f-b0d8-6ee45468441d" containerName="kube-rbac-proxy" Oct 31 00:16:32.328021 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:32.327919 5222 cpu_manager.go:414] "containerMap: removing container" podUID="26aba8fc-e2da-430f-b0d8-6ee45468441d" containerName="kube-rbac-proxy-metric" Oct 31 00:16:32.328021 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:32.327923 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26aba8fc-e2da-430f-b0d8-6ee45468441d" containerName="kube-rbac-proxy-metric" Oct 31 00:16:32.328021 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:32.327931 5222 cpu_manager.go:414] "containerMap: removing container" podUID="26aba8fc-e2da-430f-b0d8-6ee45468441d" containerName="alertmanager" Oct 31 00:16:32.328021 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:32.327935 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26aba8fc-e2da-430f-b0d8-6ee45468441d" containerName="alertmanager" Oct 31 00:16:32.328021 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:32.327974 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="26aba8fc-e2da-430f-b0d8-6ee45468441d" containerName="alertmanager" Oct 31 00:16:32.328021 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:32.327980 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="26aba8fc-e2da-430f-b0d8-6ee45468441d" containerName="config-reloader" Oct 31 00:16:32.328021 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:32.327986 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="26aba8fc-e2da-430f-b0d8-6ee45468441d" containerName="kube-rbac-proxy-web" Oct 31 00:16:32.328021 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:32.327990 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="26aba8fc-e2da-430f-b0d8-6ee45468441d" containerName="kube-rbac-proxy" Oct 31 00:16:32.328021 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:32.327995 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="26aba8fc-e2da-430f-b0d8-6ee45468441d" containerName="kube-rbac-proxy-metric" Oct 31 00:16:32.328021 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:32.328002 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="26aba8fc-e2da-430f-b0d8-6ee45468441d" containerName="prom-label-proxy" Oct 31 00:16:32.331432 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:32.331418 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:16:32.428136 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:32.428103 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/65985e4d-b22b-4392-a735-6353b05d9f77-tls-assets\") pod \"alertmanager-main-0\" (UID: \"65985e4d-b22b-4392-a735-6353b05d9f77\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:16:32.428136 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:32.428141 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/65985e4d-b22b-4392-a735-6353b05d9f77-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"65985e4d-b22b-4392-a735-6353b05d9f77\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:16:32.428359 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:32.428181 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/65985e4d-b22b-4392-a735-6353b05d9f77-config-volume\") pod \"alertmanager-main-0\" (UID: \"65985e4d-b22b-4392-a735-6353b05d9f77\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:16:32.428359 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:32.428240 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/65985e4d-b22b-4392-a735-6353b05d9f77-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"65985e4d-b22b-4392-a735-6353b05d9f77\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:16:32.428359 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:32.428316 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/65985e4d-b22b-4392-a735-6353b05d9f77-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"65985e4d-b22b-4392-a735-6353b05d9f77\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:16:32.428359 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:32.428347 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65985e4d-b22b-4392-a735-6353b05d9f77-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"65985e4d-b22b-4392-a735-6353b05d9f77\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:16:32.428517 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:32.428391 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/65985e4d-b22b-4392-a735-6353b05d9f77-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"65985e4d-b22b-4392-a735-6353b05d9f77\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:16:32.428517 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:32.428444 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/65985e4d-b22b-4392-a735-6353b05d9f77-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"65985e4d-b22b-4392-a735-6353b05d9f77\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:16:32.428517 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:32.428469 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/65985e4d-b22b-4392-a735-6353b05d9f77-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"65985e4d-b22b-4392-a735-6353b05d9f77\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:16:32.428616 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:32.428518 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/65985e4d-b22b-4392-a735-6353b05d9f77-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"65985e4d-b22b-4392-a735-6353b05d9f77\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:16:32.428616 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:32.428543 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/65985e4d-b22b-4392-a735-6353b05d9f77-config-out\") pod \"alertmanager-main-0\" (UID: \"65985e4d-b22b-4392-a735-6353b05d9f77\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:16:32.428616 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:32.428568 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swdk4\" (UniqueName: \"kubernetes.io/projected/65985e4d-b22b-4392-a735-6353b05d9f77-kube-api-access-swdk4\") pod \"alertmanager-main-0\" (UID: \"65985e4d-b22b-4392-a735-6353b05d9f77\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:16:32.428616 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:32.428602 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/65985e4d-b22b-4392-a735-6353b05d9f77-web-config\") pod \"alertmanager-main-0\" (UID: \"65985e4d-b22b-4392-a735-6353b05d9f77\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:16:32.436617 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:32.436546 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 31 00:16:32.436617 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:32.436566 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 31 00:16:32.436617 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:32.436566 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 31 00:16:32.436617 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:32.436566 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 31 00:16:32.436617 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:32.436555 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 31 00:16:32.436617 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:32.436566 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 31 00:16:32.436930 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:32.436594 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 31 00:16:32.436930 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:32.436597 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 31 00:16:32.436930 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:32.436602 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 31 00:16:32.436930 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:32.436662 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-ltd79\"" Oct 31 00:16:32.529574 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:32.529536 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/65985e4d-b22b-4392-a735-6353b05d9f77-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"65985e4d-b22b-4392-a735-6353b05d9f77\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:16:32.529760 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:32.529581 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/65985e4d-b22b-4392-a735-6353b05d9f77-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"65985e4d-b22b-4392-a735-6353b05d9f77\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:16:32.529760 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:32.529617 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/65985e4d-b22b-4392-a735-6353b05d9f77-config-out\") pod \"alertmanager-main-0\" (UID: \"65985e4d-b22b-4392-a735-6353b05d9f77\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:16:32.529760 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:32.529644 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swdk4\" (UniqueName: \"kubernetes.io/projected/65985e4d-b22b-4392-a735-6353b05d9f77-kube-api-access-swdk4\") pod \"alertmanager-main-0\" (UID: \"65985e4d-b22b-4392-a735-6353b05d9f77\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:16:32.529760 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:32.529676 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/65985e4d-b22b-4392-a735-6353b05d9f77-web-config\") pod \"alertmanager-main-0\" (UID: \"65985e4d-b22b-4392-a735-6353b05d9f77\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:16:32.529760 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:32.529704 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/65985e4d-b22b-4392-a735-6353b05d9f77-tls-assets\") pod \"alertmanager-main-0\" (UID: \"65985e4d-b22b-4392-a735-6353b05d9f77\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:16:32.529760 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:32.529730 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/65985e4d-b22b-4392-a735-6353b05d9f77-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"65985e4d-b22b-4392-a735-6353b05d9f77\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:16:32.530036 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:32.529873 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/65985e4d-b22b-4392-a735-6353b05d9f77-config-volume\") pod \"alertmanager-main-0\" (UID: \"65985e4d-b22b-4392-a735-6353b05d9f77\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:16:32.530036 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:32.529909 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/65985e4d-b22b-4392-a735-6353b05d9f77-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"65985e4d-b22b-4392-a735-6353b05d9f77\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:16:32.530036 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:32.529938 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/65985e4d-b22b-4392-a735-6353b05d9f77-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"65985e4d-b22b-4392-a735-6353b05d9f77\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:16:32.530036 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:32.529962 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65985e4d-b22b-4392-a735-6353b05d9f77-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"65985e4d-b22b-4392-a735-6353b05d9f77\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:16:32.530036 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:32.529987 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/65985e4d-b22b-4392-a735-6353b05d9f77-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"65985e4d-b22b-4392-a735-6353b05d9f77\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:16:32.530338 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:32.530313 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/65985e4d-b22b-4392-a735-6353b05d9f77-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"65985e4d-b22b-4392-a735-6353b05d9f77\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:16:32.530425 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:32.530393 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/65985e4d-b22b-4392-a735-6353b05d9f77-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"65985e4d-b22b-4392-a735-6353b05d9f77\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:16:32.531631 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:32.531591 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/65985e4d-b22b-4392-a735-6353b05d9f77-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"65985e4d-b22b-4392-a735-6353b05d9f77\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:16:32.531747 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:32.531715 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65985e4d-b22b-4392-a735-6353b05d9f77-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"65985e4d-b22b-4392-a735-6353b05d9f77\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:16:32.533145 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:32.533116 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/65985e4d-b22b-4392-a735-6353b05d9f77-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"65985e4d-b22b-4392-a735-6353b05d9f77\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:16:32.533258 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:32.533234 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/65985e4d-b22b-4392-a735-6353b05d9f77-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"65985e4d-b22b-4392-a735-6353b05d9f77\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:16:32.533716 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:32.533562 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/65985e4d-b22b-4392-a735-6353b05d9f77-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"65985e4d-b22b-4392-a735-6353b05d9f77\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:16:32.533716 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:32.533602 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/65985e4d-b22b-4392-a735-6353b05d9f77-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"65985e4d-b22b-4392-a735-6353b05d9f77\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:16:32.533716 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:32.533649 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/65985e4d-b22b-4392-a735-6353b05d9f77-web-config\") pod \"alertmanager-main-0\" (UID: \"65985e4d-b22b-4392-a735-6353b05d9f77\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:16:32.533880 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:32.533773 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/65985e4d-b22b-4392-a735-6353b05d9f77-config-out\") pod \"alertmanager-main-0\" (UID: \"65985e4d-b22b-4392-a735-6353b05d9f77\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:16:32.533880 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:32.533852 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/65985e4d-b22b-4392-a735-6353b05d9f77-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"65985e4d-b22b-4392-a735-6353b05d9f77\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:16:32.534046 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:32.534024 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/65985e4d-b22b-4392-a735-6353b05d9f77-config-volume\") pod \"alertmanager-main-0\" (UID: \"65985e4d-b22b-4392-a735-6353b05d9f77\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:16:32.534822 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:32.534807 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/65985e4d-b22b-4392-a735-6353b05d9f77-tls-assets\") pod \"alertmanager-main-0\" (UID: \"65985e4d-b22b-4392-a735-6353b05d9f77\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:16:32.546837 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:32.546813 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 00:16:32.592473 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:32.592398 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swdk4\" (UniqueName: \"kubernetes.io/projected/65985e4d-b22b-4392-a735-6353b05d9f77-kube-api-access-swdk4\") pod \"alertmanager-main-0\" (UID: \"65985e4d-b22b-4392-a735-6353b05d9f77\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:16:32.648605 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:32.648564 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:16:33.073154 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:16:33.073111 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65985e4d_b22b_4392_a735_6353b05d9f77.slice/crio-7ce731bcd13e0ce290e6ec06289750848fbdea05fd8875d61b0141d014bd909d WatchSource:0}: Error finding container 7ce731bcd13e0ce290e6ec06289750848fbdea05fd8875d61b0141d014bd909d: Status 404 returned error can't find the container with id 7ce731bcd13e0ce290e6ec06289750848fbdea05fd8875d61b0141d014bd909d Oct 31 00:16:33.074194 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:33.074171 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 00:16:33.652481 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:33.652446 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26aba8fc-e2da-430f-b0d8-6ee45468441d" path="/var/lib/kubelet/pods/26aba8fc-e2da-430f-b0d8-6ee45468441d/volumes" Oct 31 00:16:33.932139 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:33.932053 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-5bc6f9fd94-d2fk4"] Oct 31 00:16:33.934977 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:33.934955 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5bc6f9fd94-d2fk4" Oct 31 00:16:33.990248 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:33.990165 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Oct 31 00:16:33.990248 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:33.990182 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Oct 31 00:16:33.990248 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:33.990190 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-hvkf5\"" Oct 31 00:16:33.990248 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:33.990206 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Oct 31 00:16:33.990248 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:33.990171 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Oct 31 00:16:33.990248 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:33.990249 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Oct 31 00:16:34.041462 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:34.041426 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f7a951f0-22de-4adc-94d5-5bd1962e545d-metrics-client-ca\") pod \"telemeter-client-5bc6f9fd94-d2fk4\" (UID: \"f7a951f0-22de-4adc-94d5-5bd1962e545d\") " pod="openshift-monitoring/telemeter-client-5bc6f9fd94-d2fk4" Oct 31 00:16:34.041462 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:34.041468 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f7a951f0-22de-4adc-94d5-5bd1962e545d-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5bc6f9fd94-d2fk4\" (UID: \"f7a951f0-22de-4adc-94d5-5bd1962e545d\") " pod="openshift-monitoring/telemeter-client-5bc6f9fd94-d2fk4" Oct 31 00:16:34.041716 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:34.041667 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7a951f0-22de-4adc-94d5-5bd1962e545d-serving-certs-ca-bundle\") pod \"telemeter-client-5bc6f9fd94-d2fk4\" (UID: \"f7a951f0-22de-4adc-94d5-5bd1962e545d\") " pod="openshift-monitoring/telemeter-client-5bc6f9fd94-d2fk4" Oct 31 00:16:34.041716 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:34.041706 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/f7a951f0-22de-4adc-94d5-5bd1962e545d-secret-telemeter-client\") pod \"telemeter-client-5bc6f9fd94-d2fk4\" (UID: \"f7a951f0-22de-4adc-94d5-5bd1962e545d\") " pod="openshift-monitoring/telemeter-client-5bc6f9fd94-d2fk4" Oct 31 00:16:34.041868 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:34.041848 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxrn7\" (UniqueName: \"kubernetes.io/projected/f7a951f0-22de-4adc-94d5-5bd1962e545d-kube-api-access-gxrn7\") pod \"telemeter-client-5bc6f9fd94-d2fk4\" (UID: \"f7a951f0-22de-4adc-94d5-5bd1962e545d\") " pod="openshift-monitoring/telemeter-client-5bc6f9fd94-d2fk4" Oct 31 00:16:34.041936 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:34.041902 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/f7a951f0-22de-4adc-94d5-5bd1962e545d-federate-client-tls\") pod \"telemeter-client-5bc6f9fd94-d2fk4\" (UID: \"f7a951f0-22de-4adc-94d5-5bd1962e545d\") " pod="openshift-monitoring/telemeter-client-5bc6f9fd94-d2fk4" Oct 31 00:16:34.041996 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:34.041942 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/f7a951f0-22de-4adc-94d5-5bd1962e545d-telemeter-client-tls\") pod \"telemeter-client-5bc6f9fd94-d2fk4\" (UID: \"f7a951f0-22de-4adc-94d5-5bd1962e545d\") " pod="openshift-monitoring/telemeter-client-5bc6f9fd94-d2fk4" Oct 31 00:16:34.041996 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:34.041960 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7a951f0-22de-4adc-94d5-5bd1962e545d-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5bc6f9fd94-d2fk4\" (UID: \"f7a951f0-22de-4adc-94d5-5bd1962e545d\") " pod="openshift-monitoring/telemeter-client-5bc6f9fd94-d2fk4" Oct 31 00:16:34.062286 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:34.062250 5222 generic.go:356] "Generic (PLEG): container finished" podID="65985e4d-b22b-4392-a735-6353b05d9f77" containerID="968b701add8e9a3131357aafeda64f7440d831655b03905b118a66b5a68db64e" exitCode=0 Oct 31 00:16:34.062495 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:34.062303 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"65985e4d-b22b-4392-a735-6353b05d9f77","Type":"ContainerDied","Data":"968b701add8e9a3131357aafeda64f7440d831655b03905b118a66b5a68db64e"} Oct 31 00:16:34.062495 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:34.062355 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"65985e4d-b22b-4392-a735-6353b05d9f77","Type":"ContainerStarted","Data":"7ce731bcd13e0ce290e6ec06289750848fbdea05fd8875d61b0141d014bd909d"} Oct 31 00:16:34.096613 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:34.092909 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Oct 31 00:16:34.142457 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:34.142431 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/f7a951f0-22de-4adc-94d5-5bd1962e545d-telemeter-client-tls\") pod \"telemeter-client-5bc6f9fd94-d2fk4\" (UID: \"f7a951f0-22de-4adc-94d5-5bd1962e545d\") " pod="openshift-monitoring/telemeter-client-5bc6f9fd94-d2fk4" Oct 31 00:16:34.142570 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:34.142470 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7a951f0-22de-4adc-94d5-5bd1962e545d-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5bc6f9fd94-d2fk4\" (UID: \"f7a951f0-22de-4adc-94d5-5bd1962e545d\") " pod="openshift-monitoring/telemeter-client-5bc6f9fd94-d2fk4" Oct 31 00:16:34.142570 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:34.142531 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f7a951f0-22de-4adc-94d5-5bd1962e545d-metrics-client-ca\") pod \"telemeter-client-5bc6f9fd94-d2fk4\" (UID: \"f7a951f0-22de-4adc-94d5-5bd1962e545d\") " pod="openshift-monitoring/telemeter-client-5bc6f9fd94-d2fk4" Oct 31 00:16:34.142570 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:34.142563 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f7a951f0-22de-4adc-94d5-5bd1962e545d-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5bc6f9fd94-d2fk4\" (UID: \"f7a951f0-22de-4adc-94d5-5bd1962e545d\") " pod="openshift-monitoring/telemeter-client-5bc6f9fd94-d2fk4" Oct 31 00:16:34.142737 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:34.142699 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7a951f0-22de-4adc-94d5-5bd1962e545d-serving-certs-ca-bundle\") pod \"telemeter-client-5bc6f9fd94-d2fk4\" (UID: \"f7a951f0-22de-4adc-94d5-5bd1962e545d\") " pod="openshift-monitoring/telemeter-client-5bc6f9fd94-d2fk4" Oct 31 00:16:34.142822 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:34.142804 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/f7a951f0-22de-4adc-94d5-5bd1962e545d-secret-telemeter-client\") pod \"telemeter-client-5bc6f9fd94-d2fk4\" (UID: \"f7a951f0-22de-4adc-94d5-5bd1962e545d\") " pod="openshift-monitoring/telemeter-client-5bc6f9fd94-d2fk4" Oct 31 00:16:34.156692 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:34.144149 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7a951f0-22de-4adc-94d5-5bd1962e545d-serving-certs-ca-bundle\") pod \"telemeter-client-5bc6f9fd94-d2fk4\" (UID: \"f7a951f0-22de-4adc-94d5-5bd1962e545d\") " pod="openshift-monitoring/telemeter-client-5bc6f9fd94-d2fk4" Oct 31 00:16:34.156692 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:34.144827 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f7a951f0-22de-4adc-94d5-5bd1962e545d-metrics-client-ca\") pod \"telemeter-client-5bc6f9fd94-d2fk4\" (UID: \"f7a951f0-22de-4adc-94d5-5bd1962e545d\") " pod="openshift-monitoring/telemeter-client-5bc6f9fd94-d2fk4" Oct 31 00:16:34.156692 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:34.145256 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7a951f0-22de-4adc-94d5-5bd1962e545d-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5bc6f9fd94-d2fk4\" (UID: \"f7a951f0-22de-4adc-94d5-5bd1962e545d\") " pod="openshift-monitoring/telemeter-client-5bc6f9fd94-d2fk4" Oct 31 00:16:34.156692 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:34.145361 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxrn7\" (UniqueName: \"kubernetes.io/projected/f7a951f0-22de-4adc-94d5-5bd1962e545d-kube-api-access-gxrn7\") pod \"telemeter-client-5bc6f9fd94-d2fk4\" (UID: \"f7a951f0-22de-4adc-94d5-5bd1962e545d\") " pod="openshift-monitoring/telemeter-client-5bc6f9fd94-d2fk4" Oct 31 00:16:34.156692 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:34.145446 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/f7a951f0-22de-4adc-94d5-5bd1962e545d-federate-client-tls\") pod \"telemeter-client-5bc6f9fd94-d2fk4\" (UID: \"f7a951f0-22de-4adc-94d5-5bd1962e545d\") " pod="openshift-monitoring/telemeter-client-5bc6f9fd94-d2fk4" Oct 31 00:16:34.156692 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:34.148862 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/f7a951f0-22de-4adc-94d5-5bd1962e545d-telemeter-client-tls\") pod \"telemeter-client-5bc6f9fd94-d2fk4\" (UID: \"f7a951f0-22de-4adc-94d5-5bd1962e545d\") " pod="openshift-monitoring/telemeter-client-5bc6f9fd94-d2fk4" Oct 31 00:16:34.156692 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:34.155470 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5bc6f9fd94-d2fk4"] Oct 31 00:16:34.163145 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:34.163050 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/f7a951f0-22de-4adc-94d5-5bd1962e545d-secret-telemeter-client\") pod \"telemeter-client-5bc6f9fd94-d2fk4\" (UID: \"f7a951f0-22de-4adc-94d5-5bd1962e545d\") " pod="openshift-monitoring/telemeter-client-5bc6f9fd94-d2fk4" Oct 31 00:16:34.164666 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:34.164637 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f7a951f0-22de-4adc-94d5-5bd1962e545d-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5bc6f9fd94-d2fk4\" (UID: \"f7a951f0-22de-4adc-94d5-5bd1962e545d\") " pod="openshift-monitoring/telemeter-client-5bc6f9fd94-d2fk4" Oct 31 00:16:34.165284 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:34.165254 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/f7a951f0-22de-4adc-94d5-5bd1962e545d-federate-client-tls\") pod \"telemeter-client-5bc6f9fd94-d2fk4\" (UID: \"f7a951f0-22de-4adc-94d5-5bd1962e545d\") " pod="openshift-monitoring/telemeter-client-5bc6f9fd94-d2fk4" Oct 31 00:16:34.208995 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:34.208969 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxrn7\" (UniqueName: \"kubernetes.io/projected/f7a951f0-22de-4adc-94d5-5bd1962e545d-kube-api-access-gxrn7\") pod \"telemeter-client-5bc6f9fd94-d2fk4\" (UID: \"f7a951f0-22de-4adc-94d5-5bd1962e545d\") " pod="openshift-monitoring/telemeter-client-5bc6f9fd94-d2fk4" Oct 31 00:16:34.247261 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:34.247240 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5bc6f9fd94-d2fk4" Oct 31 00:16:34.673411 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:16:34.673382 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7a951f0_22de_4adc_94d5_5bd1962e545d.slice/crio-a1d7ea168919e7bd64a79059d29473e895edf44e1c518cb4f4694e43d75f66dc WatchSource:0}: Error finding container a1d7ea168919e7bd64a79059d29473e895edf44e1c518cb4f4694e43d75f66dc: Status 404 returned error can't find the container with id a1d7ea168919e7bd64a79059d29473e895edf44e1c518cb4f4694e43d75f66dc Oct 31 00:16:34.675228 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:34.675208 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5bc6f9fd94-d2fk4"] Oct 31 00:16:35.068694 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:35.068658 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"65985e4d-b22b-4392-a735-6353b05d9f77","Type":"ContainerStarted","Data":"062858b1ab6951bf6e341afebdfb3f3d391160a0547fdee0da611d70aa37a16f"} Oct 31 00:16:35.068852 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:35.068725 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"65985e4d-b22b-4392-a735-6353b05d9f77","Type":"ContainerStarted","Data":"59d3fd05d51ac8c9e085c71b75d5c93dc7ff9323be81154a80fa96322d9fd62d"} Oct 31 00:16:35.068852 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:35.068753 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"65985e4d-b22b-4392-a735-6353b05d9f77","Type":"ContainerStarted","Data":"bc581fec5f7405aab81c3025df19afec34e6178c3af0c62c5581bad18d94f27e"} Oct 31 00:16:35.068852 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:35.068780 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"65985e4d-b22b-4392-a735-6353b05d9f77","Type":"ContainerStarted","Data":"2b2f42cd1f4e26517e8f1fded9cf37be397c811fbd711bce916e6a27b70c758a"} Oct 31 00:16:35.068852 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:35.068815 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"65985e4d-b22b-4392-a735-6353b05d9f77","Type":"ContainerStarted","Data":"1f8cebaa607e43fd5eb4a624a05589973fc5e556cdde1737f8fe6f0784125663"} Oct 31 00:16:35.068852 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:35.068841 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"65985e4d-b22b-4392-a735-6353b05d9f77","Type":"ContainerStarted","Data":"99ed8ab1455fb7d159278d1d279b975ceeda2753c71532c8c44b5d9927eb53f0"} Oct 31 00:16:35.070020 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:35.069999 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5bc6f9fd94-d2fk4" event={"ID":"f7a951f0-22de-4adc-94d5-5bd1962e545d","Type":"ContainerStarted","Data":"a1d7ea168919e7bd64a79059d29473e895edf44e1c518cb4f4694e43d75f66dc"} Oct 31 00:16:35.201973 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:35.201925 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.189933448 podStartE2EDuration="3.201912339s" totalImagesPullingTime="0s" totalInitContainerRuntime="11.978891ms" isStatefulPod=false podCreationTimestamp="2026-10-31 00:16:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:16:35.200864458 +0000 UTC m=+87.672034207" watchObservedRunningTime="2026-10-31 00:16:35.201912339 +0000 UTC m=+87.673082087" Oct 31 00:16:35.890189 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:16:35.890150 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 00:16:38.084956 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:38.084917 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5bc6f9fd94-d2fk4" event={"ID":"f7a951f0-22de-4adc-94d5-5bd1962e545d","Type":"ContainerStarted","Data":"257b8a92f1fe3cbc92ebed97d3219968493de919a3f7a57dfbe953e7c7201a99"} Oct 31 00:16:38.085391 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:38.084987 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5bc6f9fd94-d2fk4" event={"ID":"f7a951f0-22de-4adc-94d5-5bd1962e545d","Type":"ContainerStarted","Data":"81c2de26e8ca83bd8009cda66a309317cf99623ccf2daf577c87765763e39318"} Oct 31 00:16:38.085391 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:38.085017 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5bc6f9fd94-d2fk4" event={"ID":"f7a951f0-22de-4adc-94d5-5bd1962e545d","Type":"ContainerStarted","Data":"dd8290aec9772bb10cda9cb07832c78a30f27ffd80a592a58b688597b7dba636"} Oct 31 00:16:38.312738 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:38.312674 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-5bc6f9fd94-d2fk4" podStartSLOduration=2.71210798 podStartE2EDuration="5.312659932s" totalImagesPullingTime="2.600551952s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:16:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:16:38.260793676 +0000 UTC m=+90.731963424" watchObservedRunningTime="2026-10-31 00:16:38.312659932 +0000 UTC m=+90.783829682" Oct 31 00:16:38.717391 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:38.717336 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-9c784f57c-8f2zj"] Oct 31 00:16:38.719498 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:38.719481 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9c784f57c-8f2zj" Oct 31 00:16:38.776902 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:38.776871 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-9c784f57c-8f2zj"] Oct 31 00:16:38.890741 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:38.890700 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/de7fa613-44c4-4e72-974d-9890609d4582-console-config\") pod \"console-9c784f57c-8f2zj\" (UID: \"de7fa613-44c4-4e72-974d-9890609d4582\") " pod="openshift-console/console-9c784f57c-8f2zj" Oct 31 00:16:38.890741 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:38.890742 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/de7fa613-44c4-4e72-974d-9890609d4582-console-oauth-config\") pod \"console-9c784f57c-8f2zj\" (UID: \"de7fa613-44c4-4e72-974d-9890609d4582\") " pod="openshift-console/console-9c784f57c-8f2zj" Oct 31 00:16:38.890971 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:38.890761 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de7fa613-44c4-4e72-974d-9890609d4582-trusted-ca-bundle\") pod \"console-9c784f57c-8f2zj\" (UID: \"de7fa613-44c4-4e72-974d-9890609d4582\") " pod="openshift-console/console-9c784f57c-8f2zj" Oct 31 00:16:38.890971 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:38.890927 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/de7fa613-44c4-4e72-974d-9890609d4582-tmp\") pod \"console-9c784f57c-8f2zj\" (UID: \"de7fa613-44c4-4e72-974d-9890609d4582\") " pod="openshift-console/console-9c784f57c-8f2zj" Oct 31 00:16:38.891066 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:38.890971 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/de7fa613-44c4-4e72-974d-9890609d4582-service-ca\") pod \"console-9c784f57c-8f2zj\" (UID: \"de7fa613-44c4-4e72-974d-9890609d4582\") " pod="openshift-console/console-9c784f57c-8f2zj" Oct 31 00:16:38.891066 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:38.890993 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/de7fa613-44c4-4e72-974d-9890609d4582-oauth-serving-cert\") pod \"console-9c784f57c-8f2zj\" (UID: \"de7fa613-44c4-4e72-974d-9890609d4582\") " pod="openshift-console/console-9c784f57c-8f2zj" Oct 31 00:16:38.891150 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:38.891074 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/de7fa613-44c4-4e72-974d-9890609d4582-console-serving-cert\") pod \"console-9c784f57c-8f2zj\" (UID: \"de7fa613-44c4-4e72-974d-9890609d4582\") " pod="openshift-console/console-9c784f57c-8f2zj" Oct 31 00:16:38.891150 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:38.891097 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbgxt\" (UniqueName: \"kubernetes.io/projected/de7fa613-44c4-4e72-974d-9890609d4582-kube-api-access-kbgxt\") pod \"console-9c784f57c-8f2zj\" (UID: \"de7fa613-44c4-4e72-974d-9890609d4582\") " pod="openshift-console/console-9c784f57c-8f2zj" Oct 31 00:16:38.991574 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:38.991486 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/de7fa613-44c4-4e72-974d-9890609d4582-service-ca\") pod \"console-9c784f57c-8f2zj\" (UID: \"de7fa613-44c4-4e72-974d-9890609d4582\") " pod="openshift-console/console-9c784f57c-8f2zj" Oct 31 00:16:38.991574 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:38.991527 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/de7fa613-44c4-4e72-974d-9890609d4582-oauth-serving-cert\") pod \"console-9c784f57c-8f2zj\" (UID: \"de7fa613-44c4-4e72-974d-9890609d4582\") " pod="openshift-console/console-9c784f57c-8f2zj" Oct 31 00:16:38.991574 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:38.991548 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/de7fa613-44c4-4e72-974d-9890609d4582-console-serving-cert\") pod \"console-9c784f57c-8f2zj\" (UID: \"de7fa613-44c4-4e72-974d-9890609d4582\") " pod="openshift-console/console-9c784f57c-8f2zj" Oct 31 00:16:38.991856 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:38.991752 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbgxt\" (UniqueName: \"kubernetes.io/projected/de7fa613-44c4-4e72-974d-9890609d4582-kube-api-access-kbgxt\") pod \"console-9c784f57c-8f2zj\" (UID: \"de7fa613-44c4-4e72-974d-9890609d4582\") " pod="openshift-console/console-9c784f57c-8f2zj" Oct 31 00:16:38.991856 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:38.991814 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/de7fa613-44c4-4e72-974d-9890609d4582-console-config\") pod \"console-9c784f57c-8f2zj\" (UID: \"de7fa613-44c4-4e72-974d-9890609d4582\") " pod="openshift-console/console-9c784f57c-8f2zj" Oct 31 00:16:38.991856 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:38.991850 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/de7fa613-44c4-4e72-974d-9890609d4582-console-oauth-config\") pod \"console-9c784f57c-8f2zj\" (UID: \"de7fa613-44c4-4e72-974d-9890609d4582\") " pod="openshift-console/console-9c784f57c-8f2zj" Oct 31 00:16:38.992024 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:38.991867 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de7fa613-44c4-4e72-974d-9890609d4582-trusted-ca-bundle\") pod \"console-9c784f57c-8f2zj\" (UID: \"de7fa613-44c4-4e72-974d-9890609d4582\") " pod="openshift-console/console-9c784f57c-8f2zj" Oct 31 00:16:38.992085 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:38.992063 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/de7fa613-44c4-4e72-974d-9890609d4582-tmp\") pod \"console-9c784f57c-8f2zj\" (UID: \"de7fa613-44c4-4e72-974d-9890609d4582\") " pod="openshift-console/console-9c784f57c-8f2zj" Oct 31 00:16:38.992471 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:38.992441 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/de7fa613-44c4-4e72-974d-9890609d4582-service-ca\") pod \"console-9c784f57c-8f2zj\" (UID: \"de7fa613-44c4-4e72-974d-9890609d4582\") " pod="openshift-console/console-9c784f57c-8f2zj" Oct 31 00:16:38.992613 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:38.992472 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/de7fa613-44c4-4e72-974d-9890609d4582-oauth-serving-cert\") pod \"console-9c784f57c-8f2zj\" (UID: \"de7fa613-44c4-4e72-974d-9890609d4582\") " pod="openshift-console/console-9c784f57c-8f2zj" Oct 31 00:16:38.992613 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:38.992479 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/de7fa613-44c4-4e72-974d-9890609d4582-tmp\") pod \"console-9c784f57c-8f2zj\" (UID: \"de7fa613-44c4-4e72-974d-9890609d4582\") " pod="openshift-console/console-9c784f57c-8f2zj" Oct 31 00:16:38.992613 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:38.992502 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/de7fa613-44c4-4e72-974d-9890609d4582-console-config\") pod \"console-9c784f57c-8f2zj\" (UID: \"de7fa613-44c4-4e72-974d-9890609d4582\") " pod="openshift-console/console-9c784f57c-8f2zj" Oct 31 00:16:38.992996 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:38.992975 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de7fa613-44c4-4e72-974d-9890609d4582-trusted-ca-bundle\") pod \"console-9c784f57c-8f2zj\" (UID: \"de7fa613-44c4-4e72-974d-9890609d4582\") " pod="openshift-console/console-9c784f57c-8f2zj" Oct 31 00:16:38.994227 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:38.994208 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/de7fa613-44c4-4e72-974d-9890609d4582-console-serving-cert\") pod \"console-9c784f57c-8f2zj\" (UID: \"de7fa613-44c4-4e72-974d-9890609d4582\") " pod="openshift-console/console-9c784f57c-8f2zj" Oct 31 00:16:38.994391 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:38.994357 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/de7fa613-44c4-4e72-974d-9890609d4582-console-oauth-config\") pod \"console-9c784f57c-8f2zj\" (UID: \"de7fa613-44c4-4e72-974d-9890609d4582\") " pod="openshift-console/console-9c784f57c-8f2zj" Oct 31 00:16:39.062975 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:39.062944 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbgxt\" (UniqueName: \"kubernetes.io/projected/de7fa613-44c4-4e72-974d-9890609d4582-kube-api-access-kbgxt\") pod \"console-9c784f57c-8f2zj\" (UID: \"de7fa613-44c4-4e72-974d-9890609d4582\") " pod="openshift-console/console-9c784f57c-8f2zj" Oct 31 00:16:39.329835 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:39.329790 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9c784f57c-8f2zj" Oct 31 00:16:39.705258 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:39.703107 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-9c784f57c-8f2zj"] Oct 31 00:16:39.728599 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:16:39.728561 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde7fa613_44c4_4e72_974d_9890609d4582.slice/crio-5a1fd723caaab229a70ed883d401f71920ea7555bf9ed21f1468917056ec7ab6 WatchSource:0}: Error finding container 5a1fd723caaab229a70ed883d401f71920ea7555bf9ed21f1468917056ec7ab6: Status 404 returned error can't find the container with id 5a1fd723caaab229a70ed883d401f71920ea7555bf9ed21f1468917056ec7ab6 Oct 31 00:16:40.094175 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:40.094141 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9c784f57c-8f2zj" event={"ID":"de7fa613-44c4-4e72-974d-9890609d4582","Type":"ContainerStarted","Data":"65f378b175f206df1bf24f8e1433c90ccf048d8cda4075f222f99170bed037f0"} Oct 31 00:16:40.094360 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:40.094188 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9c784f57c-8f2zj" event={"ID":"de7fa613-44c4-4e72-974d-9890609d4582","Type":"ContainerStarted","Data":"5a1fd723caaab229a70ed883d401f71920ea7555bf9ed21f1468917056ec7ab6"} Oct 31 00:16:40.219692 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:40.219642 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-9c784f57c-8f2zj" podStartSLOduration=2.219626687 podStartE2EDuration="2.219626687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:16:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:16:40.219558208 +0000 UTC m=+92.690727956" watchObservedRunningTime="2026-10-31 00:16:40.219626687 +0000 UTC m=+92.690796430" Oct 31 00:16:49.329961 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:49.329881 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-9c784f57c-8f2zj" Oct 31 00:16:49.329961 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:49.329940 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-9c784f57c-8f2zj" Oct 31 00:16:49.334753 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:49.334731 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-9c784f57c-8f2zj" Oct 31 00:16:50.138331 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:50.138302 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-9c784f57c-8f2zj" Oct 31 00:16:50.311603 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:16:50.311566 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-59c7d45ccf-r8m6l"] Oct 31 00:17:05.889641 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:17:05.889598 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 00:17:15.331328 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:17:15.331260 5222 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-59c7d45ccf-r8m6l" podUID="8c8bf8e8-c2a1-4c3e-ae23-9446233d0924" containerName="console" containerID="cri-o://41ba21539915ea1331b33805c4c4e714870b479dfb8c0e57d8d152ece64d4850" gracePeriod=15 Oct 31 00:17:15.582354 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:17:15.582290 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59c7d45ccf-r8m6l" Oct 31 00:17:15.686606 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:17:15.686575 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c8bf8e8-c2a1-4c3e-ae23-9446233d0924-tmp\" (UniqueName: \"kubernetes.io/empty-dir/8c8bf8e8-c2a1-4c3e-ae23-9446233d0924-tmp\") pod \"8c8bf8e8-c2a1-4c3e-ae23-9446233d0924\" (UID: \"8c8bf8e8-c2a1-4c3e-ae23-9446233d0924\") " Oct 31 00:17:15.686606 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:17:15.686621 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c8bf8e8-c2a1-4c3e-ae23-9446233d0924-kube-api-access-grw4v\" (UniqueName: \"kubernetes.io/projected/8c8bf8e8-c2a1-4c3e-ae23-9446233d0924-kube-api-access-grw4v\") pod \"8c8bf8e8-c2a1-4c3e-ae23-9446233d0924\" (UID: \"8c8bf8e8-c2a1-4c3e-ae23-9446233d0924\") " Oct 31 00:17:15.686849 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:17:15.686674 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c8bf8e8-c2a1-4c3e-ae23-9446233d0924-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8c8bf8e8-c2a1-4c3e-ae23-9446233d0924-console-serving-cert\") pod \"8c8bf8e8-c2a1-4c3e-ae23-9446233d0924\" (UID: \"8c8bf8e8-c2a1-4c3e-ae23-9446233d0924\") " Oct 31 00:17:15.686849 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:17:15.686707 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c8bf8e8-c2a1-4c3e-ae23-9446233d0924-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8c8bf8e8-c2a1-4c3e-ae23-9446233d0924-console-oauth-config\") pod \"8c8bf8e8-c2a1-4c3e-ae23-9446233d0924\" (UID: \"8c8bf8e8-c2a1-4c3e-ae23-9446233d0924\") " Oct 31 00:17:15.686849 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:17:15.686734 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c8bf8e8-c2a1-4c3e-ae23-9446233d0924-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c8bf8e8-c2a1-4c3e-ae23-9446233d0924-trusted-ca-bundle\") pod \"8c8bf8e8-c2a1-4c3e-ae23-9446233d0924\" (UID: \"8c8bf8e8-c2a1-4c3e-ae23-9446233d0924\") " Oct 31 00:17:15.687062 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:17:15.686878 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c8bf8e8-c2a1-4c3e-ae23-9446233d0924-console-config\" (UniqueName: \"kubernetes.io/configmap/8c8bf8e8-c2a1-4c3e-ae23-9446233d0924-console-config\") pod \"8c8bf8e8-c2a1-4c3e-ae23-9446233d0924\" (UID: \"8c8bf8e8-c2a1-4c3e-ae23-9446233d0924\") " Oct 31 00:17:15.687062 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:17:15.686907 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c8bf8e8-c2a1-4c3e-ae23-9446233d0924-tmp" pod "8c8bf8e8-c2a1-4c3e-ae23-9446233d0924" (UID: "8c8bf8e8-c2a1-4c3e-ae23-9446233d0924"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:17:15.687062 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:17:15.686939 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c8bf8e8-c2a1-4c3e-ae23-9446233d0924-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c8bf8e8-c2a1-4c3e-ae23-9446233d0924-service-ca\") pod \"8c8bf8e8-c2a1-4c3e-ae23-9446233d0924\" (UID: \"8c8bf8e8-c2a1-4c3e-ae23-9446233d0924\") " Oct 31 00:17:15.687062 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:17:15.687001 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c8bf8e8-c2a1-4c3e-ae23-9446233d0924-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8c8bf8e8-c2a1-4c3e-ae23-9446233d0924-oauth-serving-cert\") pod \"8c8bf8e8-c2a1-4c3e-ae23-9446233d0924\" (UID: \"8c8bf8e8-c2a1-4c3e-ae23-9446233d0924\") " Oct 31 00:17:15.687347 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:17:15.687280 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c8bf8e8-c2a1-4c3e-ae23-9446233d0924-trusted-ca-bundle" pod "8c8bf8e8-c2a1-4c3e-ae23-9446233d0924" (UID: "8c8bf8e8-c2a1-4c3e-ae23-9446233d0924"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:17:15.687347 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:17:15.687290 5222 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8c8bf8e8-c2a1-4c3e-ae23-9446233d0924-tmp\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:17:15.687347 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:17:15.687310 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c8bf8e8-c2a1-4c3e-ae23-9446233d0924-console-config" pod "8c8bf8e8-c2a1-4c3e-ae23-9446233d0924" (UID: "8c8bf8e8-c2a1-4c3e-ae23-9446233d0924"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:17:15.687695 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:17:15.687469 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c8bf8e8-c2a1-4c3e-ae23-9446233d0924-service-ca" pod "8c8bf8e8-c2a1-4c3e-ae23-9446233d0924" (UID: "8c8bf8e8-c2a1-4c3e-ae23-9446233d0924"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:17:15.687818 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:17:15.687797 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c8bf8e8-c2a1-4c3e-ae23-9446233d0924-oauth-serving-cert" pod "8c8bf8e8-c2a1-4c3e-ae23-9446233d0924" (UID: "8c8bf8e8-c2a1-4c3e-ae23-9446233d0924"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:17:15.689158 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:17:15.689129 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c8bf8e8-c2a1-4c3e-ae23-9446233d0924-console-oauth-config" pod "8c8bf8e8-c2a1-4c3e-ae23-9446233d0924" (UID: "8c8bf8e8-c2a1-4c3e-ae23-9446233d0924"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:17:15.689235 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:17:15.689165 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c8bf8e8-c2a1-4c3e-ae23-9446233d0924-kube-api-access-grw4v" pod "8c8bf8e8-c2a1-4c3e-ae23-9446233d0924" (UID: "8c8bf8e8-c2a1-4c3e-ae23-9446233d0924"). InnerVolumeSpecName "kube-api-access-grw4v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:17:15.689235 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:17:15.689206 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c8bf8e8-c2a1-4c3e-ae23-9446233d0924-console-serving-cert" pod "8c8bf8e8-c2a1-4c3e-ae23-9446233d0924" (UID: "8c8bf8e8-c2a1-4c3e-ae23-9446233d0924"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:17:15.788465 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:17:15.788427 5222 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8c8bf8e8-c2a1-4c3e-ae23-9446233d0924-console-config\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:17:15.788465 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:17:15.788459 5222 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8c8bf8e8-c2a1-4c3e-ae23-9446233d0924-service-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:17:15.788465 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:17:15.788469 5222 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8c8bf8e8-c2a1-4c3e-ae23-9446233d0924-oauth-serving-cert\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:17:15.788710 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:17:15.788478 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grw4v\" (UniqueName: \"kubernetes.io/projected/8c8bf8e8-c2a1-4c3e-ae23-9446233d0924-kube-api-access-grw4v\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:17:15.788710 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:17:15.788489 5222 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8c8bf8e8-c2a1-4c3e-ae23-9446233d0924-console-serving-cert\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:17:15.788710 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:17:15.788498 5222 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8c8bf8e8-c2a1-4c3e-ae23-9446233d0924-console-oauth-config\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:17:15.788710 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:17:15.788506 5222 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c8bf8e8-c2a1-4c3e-ae23-9446233d0924-trusted-ca-bundle\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:17:16.230316 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:17:16.230276 5222 generic.go:356] "Generic (PLEG): container finished" podID="8c8bf8e8-c2a1-4c3e-ae23-9446233d0924" containerID="41ba21539915ea1331b33805c4c4e714870b479dfb8c0e57d8d152ece64d4850" exitCode=0 Oct 31 00:17:16.230523 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:17:16.230326 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59c7d45ccf-r8m6l" Oct 31 00:17:16.230523 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:17:16.230350 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59c7d45ccf-r8m6l" event={"ID":"8c8bf8e8-c2a1-4c3e-ae23-9446233d0924","Type":"ContainerDied","Data":"41ba21539915ea1331b33805c4c4e714870b479dfb8c0e57d8d152ece64d4850"} Oct 31 00:17:16.230523 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:17:16.230418 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59c7d45ccf-r8m6l" event={"ID":"8c8bf8e8-c2a1-4c3e-ae23-9446233d0924","Type":"ContainerDied","Data":"f214561a97a576ec851ee7043c67238108c0a061bbdfdb02531b7969325c2a41"} Oct 31 00:17:16.230523 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:17:16.230445 5222 scope.go:122] "RemoveContainer" containerID="41ba21539915ea1331b33805c4c4e714870b479dfb8c0e57d8d152ece64d4850" Oct 31 00:17:16.238878 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:17:16.238861 5222 scope.go:122] "RemoveContainer" containerID="41ba21539915ea1331b33805c4c4e714870b479dfb8c0e57d8d152ece64d4850" Oct 31 00:17:16.239163 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:17:16.239145 5222 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41ba21539915ea1331b33805c4c4e714870b479dfb8c0e57d8d152ece64d4850\": container with ID starting with 41ba21539915ea1331b33805c4c4e714870b479dfb8c0e57d8d152ece64d4850 not found: ID does not exist" containerID="41ba21539915ea1331b33805c4c4e714870b479dfb8c0e57d8d152ece64d4850" Oct 31 00:17:16.239199 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:17:16.239175 5222 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41ba21539915ea1331b33805c4c4e714870b479dfb8c0e57d8d152ece64d4850"} err="failed to get container status \"41ba21539915ea1331b33805c4c4e714870b479dfb8c0e57d8d152ece64d4850\": rpc error: code = NotFound desc = could not find container \"41ba21539915ea1331b33805c4c4e714870b479dfb8c0e57d8d152ece64d4850\": container with ID starting with 41ba21539915ea1331b33805c4c4e714870b479dfb8c0e57d8d152ece64d4850 not found: ID does not exist" Oct 31 00:17:16.400570 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:17:16.400531 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-59c7d45ccf-r8m6l"] Oct 31 00:17:16.451794 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:17:16.451760 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-59c7d45ccf-r8m6l"] Oct 31 00:17:17.653314 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:17:17.653280 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c8bf8e8-c2a1-4c3e-ae23-9446233d0924" path="/var/lib/kubelet/pods/8c8bf8e8-c2a1-4c3e-ae23-9446233d0924/volumes" Oct 31 00:17:35.888451 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:17:35.888411 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 00:17:47.974077 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:17:47.974037 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-5d4d97f486-fwwz6"] Oct 31 00:17:47.974528 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:17:47.974345 5222 cpu_manager.go:414] "containerMap: removing container" podUID="8c8bf8e8-c2a1-4c3e-ae23-9446233d0924" containerName="console" Oct 31 00:17:47.974528 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:17:47.974357 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c8bf8e8-c2a1-4c3e-ae23-9446233d0924" containerName="console" Oct 31 00:17:47.974528 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:17:47.974445 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c8bf8e8-c2a1-4c3e-ae23-9446233d0924" containerName="console" Oct 31 00:17:47.976238 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:17:47.976220 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d4d97f486-fwwz6" Oct 31 00:17:48.036682 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:17:48.036645 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/11dbf2c5-359a-46e0-a2c4-a93065c4678e-console-config\") pod \"console-5d4d97f486-fwwz6\" (UID: \"11dbf2c5-359a-46e0-a2c4-a93065c4678e\") " pod="openshift-console/console-5d4d97f486-fwwz6" Oct 31 00:17:48.036682 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:17:48.036685 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/11dbf2c5-359a-46e0-a2c4-a93065c4678e-oauth-serving-cert\") pod \"console-5d4d97f486-fwwz6\" (UID: \"11dbf2c5-359a-46e0-a2c4-a93065c4678e\") " pod="openshift-console/console-5d4d97f486-fwwz6" Oct 31 00:17:48.036891 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:17:48.036787 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11dbf2c5-359a-46e0-a2c4-a93065c4678e-trusted-ca-bundle\") pod \"console-5d4d97f486-fwwz6\" (UID: \"11dbf2c5-359a-46e0-a2c4-a93065c4678e\") " pod="openshift-console/console-5d4d97f486-fwwz6" Oct 31 00:17:48.036891 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:17:48.036806 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/11dbf2c5-359a-46e0-a2c4-a93065c4678e-service-ca\") pod \"console-5d4d97f486-fwwz6\" (UID: \"11dbf2c5-359a-46e0-a2c4-a93065c4678e\") " pod="openshift-console/console-5d4d97f486-fwwz6" Oct 31 00:17:48.036891 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:17:48.036840 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/11dbf2c5-359a-46e0-a2c4-a93065c4678e-console-oauth-config\") pod \"console-5d4d97f486-fwwz6\" (UID: \"11dbf2c5-359a-46e0-a2c4-a93065c4678e\") " pod="openshift-console/console-5d4d97f486-fwwz6" Oct 31 00:17:48.036986 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:17:48.036906 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/11dbf2c5-359a-46e0-a2c4-a93065c4678e-console-serving-cert\") pod \"console-5d4d97f486-fwwz6\" (UID: \"11dbf2c5-359a-46e0-a2c4-a93065c4678e\") " pod="openshift-console/console-5d4d97f486-fwwz6" Oct 31 00:17:48.037021 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:17:48.036986 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/11dbf2c5-359a-46e0-a2c4-a93065c4678e-tmp\") pod \"console-5d4d97f486-fwwz6\" (UID: \"11dbf2c5-359a-46e0-a2c4-a93065c4678e\") " pod="openshift-console/console-5d4d97f486-fwwz6" Oct 31 00:17:48.037053 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:17:48.037023 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5kgm\" (UniqueName: \"kubernetes.io/projected/11dbf2c5-359a-46e0-a2c4-a93065c4678e-kube-api-access-n5kgm\") pod \"console-5d4d97f486-fwwz6\" (UID: \"11dbf2c5-359a-46e0-a2c4-a93065c4678e\") " pod="openshift-console/console-5d4d97f486-fwwz6" Oct 31 00:17:48.075687 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:17:48.075657 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d4d97f486-fwwz6"] Oct 31 00:17:48.137891 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:17:48.137850 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/11dbf2c5-359a-46e0-a2c4-a93065c4678e-console-oauth-config\") pod \"console-5d4d97f486-fwwz6\" (UID: \"11dbf2c5-359a-46e0-a2c4-a93065c4678e\") " pod="openshift-console/console-5d4d97f486-fwwz6" Oct 31 00:17:48.137891 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:17:48.137894 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/11dbf2c5-359a-46e0-a2c4-a93065c4678e-console-serving-cert\") pod \"console-5d4d97f486-fwwz6\" (UID: \"11dbf2c5-359a-46e0-a2c4-a93065c4678e\") " pod="openshift-console/console-5d4d97f486-fwwz6" Oct 31 00:17:48.138119 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:17:48.137917 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/11dbf2c5-359a-46e0-a2c4-a93065c4678e-tmp\") pod \"console-5d4d97f486-fwwz6\" (UID: \"11dbf2c5-359a-46e0-a2c4-a93065c4678e\") " pod="openshift-console/console-5d4d97f486-fwwz6" Oct 31 00:17:48.138119 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:17:48.137976 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5kgm\" (UniqueName: \"kubernetes.io/projected/11dbf2c5-359a-46e0-a2c4-a93065c4678e-kube-api-access-n5kgm\") pod \"console-5d4d97f486-fwwz6\" (UID: \"11dbf2c5-359a-46e0-a2c4-a93065c4678e\") " pod="openshift-console/console-5d4d97f486-fwwz6" Oct 31 00:17:48.138119 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:17:48.138029 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/11dbf2c5-359a-46e0-a2c4-a93065c4678e-console-config\") pod \"console-5d4d97f486-fwwz6\" (UID: \"11dbf2c5-359a-46e0-a2c4-a93065c4678e\") " pod="openshift-console/console-5d4d97f486-fwwz6" Oct 31 00:17:48.138119 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:17:48.138048 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/11dbf2c5-359a-46e0-a2c4-a93065c4678e-oauth-serving-cert\") pod \"console-5d4d97f486-fwwz6\" (UID: \"11dbf2c5-359a-46e0-a2c4-a93065c4678e\") " pod="openshift-console/console-5d4d97f486-fwwz6" Oct 31 00:17:48.138119 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:17:48.138105 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11dbf2c5-359a-46e0-a2c4-a93065c4678e-trusted-ca-bundle\") pod \"console-5d4d97f486-fwwz6\" (UID: \"11dbf2c5-359a-46e0-a2c4-a93065c4678e\") " pod="openshift-console/console-5d4d97f486-fwwz6" Oct 31 00:17:48.138394 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:17:48.138129 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/11dbf2c5-359a-46e0-a2c4-a93065c4678e-service-ca\") pod \"console-5d4d97f486-fwwz6\" (UID: \"11dbf2c5-359a-46e0-a2c4-a93065c4678e\") " pod="openshift-console/console-5d4d97f486-fwwz6" Oct 31 00:17:48.138394 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:17:48.138280 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/11dbf2c5-359a-46e0-a2c4-a93065c4678e-tmp\") pod \"console-5d4d97f486-fwwz6\" (UID: \"11dbf2c5-359a-46e0-a2c4-a93065c4678e\") " pod="openshift-console/console-5d4d97f486-fwwz6" Oct 31 00:17:48.138823 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:17:48.138795 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/11dbf2c5-359a-46e0-a2c4-a93065c4678e-console-config\") pod \"console-5d4d97f486-fwwz6\" (UID: \"11dbf2c5-359a-46e0-a2c4-a93065c4678e\") " pod="openshift-console/console-5d4d97f486-fwwz6" Oct 31 00:17:48.138823 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:17:48.138818 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/11dbf2c5-359a-46e0-a2c4-a93065c4678e-service-ca\") pod \"console-5d4d97f486-fwwz6\" (UID: \"11dbf2c5-359a-46e0-a2c4-a93065c4678e\") " pod="openshift-console/console-5d4d97f486-fwwz6" Oct 31 00:17:48.139001 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:17:48.138974 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/11dbf2c5-359a-46e0-a2c4-a93065c4678e-oauth-serving-cert\") pod \"console-5d4d97f486-fwwz6\" (UID: \"11dbf2c5-359a-46e0-a2c4-a93065c4678e\") " pod="openshift-console/console-5d4d97f486-fwwz6" Oct 31 00:17:48.139128 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:17:48.139109 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11dbf2c5-359a-46e0-a2c4-a93065c4678e-trusted-ca-bundle\") pod \"console-5d4d97f486-fwwz6\" (UID: \"11dbf2c5-359a-46e0-a2c4-a93065c4678e\") " pod="openshift-console/console-5d4d97f486-fwwz6" Oct 31 00:17:48.140878 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:17:48.140856 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/11dbf2c5-359a-46e0-a2c4-a93065c4678e-console-oauth-config\") pod \"console-5d4d97f486-fwwz6\" (UID: \"11dbf2c5-359a-46e0-a2c4-a93065c4678e\") " pod="openshift-console/console-5d4d97f486-fwwz6" Oct 31 00:17:48.141115 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:17:48.141095 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/11dbf2c5-359a-46e0-a2c4-a93065c4678e-console-serving-cert\") pod \"console-5d4d97f486-fwwz6\" (UID: \"11dbf2c5-359a-46e0-a2c4-a93065c4678e\") " pod="openshift-console/console-5d4d97f486-fwwz6" Oct 31 00:17:48.202542 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:17:48.202507 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5kgm\" (UniqueName: \"kubernetes.io/projected/11dbf2c5-359a-46e0-a2c4-a93065c4678e-kube-api-access-n5kgm\") pod \"console-5d4d97f486-fwwz6\" (UID: \"11dbf2c5-359a-46e0-a2c4-a93065c4678e\") " pod="openshift-console/console-5d4d97f486-fwwz6" Oct 31 00:17:48.286625 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:17:48.286586 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d4d97f486-fwwz6" Oct 31 00:17:48.658818 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:17:48.658772 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11dbf2c5_359a_46e0_a2c4_a93065c4678e.slice/crio-f2fd153981c3bb181e30e623d1f5535fb97d7ca4fbcd61c9de0eb438ddb1f264 WatchSource:0}: Error finding container f2fd153981c3bb181e30e623d1f5535fb97d7ca4fbcd61c9de0eb438ddb1f264: Status 404 returned error can't find the container with id f2fd153981c3bb181e30e623d1f5535fb97d7ca4fbcd61c9de0eb438ddb1f264 Oct 31 00:17:48.659274 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:17:48.659250 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d4d97f486-fwwz6"] Oct 31 00:17:49.351308 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:17:49.351268 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d4d97f486-fwwz6" event={"ID":"11dbf2c5-359a-46e0-a2c4-a93065c4678e","Type":"ContainerStarted","Data":"22a954a7606c6655f243e782730a401de9685e3002ecca1e16b04fa81474113e"} Oct 31 00:17:49.351743 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:17:49.351333 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d4d97f486-fwwz6" event={"ID":"11dbf2c5-359a-46e0-a2c4-a93065c4678e","Type":"ContainerStarted","Data":"f2fd153981c3bb181e30e623d1f5535fb97d7ca4fbcd61c9de0eb438ddb1f264"} Oct 31 00:17:49.472626 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:17:49.472576 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-5d4d97f486-fwwz6" podStartSLOduration=2.472561154 podStartE2EDuration="2.472561154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:17:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:17:49.472090952 +0000 UTC m=+161.943260700" watchObservedRunningTime="2026-10-31 00:17:49.472561154 +0000 UTC m=+161.943730902" Oct 31 00:17:58.287090 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:17:58.287035 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5d4d97f486-fwwz6" Oct 31 00:17:58.287513 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:17:58.287116 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5d4d97f486-fwwz6" Oct 31 00:17:58.291805 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:17:58.291781 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5d4d97f486-fwwz6" Oct 31 00:17:58.392968 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:17:58.392933 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5d4d97f486-fwwz6" Oct 31 00:17:58.567469 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:17:58.567389 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-9c784f57c-8f2zj"] Oct 31 00:18:05.889092 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:18:05.889054 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 00:18:23.590823 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:18:23.590712 5222 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-9c784f57c-8f2zj" podUID="de7fa613-44c4-4e72-974d-9890609d4582" containerName="console" containerID="cri-o://65f378b175f206df1bf24f8e1433c90ccf048d8cda4075f222f99170bed037f0" gracePeriod=15 Oct 31 00:18:23.831541 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:18:23.831513 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9c784f57c-8f2zj" Oct 31 00:18:23.936757 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:18:23.936667 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de7fa613-44c4-4e72-974d-9890609d4582-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de7fa613-44c4-4e72-974d-9890609d4582-trusted-ca-bundle\") pod \"de7fa613-44c4-4e72-974d-9890609d4582\" (UID: \"de7fa613-44c4-4e72-974d-9890609d4582\") " Oct 31 00:18:23.936757 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:18:23.936717 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de7fa613-44c4-4e72-974d-9890609d4582-kube-api-access-kbgxt\" (UniqueName: \"kubernetes.io/projected/de7fa613-44c4-4e72-974d-9890609d4582-kube-api-access-kbgxt\") pod \"de7fa613-44c4-4e72-974d-9890609d4582\" (UID: \"de7fa613-44c4-4e72-974d-9890609d4582\") " Oct 31 00:18:23.936757 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:18:23.936736 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de7fa613-44c4-4e72-974d-9890609d4582-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/de7fa613-44c4-4e72-974d-9890609d4582-oauth-serving-cert\") pod \"de7fa613-44c4-4e72-974d-9890609d4582\" (UID: \"de7fa613-44c4-4e72-974d-9890609d4582\") " Oct 31 00:18:23.936991 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:18:23.936832 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de7fa613-44c4-4e72-974d-9890609d4582-tmp\" (UniqueName: \"kubernetes.io/empty-dir/de7fa613-44c4-4e72-974d-9890609d4582-tmp\") pod \"de7fa613-44c4-4e72-974d-9890609d4582\" (UID: \"de7fa613-44c4-4e72-974d-9890609d4582\") " Oct 31 00:18:23.936991 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:18:23.936869 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de7fa613-44c4-4e72-974d-9890609d4582-console-config\" (UniqueName: \"kubernetes.io/configmap/de7fa613-44c4-4e72-974d-9890609d4582-console-config\") pod \"de7fa613-44c4-4e72-974d-9890609d4582\" (UID: \"de7fa613-44c4-4e72-974d-9890609d4582\") " Oct 31 00:18:23.936991 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:18:23.936913 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de7fa613-44c4-4e72-974d-9890609d4582-service-ca\" (UniqueName: \"kubernetes.io/configmap/de7fa613-44c4-4e72-974d-9890609d4582-service-ca\") pod \"de7fa613-44c4-4e72-974d-9890609d4582\" (UID: \"de7fa613-44c4-4e72-974d-9890609d4582\") " Oct 31 00:18:23.936991 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:18:23.936944 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de7fa613-44c4-4e72-974d-9890609d4582-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/de7fa613-44c4-4e72-974d-9890609d4582-console-serving-cert\") pod \"de7fa613-44c4-4e72-974d-9890609d4582\" (UID: \"de7fa613-44c4-4e72-974d-9890609d4582\") " Oct 31 00:18:23.936991 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:18:23.936985 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de7fa613-44c4-4e72-974d-9890609d4582-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/de7fa613-44c4-4e72-974d-9890609d4582-console-oauth-config\") pod \"de7fa613-44c4-4e72-974d-9890609d4582\" (UID: \"de7fa613-44c4-4e72-974d-9890609d4582\") " Oct 31 00:18:23.937246 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:18:23.937152 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de7fa613-44c4-4e72-974d-9890609d4582-oauth-serving-cert" pod "de7fa613-44c4-4e72-974d-9890609d4582" (UID: "de7fa613-44c4-4e72-974d-9890609d4582"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:18:23.937305 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:18:23.937204 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de7fa613-44c4-4e72-974d-9890609d4582-trusted-ca-bundle" pod "de7fa613-44c4-4e72-974d-9890609d4582" (UID: "de7fa613-44c4-4e72-974d-9890609d4582"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:18:23.937305 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:18:23.937173 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de7fa613-44c4-4e72-974d-9890609d4582-tmp" pod "de7fa613-44c4-4e72-974d-9890609d4582" (UID: "de7fa613-44c4-4e72-974d-9890609d4582"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:18:23.937446 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:18:23.937333 5222 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/de7fa613-44c4-4e72-974d-9890609d4582-oauth-serving-cert\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:18:23.937446 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:18:23.937351 5222 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/de7fa613-44c4-4e72-974d-9890609d4582-tmp\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:18:23.937446 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:18:23.937347 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de7fa613-44c4-4e72-974d-9890609d4582-console-config" pod "de7fa613-44c4-4e72-974d-9890609d4582" (UID: "de7fa613-44c4-4e72-974d-9890609d4582"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:18:23.937585 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:18:23.937384 5222 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de7fa613-44c4-4e72-974d-9890609d4582-trusted-ca-bundle\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:18:23.937585 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:18:23.937529 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de7fa613-44c4-4e72-974d-9890609d4582-service-ca" pod "de7fa613-44c4-4e72-974d-9890609d4582" (UID: "de7fa613-44c4-4e72-974d-9890609d4582"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:18:23.939395 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:18:23.939361 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de7fa613-44c4-4e72-974d-9890609d4582-console-oauth-config" pod "de7fa613-44c4-4e72-974d-9890609d4582" (UID: "de7fa613-44c4-4e72-974d-9890609d4582"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:18:23.939659 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:18:23.939638 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de7fa613-44c4-4e72-974d-9890609d4582-kube-api-access-kbgxt" pod "de7fa613-44c4-4e72-974d-9890609d4582" (UID: "de7fa613-44c4-4e72-974d-9890609d4582"). InnerVolumeSpecName "kube-api-access-kbgxt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:18:23.939659 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:18:23.939645 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de7fa613-44c4-4e72-974d-9890609d4582-console-serving-cert" pod "de7fa613-44c4-4e72-974d-9890609d4582" (UID: "de7fa613-44c4-4e72-974d-9890609d4582"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:18:24.038036 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:18:24.037992 5222 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/de7fa613-44c4-4e72-974d-9890609d4582-console-config\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:18:24.038036 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:18:24.038026 5222 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/de7fa613-44c4-4e72-974d-9890609d4582-service-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:18:24.038036 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:18:24.038036 5222 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/de7fa613-44c4-4e72-974d-9890609d4582-console-serving-cert\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:18:24.038036 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:18:24.038045 5222 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/de7fa613-44c4-4e72-974d-9890609d4582-console-oauth-config\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:18:24.038303 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:18:24.038053 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbgxt\" (UniqueName: \"kubernetes.io/projected/de7fa613-44c4-4e72-974d-9890609d4582-kube-api-access-kbgxt\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:18:24.480075 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:18:24.480038 5222 generic.go:356] "Generic (PLEG): container finished" podID="de7fa613-44c4-4e72-974d-9890609d4582" containerID="65f378b175f206df1bf24f8e1433c90ccf048d8cda4075f222f99170bed037f0" exitCode=0 Oct 31 00:18:24.480273 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:18:24.480079 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9c784f57c-8f2zj" event={"ID":"de7fa613-44c4-4e72-974d-9890609d4582","Type":"ContainerDied","Data":"65f378b175f206df1bf24f8e1433c90ccf048d8cda4075f222f99170bed037f0"} Oct 31 00:18:24.480273 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:18:24.480093 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9c784f57c-8f2zj" Oct 31 00:18:24.480273 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:18:24.480165 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9c784f57c-8f2zj" event={"ID":"de7fa613-44c4-4e72-974d-9890609d4582","Type":"ContainerDied","Data":"5a1fd723caaab229a70ed883d401f71920ea7555bf9ed21f1468917056ec7ab6"} Oct 31 00:18:24.480273 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:18:24.480192 5222 scope.go:122] "RemoveContainer" containerID="65f378b175f206df1bf24f8e1433c90ccf048d8cda4075f222f99170bed037f0" Oct 31 00:18:24.489653 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:18:24.489630 5222 scope.go:122] "RemoveContainer" containerID="65f378b175f206df1bf24f8e1433c90ccf048d8cda4075f222f99170bed037f0" Oct 31 00:18:24.489957 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:18:24.489938 5222 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65f378b175f206df1bf24f8e1433c90ccf048d8cda4075f222f99170bed037f0\": container with ID starting with 65f378b175f206df1bf24f8e1433c90ccf048d8cda4075f222f99170bed037f0 not found: ID does not exist" containerID="65f378b175f206df1bf24f8e1433c90ccf048d8cda4075f222f99170bed037f0" Oct 31 00:18:24.490005 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:18:24.489968 5222 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65f378b175f206df1bf24f8e1433c90ccf048d8cda4075f222f99170bed037f0"} err="failed to get container status \"65f378b175f206df1bf24f8e1433c90ccf048d8cda4075f222f99170bed037f0\": rpc error: code = NotFound desc = could not find container \"65f378b175f206df1bf24f8e1433c90ccf048d8cda4075f222f99170bed037f0\": container with ID starting with 65f378b175f206df1bf24f8e1433c90ccf048d8cda4075f222f99170bed037f0 not found: ID does not exist" Oct 31 00:18:24.649915 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:18:24.649881 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-9c784f57c-8f2zj"] Oct 31 00:18:24.657035 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:18:24.657006 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-9c784f57c-8f2zj"] Oct 31 00:18:25.653616 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:18:25.653585 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de7fa613-44c4-4e72-974d-9890609d4582" path="/var/lib/kubelet/pods/de7fa613-44c4-4e72-974d-9890609d4582/volumes" Oct 31 00:18:35.889070 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:18:35.889024 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 00:19:05.887537 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:19:05.887498 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 00:19:35.888048 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:19:35.888012 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 00:19:51.147017 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:19:51.146934 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449gjvm"] Oct 31 00:19:51.147423 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:19:51.147238 5222 cpu_manager.go:414] "containerMap: removing container" podUID="de7fa613-44c4-4e72-974d-9890609d4582" containerName="console" Oct 31 00:19:51.147423 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:19:51.147256 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de7fa613-44c4-4e72-974d-9890609d4582" containerName="console" Oct 31 00:19:51.147423 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:19:51.147334 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="de7fa613-44c4-4e72-974d-9890609d4582" containerName="console" Oct 31 00:19:51.150385 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:19:51.150353 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449gjvm" Oct 31 00:19:51.182659 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:19:51.182622 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7fxtx"] Oct 31 00:19:51.185932 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:19:51.185914 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7fxtx" Oct 31 00:19:51.206263 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:19:51.206237 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-rqwz5\"" Oct 31 00:19:51.233173 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:19:51.233143 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 31 00:19:51.233173 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:19:51.233143 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 31 00:19:51.251433 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:19:51.251396 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ed46bf63-2f4f-4408-b5d3-7208ecffe89c-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7fxtx\" (UID: \"ed46bf63-2f4f-4408-b5d3-7208ecffe89c\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7fxtx" Oct 31 00:19:51.251623 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:19:51.251445 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b961ce20-bb4c-4916-8f7a-6762c24e7014-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449gjvm\" (UID: \"b961ce20-bb4c-4916-8f7a-6762c24e7014\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449gjvm" Oct 31 00:19:51.251623 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:19:51.251533 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnznl\" (UniqueName: \"kubernetes.io/projected/ed46bf63-2f4f-4408-b5d3-7208ecffe89c-kube-api-access-dnznl\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7fxtx\" (UID: \"ed46bf63-2f4f-4408-b5d3-7208ecffe89c\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7fxtx" Oct 31 00:19:51.251623 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:19:51.251581 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ed46bf63-2f4f-4408-b5d3-7208ecffe89c-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7fxtx\" (UID: \"ed46bf63-2f4f-4408-b5d3-7208ecffe89c\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7fxtx" Oct 31 00:19:51.251803 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:19:51.251660 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwm69\" (UniqueName: \"kubernetes.io/projected/b961ce20-bb4c-4916-8f7a-6762c24e7014-kube-api-access-jwm69\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449gjvm\" (UID: \"b961ce20-bb4c-4916-8f7a-6762c24e7014\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449gjvm" Oct 31 00:19:51.251803 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:19:51.251681 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b961ce20-bb4c-4916-8f7a-6762c24e7014-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449gjvm\" (UID: \"b961ce20-bb4c-4916-8f7a-6762c24e7014\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449gjvm" Oct 31 00:19:51.257562 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:19:51.257527 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-cwcnd\"" Oct 31 00:19:51.258464 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:19:51.258439 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449gjvm"] Oct 31 00:19:51.307718 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:19:51.307681 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7fxtx"] Oct 31 00:19:51.352591 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:19:51.352560 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ed46bf63-2f4f-4408-b5d3-7208ecffe89c-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7fxtx\" (UID: \"ed46bf63-2f4f-4408-b5d3-7208ecffe89c\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7fxtx" Oct 31 00:19:51.352805 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:19:51.352611 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b961ce20-bb4c-4916-8f7a-6762c24e7014-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449gjvm\" (UID: \"b961ce20-bb4c-4916-8f7a-6762c24e7014\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449gjvm" Oct 31 00:19:51.352805 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:19:51.352735 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnznl\" (UniqueName: \"kubernetes.io/projected/ed46bf63-2f4f-4408-b5d3-7208ecffe89c-kube-api-access-dnznl\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7fxtx\" (UID: \"ed46bf63-2f4f-4408-b5d3-7208ecffe89c\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7fxtx" Oct 31 00:19:51.352805 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:19:51.352796 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ed46bf63-2f4f-4408-b5d3-7208ecffe89c-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7fxtx\" (UID: \"ed46bf63-2f4f-4408-b5d3-7208ecffe89c\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7fxtx" Oct 31 00:19:51.352937 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:19:51.352846 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwm69\" (UniqueName: \"kubernetes.io/projected/b961ce20-bb4c-4916-8f7a-6762c24e7014-kube-api-access-jwm69\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449gjvm\" (UID: \"b961ce20-bb4c-4916-8f7a-6762c24e7014\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449gjvm" Oct 31 00:19:51.352937 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:19:51.352880 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b961ce20-bb4c-4916-8f7a-6762c24e7014-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449gjvm\" (UID: \"b961ce20-bb4c-4916-8f7a-6762c24e7014\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449gjvm" Oct 31 00:19:51.353052 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:19:51.353030 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ed46bf63-2f4f-4408-b5d3-7208ecffe89c-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7fxtx\" (UID: \"ed46bf63-2f4f-4408-b5d3-7208ecffe89c\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7fxtx" Oct 31 00:19:51.353093 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:19:51.353048 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b961ce20-bb4c-4916-8f7a-6762c24e7014-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449gjvm\" (UID: \"b961ce20-bb4c-4916-8f7a-6762c24e7014\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449gjvm" Oct 31 00:19:51.353127 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:19:51.353091 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ed46bf63-2f4f-4408-b5d3-7208ecffe89c-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7fxtx\" (UID: \"ed46bf63-2f4f-4408-b5d3-7208ecffe89c\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7fxtx" Oct 31 00:19:51.353174 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:19:51.353154 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b961ce20-bb4c-4916-8f7a-6762c24e7014-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449gjvm\" (UID: \"b961ce20-bb4c-4916-8f7a-6762c24e7014\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449gjvm" Oct 31 00:19:51.415322 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:19:51.415251 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwm69\" (UniqueName: \"kubernetes.io/projected/b961ce20-bb4c-4916-8f7a-6762c24e7014-kube-api-access-jwm69\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449gjvm\" (UID: \"b961ce20-bb4c-4916-8f7a-6762c24e7014\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449gjvm" Oct 31 00:19:51.415672 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:19:51.415648 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnznl\" (UniqueName: \"kubernetes.io/projected/ed46bf63-2f4f-4408-b5d3-7208ecffe89c-kube-api-access-dnznl\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7fxtx\" (UID: \"ed46bf63-2f4f-4408-b5d3-7208ecffe89c\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7fxtx" Oct 31 00:19:51.466623 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:19:51.466592 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449gjvm" Oct 31 00:19:51.501785 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:19:51.501747 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7fxtx" Oct 31 00:19:51.660591 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:19:51.660554 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-qbvvz"] Oct 31 00:19:51.664036 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:19:51.664018 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-qbvvz" Oct 31 00:19:51.718627 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:19:51.718587 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-qbvvz"] Oct 31 00:19:51.723579 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:19:51.723552 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 31 00:19:51.733221 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:19:51.733195 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 31 00:19:51.733404 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:19:51.733240 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 31 00:19:51.733404 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:19:51.733283 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 31 00:19:51.733404 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:19:51.733283 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 31 00:19:51.756762 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:19:51.756728 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f75a02c-5239-4529-a375-bbcee7e5118d-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-qbvvz\" (UID: \"2f75a02c-5239-4529-a375-bbcee7e5118d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-qbvvz" Oct 31 00:19:51.756762 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:19:51.756766 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f75a02c-5239-4529-a375-bbcee7e5118d-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-qbvvz\" (UID: \"2f75a02c-5239-4529-a375-bbcee7e5118d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-qbvvz" Oct 31 00:19:51.756960 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:19:51.756791 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2f75a02c-5239-4529-a375-bbcee7e5118d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-qbvvz\" (UID: \"2f75a02c-5239-4529-a375-bbcee7e5118d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-qbvvz" Oct 31 00:19:51.756960 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:19:51.756813 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2f75a02c-5239-4529-a375-bbcee7e5118d-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-qbvvz\" (UID: \"2f75a02c-5239-4529-a375-bbcee7e5118d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-qbvvz" Oct 31 00:19:51.756960 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:19:51.756926 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n86sg\" (UniqueName: \"kubernetes.io/projected/2f75a02c-5239-4529-a375-bbcee7e5118d-kube-api-access-n86sg\") pod \"istiod-openshift-gateway-7876bfcb74-qbvvz\" (UID: \"2f75a02c-5239-4529-a375-bbcee7e5118d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-qbvvz" Oct 31 00:19:51.756960 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:19:51.756954 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2f75a02c-5239-4529-a375-bbcee7e5118d-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-qbvvz\" (UID: \"2f75a02c-5239-4529-a375-bbcee7e5118d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-qbvvz" Oct 31 00:19:51.757084 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:19:51.757022 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2f75a02c-5239-4529-a375-bbcee7e5118d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-qbvvz\" (UID: \"2f75a02c-5239-4529-a375-bbcee7e5118d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-qbvvz" Oct 31 00:19:51.824238 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:19:51.824212 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 31 00:19:51.824238 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:19:51.824212 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-pbwgm\"" Oct 31 00:19:51.841316 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:19:51.839914 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449gjvm"] Oct 31 00:19:51.841585 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:19:51.841423 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb961ce20_bb4c_4916_8f7a_6762c24e7014.slice/crio-73932c54fc569d5d5fbf55c83308b133c0948ea5b0cf9197dccdeaa4384f3795 WatchSource:0}: Error finding container 73932c54fc569d5d5fbf55c83308b133c0948ea5b0cf9197dccdeaa4384f3795: Status 404 returned error can't find the container with id 73932c54fc569d5d5fbf55c83308b133c0948ea5b0cf9197dccdeaa4384f3795 Oct 31 00:19:51.857444 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:19:51.857412 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f75a02c-5239-4529-a375-bbcee7e5118d-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-qbvvz\" (UID: \"2f75a02c-5239-4529-a375-bbcee7e5118d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-qbvvz" Oct 31 00:19:51.857444 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:19:51.857449 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f75a02c-5239-4529-a375-bbcee7e5118d-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-qbvvz\" (UID: \"2f75a02c-5239-4529-a375-bbcee7e5118d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-qbvvz" Oct 31 00:19:51.857683 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:19:51.857473 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2f75a02c-5239-4529-a375-bbcee7e5118d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-qbvvz\" (UID: \"2f75a02c-5239-4529-a375-bbcee7e5118d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-qbvvz" Oct 31 00:19:51.857683 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:19:51.857524 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2f75a02c-5239-4529-a375-bbcee7e5118d-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-qbvvz\" (UID: \"2f75a02c-5239-4529-a375-bbcee7e5118d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-qbvvz" Oct 31 00:19:51.857683 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:19:51.857557 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n86sg\" (UniqueName: \"kubernetes.io/projected/2f75a02c-5239-4529-a375-bbcee7e5118d-kube-api-access-n86sg\") pod \"istiod-openshift-gateway-7876bfcb74-qbvvz\" (UID: \"2f75a02c-5239-4529-a375-bbcee7e5118d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-qbvvz" Oct 31 00:19:51.857683 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:19:51.857585 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2f75a02c-5239-4529-a375-bbcee7e5118d-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-qbvvz\" (UID: \"2f75a02c-5239-4529-a375-bbcee7e5118d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-qbvvz" Oct 31 00:19:51.857683 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:19:51.857651 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2f75a02c-5239-4529-a375-bbcee7e5118d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-qbvvz\" (UID: \"2f75a02c-5239-4529-a375-bbcee7e5118d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-qbvvz" Oct 31 00:19:51.858303 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:19:51.858227 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2f75a02c-5239-4529-a375-bbcee7e5118d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-qbvvz\" (UID: \"2f75a02c-5239-4529-a375-bbcee7e5118d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-qbvvz" Oct 31 00:19:51.860058 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:19:51.860033 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f75a02c-5239-4529-a375-bbcee7e5118d-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-qbvvz\" (UID: \"2f75a02c-5239-4529-a375-bbcee7e5118d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-qbvvz" Oct 31 00:19:51.860164 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:19:51.860032 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2f75a02c-5239-4529-a375-bbcee7e5118d-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-qbvvz\" (UID: \"2f75a02c-5239-4529-a375-bbcee7e5118d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-qbvvz" Oct 31 00:19:51.860164 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:19:51.860150 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2f75a02c-5239-4529-a375-bbcee7e5118d-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-qbvvz\" (UID: \"2f75a02c-5239-4529-a375-bbcee7e5118d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-qbvvz" Oct 31 00:19:51.860288 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:19:51.860241 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2f75a02c-5239-4529-a375-bbcee7e5118d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-qbvvz\" (UID: \"2f75a02c-5239-4529-a375-bbcee7e5118d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-qbvvz" Oct 31 00:19:51.920176 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:19:51.920134 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f75a02c-5239-4529-a375-bbcee7e5118d-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-qbvvz\" (UID: \"2f75a02c-5239-4529-a375-bbcee7e5118d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-qbvvz" Oct 31 00:19:51.920387 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:19:51.920305 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n86sg\" (UniqueName: \"kubernetes.io/projected/2f75a02c-5239-4529-a375-bbcee7e5118d-kube-api-access-n86sg\") pod \"istiod-openshift-gateway-7876bfcb74-qbvvz\" (UID: \"2f75a02c-5239-4529-a375-bbcee7e5118d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-qbvvz" Oct 31 00:19:51.935315 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:19:51.935283 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded46bf63_2f4f_4408_b5d3_7208ecffe89c.slice/crio-ffc28d1a6b5b2ff3e78f28a2eaa5d2224f452c0bacbaa38b0061b80df952031f WatchSource:0}: Error finding container ffc28d1a6b5b2ff3e78f28a2eaa5d2224f452c0bacbaa38b0061b80df952031f: Status 404 returned error can't find the container with id ffc28d1a6b5b2ff3e78f28a2eaa5d2224f452c0bacbaa38b0061b80df952031f Oct 31 00:19:51.936484 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:19:51.936464 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7fxtx"] Oct 31 00:19:51.976483 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:19:51.976440 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-qbvvz" Oct 31 00:19:52.352799 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:19:52.350712 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-qbvvz"] Oct 31 00:19:52.353148 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:19:52.353050 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f75a02c_5239_4529_a375_bbcee7e5118d.slice/crio-ac6eda2ddc393feceeb17b3671ed760f5213ade97b95e190937ee52aada8db88 WatchSource:0}: Error finding container ac6eda2ddc393feceeb17b3671ed760f5213ade97b95e190937ee52aada8db88: Status 404 returned error can't find the container with id ac6eda2ddc393feceeb17b3671ed760f5213ade97b95e190937ee52aada8db88 Oct 31 00:19:52.790331 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:19:52.790297 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7fxtx" event={"ID":"ed46bf63-2f4f-4408-b5d3-7208ecffe89c","Type":"ContainerStarted","Data":"ffc28d1a6b5b2ff3e78f28a2eaa5d2224f452c0bacbaa38b0061b80df952031f"} Oct 31 00:19:52.791515 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:19:52.791491 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449gjvm" event={"ID":"b961ce20-bb4c-4916-8f7a-6762c24e7014","Type":"ContainerStarted","Data":"73932c54fc569d5d5fbf55c83308b133c0948ea5b0cf9197dccdeaa4384f3795"} Oct 31 00:19:52.792634 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:19:52.792612 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-qbvvz" event={"ID":"2f75a02c-5239-4529-a375-bbcee7e5118d","Type":"ContainerStarted","Data":"ac6eda2ddc393feceeb17b3671ed760f5213ade97b95e190937ee52aada8db88"} Oct 31 00:19:57.026185 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:19:57.026147 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:19:57.026674 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:19:57.026232 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:19:57.824129 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:19:57.824090 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-qbvvz" event={"ID":"2f75a02c-5239-4529-a375-bbcee7e5118d","Type":"ContainerStarted","Data":"ffeb147b2eedd37bed4ea352d05c1cb76963feb2d9f4a93f0b91bf05c84c5c39"} Oct 31 00:19:57.824305 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:19:57.824227 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-qbvvz" Oct 31 00:19:58.054247 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:19:58.054175 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-qbvvz" podStartSLOduration=2.383876602 podStartE2EDuration="7.054145348s" totalImagesPullingTime="4.670268746s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:19:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:19:58.001779063 +0000 UTC m=+290.472948812" watchObservedRunningTime="2026-10-31 00:19:58.054145348 +0000 UTC m=+290.525315098" Oct 31 00:19:58.831524 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:19:58.831480 5222 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-qbvvz container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:19:58.831712 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:19:58.831566 5222 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-qbvvz" podUID="2f75a02c-5239-4529-a375-bbcee7e5118d" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:20:00.839201 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:20:00.839163 5222 generic.go:356] "Generic (PLEG): container finished" podID="b961ce20-bb4c-4916-8f7a-6762c24e7014" containerID="303443d72c5245ddf6531ec90d618b770c91c7c9391c1ba7f022f5d253d6ec4b" exitCode=0 Oct 31 00:20:00.839709 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:20:00.839245 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449gjvm" event={"ID":"b961ce20-bb4c-4916-8f7a-6762c24e7014","Type":"ContainerDied","Data":"303443d72c5245ddf6531ec90d618b770c91c7c9391c1ba7f022f5d253d6ec4b"} Oct 31 00:20:01.830773 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:20:01.830735 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-qbvvz" Oct 31 00:20:03.857587 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:20:03.857493 5222 generic.go:356] "Generic (PLEG): container finished" podID="ed46bf63-2f4f-4408-b5d3-7208ecffe89c" containerID="007371f4cc300643424af68d1c730962592c3320b0854f9eacc02ee2197b584e" exitCode=0 Oct 31 00:20:03.858043 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:20:03.857583 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7fxtx" event={"ID":"ed46bf63-2f4f-4408-b5d3-7208ecffe89c","Type":"ContainerDied","Data":"007371f4cc300643424af68d1c730962592c3320b0854f9eacc02ee2197b584e"} Oct 31 00:20:05.868398 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:20:05.868327 5222 generic.go:356] "Generic (PLEG): container finished" podID="b961ce20-bb4c-4916-8f7a-6762c24e7014" containerID="aeb5d867cce824ca0939b7e713898e35e5888d323dcfc62d6c302b9015b73cd8" exitCode=0 Oct 31 00:20:05.868869 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:20:05.868434 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449gjvm" event={"ID":"b961ce20-bb4c-4916-8f7a-6762c24e7014","Type":"ContainerDied","Data":"aeb5d867cce824ca0939b7e713898e35e5888d323dcfc62d6c302b9015b73cd8"} Oct 31 00:20:05.895396 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:20:05.894407 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 00:20:07.680742 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:20:07.680667 5222 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h9hsf_0f8c3c03-a1fc-4376-98c6-feeeefa08068/ovn-acl-logging/0.log" Oct 31 00:20:07.681213 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:20:07.681196 5222 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h9hsf_0f8c3c03-a1fc-4376-98c6-feeeefa08068/ovn-acl-logging/0.log" Oct 31 00:20:07.878900 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:20:07.878866 5222 generic.go:356] "Generic (PLEG): container finished" podID="ed46bf63-2f4f-4408-b5d3-7208ecffe89c" containerID="a21502bf9cf8b53a9b417f1e042b9111af3f50f6ad82c07efded1feeb06eb3a4" exitCode=0 Oct 31 00:20:07.879068 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:20:07.878976 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7fxtx" event={"ID":"ed46bf63-2f4f-4408-b5d3-7208ecffe89c","Type":"ContainerDied","Data":"a21502bf9cf8b53a9b417f1e042b9111af3f50f6ad82c07efded1feeb06eb3a4"} Oct 31 00:20:14.913487 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:20:14.913363 5222 generic.go:356] "Generic (PLEG): container finished" podID="b961ce20-bb4c-4916-8f7a-6762c24e7014" containerID="f517e236ea63054675c5fd93a895b3218ad9bc5ab9417e599adc2d8ed098edd9" exitCode=0 Oct 31 00:20:14.913487 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:20:14.913451 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449gjvm" event={"ID":"b961ce20-bb4c-4916-8f7a-6762c24e7014","Type":"ContainerDied","Data":"f517e236ea63054675c5fd93a895b3218ad9bc5ab9417e599adc2d8ed098edd9"} Oct 31 00:20:15.919473 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:20:15.919430 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7fxtx" event={"ID":"ed46bf63-2f4f-4408-b5d3-7208ecffe89c","Type":"ContainerStarted","Data":"d9b09fe51cef1b9e7bb883fb176e045056acbca35154aab6a79486caacc06a1e"} Oct 31 00:20:16.050792 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:20:16.050748 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7fxtx" podStartSLOduration=2.997044104 podStartE2EDuration="25.050733311s" totalImagesPullingTime="22.053689207s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:19:51 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:20:16.050584291 +0000 UTC m=+308.521754042" watchObservedRunningTime="2026-10-31 00:20:16.050733311 +0000 UTC m=+308.521903053" Oct 31 00:20:16.062136 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:20:16.062105 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449gjvm" Oct 31 00:20:16.192040 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:20:16.191947 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b961ce20-bb4c-4916-8f7a-6762c24e7014-util\" (UniqueName: \"kubernetes.io/empty-dir/b961ce20-bb4c-4916-8f7a-6762c24e7014-util\") pod \"b961ce20-bb4c-4916-8f7a-6762c24e7014\" (UID: \"b961ce20-bb4c-4916-8f7a-6762c24e7014\") " Oct 31 00:20:16.192040 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:20:16.192024 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b961ce20-bb4c-4916-8f7a-6762c24e7014-bundle\" (UniqueName: \"kubernetes.io/empty-dir/b961ce20-bb4c-4916-8f7a-6762c24e7014-bundle\") pod \"b961ce20-bb4c-4916-8f7a-6762c24e7014\" (UID: \"b961ce20-bb4c-4916-8f7a-6762c24e7014\") " Oct 31 00:20:16.192288 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:20:16.192077 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b961ce20-bb4c-4916-8f7a-6762c24e7014-kube-api-access-jwm69\" (UniqueName: \"kubernetes.io/projected/b961ce20-bb4c-4916-8f7a-6762c24e7014-kube-api-access-jwm69\") pod \"b961ce20-bb4c-4916-8f7a-6762c24e7014\" (UID: \"b961ce20-bb4c-4916-8f7a-6762c24e7014\") " Oct 31 00:20:16.192461 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:20:16.192435 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b961ce20-bb4c-4916-8f7a-6762c24e7014-bundle" pod "b961ce20-bb4c-4916-8f7a-6762c24e7014" (UID: "b961ce20-bb4c-4916-8f7a-6762c24e7014"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:20:16.194526 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:20:16.194500 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b961ce20-bb4c-4916-8f7a-6762c24e7014-kube-api-access-jwm69" pod "b961ce20-bb4c-4916-8f7a-6762c24e7014" (UID: "b961ce20-bb4c-4916-8f7a-6762c24e7014"). InnerVolumeSpecName "kube-api-access-jwm69". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:20:16.197273 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:20:16.197247 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b961ce20-bb4c-4916-8f7a-6762c24e7014-util" pod "b961ce20-bb4c-4916-8f7a-6762c24e7014" (UID: "b961ce20-bb4c-4916-8f7a-6762c24e7014"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:20:16.293108 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:20:16.293068 5222 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b961ce20-bb4c-4916-8f7a-6762c24e7014-util\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:20:16.293108 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:20:16.293101 5222 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b961ce20-bb4c-4916-8f7a-6762c24e7014-bundle\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:20:16.293108 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:20:16.293115 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwm69\" (UniqueName: \"kubernetes.io/projected/b961ce20-bb4c-4916-8f7a-6762c24e7014-kube-api-access-jwm69\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:20:16.928178 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:20:16.928142 5222 generic.go:356] "Generic (PLEG): container finished" podID="ed46bf63-2f4f-4408-b5d3-7208ecffe89c" containerID="d9b09fe51cef1b9e7bb883fb176e045056acbca35154aab6a79486caacc06a1e" exitCode=0 Oct 31 00:20:16.929060 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:20:16.928243 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7fxtx" event={"ID":"ed46bf63-2f4f-4408-b5d3-7208ecffe89c","Type":"ContainerDied","Data":"d9b09fe51cef1b9e7bb883fb176e045056acbca35154aab6a79486caacc06a1e"} Oct 31 00:20:16.930745 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:20:16.930721 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449gjvm" Oct 31 00:20:16.930876 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:20:16.930718 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449gjvm" event={"ID":"b961ce20-bb4c-4916-8f7a-6762c24e7014","Type":"ContainerDied","Data":"73932c54fc569d5d5fbf55c83308b133c0948ea5b0cf9197dccdeaa4384f3795"} Oct 31 00:20:16.930876 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:20:16.930835 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="73932c54fc569d5d5fbf55c83308b133c0948ea5b0cf9197dccdeaa4384f3795" Oct 31 00:20:18.078624 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:20:18.078592 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7fxtx" Oct 31 00:20:18.209661 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:20:18.209580 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed46bf63-2f4f-4408-b5d3-7208ecffe89c-bundle\" (UniqueName: \"kubernetes.io/empty-dir/ed46bf63-2f4f-4408-b5d3-7208ecffe89c-bundle\") pod \"ed46bf63-2f4f-4408-b5d3-7208ecffe89c\" (UID: \"ed46bf63-2f4f-4408-b5d3-7208ecffe89c\") " Oct 31 00:20:18.209828 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:20:18.209662 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed46bf63-2f4f-4408-b5d3-7208ecffe89c-util\" (UniqueName: \"kubernetes.io/empty-dir/ed46bf63-2f4f-4408-b5d3-7208ecffe89c-util\") pod \"ed46bf63-2f4f-4408-b5d3-7208ecffe89c\" (UID: \"ed46bf63-2f4f-4408-b5d3-7208ecffe89c\") " Oct 31 00:20:18.209828 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:20:18.209729 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed46bf63-2f4f-4408-b5d3-7208ecffe89c-kube-api-access-dnznl\" (UniqueName: \"kubernetes.io/projected/ed46bf63-2f4f-4408-b5d3-7208ecffe89c-kube-api-access-dnznl\") pod \"ed46bf63-2f4f-4408-b5d3-7208ecffe89c\" (UID: \"ed46bf63-2f4f-4408-b5d3-7208ecffe89c\") " Oct 31 00:20:18.210280 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:20:18.210248 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed46bf63-2f4f-4408-b5d3-7208ecffe89c-bundle" pod "ed46bf63-2f4f-4408-b5d3-7208ecffe89c" (UID: "ed46bf63-2f4f-4408-b5d3-7208ecffe89c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:20:18.212082 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:20:18.212048 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed46bf63-2f4f-4408-b5d3-7208ecffe89c-kube-api-access-dnznl" pod "ed46bf63-2f4f-4408-b5d3-7208ecffe89c" (UID: "ed46bf63-2f4f-4408-b5d3-7208ecffe89c"). InnerVolumeSpecName "kube-api-access-dnznl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:20:18.216763 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:20:18.216727 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed46bf63-2f4f-4408-b5d3-7208ecffe89c-util" pod "ed46bf63-2f4f-4408-b5d3-7208ecffe89c" (UID: "ed46bf63-2f4f-4408-b5d3-7208ecffe89c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:20:18.310759 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:20:18.310713 5222 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ed46bf63-2f4f-4408-b5d3-7208ecffe89c-bundle\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:20:18.310759 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:20:18.310750 5222 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ed46bf63-2f4f-4408-b5d3-7208ecffe89c-util\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:20:18.310759 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:20:18.310763 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnznl\" (UniqueName: \"kubernetes.io/projected/ed46bf63-2f4f-4408-b5d3-7208ecffe89c-kube-api-access-dnznl\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:20:18.939689 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:20:18.939659 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7fxtx" Oct 31 00:20:18.939864 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:20:18.939662 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7fxtx" event={"ID":"ed46bf63-2f4f-4408-b5d3-7208ecffe89c","Type":"ContainerDied","Data":"ffc28d1a6b5b2ff3e78f28a2eaa5d2224f452c0bacbaa38b0061b80df952031f"} Oct 31 00:20:18.939864 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:20:18.939773 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ffc28d1a6b5b2ff3e78f28a2eaa5d2224f452c0bacbaa38b0061b80df952031f" Oct 31 00:20:27.433291 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:20:27.433248 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7fbb8789b-nj5rx"] Oct 31 00:20:27.433866 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:20:27.433691 5222 cpu_manager.go:414] "containerMap: removing container" podUID="ed46bf63-2f4f-4408-b5d3-7208ecffe89c" containerName="pull" Oct 31 00:20:27.433866 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:20:27.433708 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed46bf63-2f4f-4408-b5d3-7208ecffe89c" containerName="pull" Oct 31 00:20:27.433866 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:20:27.433730 5222 cpu_manager.go:414] "containerMap: removing container" podUID="b961ce20-bb4c-4916-8f7a-6762c24e7014" containerName="pull" Oct 31 00:20:27.433866 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:20:27.433737 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b961ce20-bb4c-4916-8f7a-6762c24e7014" containerName="pull" Oct 31 00:20:27.433866 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:20:27.433745 5222 cpu_manager.go:414] "containerMap: removing container" podUID="b961ce20-bb4c-4916-8f7a-6762c24e7014" containerName="extract" Oct 31 00:20:27.433866 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:20:27.433752 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b961ce20-bb4c-4916-8f7a-6762c24e7014" containerName="extract" Oct 31 00:20:27.433866 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:20:27.433764 5222 cpu_manager.go:414] "containerMap: removing container" podUID="ed46bf63-2f4f-4408-b5d3-7208ecffe89c" containerName="extract" Oct 31 00:20:27.433866 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:20:27.433771 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed46bf63-2f4f-4408-b5d3-7208ecffe89c" containerName="extract" Oct 31 00:20:27.433866 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:20:27.433800 5222 cpu_manager.go:414] "containerMap: removing container" podUID="b961ce20-bb4c-4916-8f7a-6762c24e7014" containerName="util" Oct 31 00:20:27.433866 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:20:27.433806 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b961ce20-bb4c-4916-8f7a-6762c24e7014" containerName="util" Oct 31 00:20:27.433866 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:20:27.433831 5222 cpu_manager.go:414] "containerMap: removing container" podUID="ed46bf63-2f4f-4408-b5d3-7208ecffe89c" containerName="util" Oct 31 00:20:27.433866 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:20:27.433837 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed46bf63-2f4f-4408-b5d3-7208ecffe89c" containerName="util" Oct 31 00:20:27.434457 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:20:27.433902 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="b961ce20-bb4c-4916-8f7a-6762c24e7014" containerName="extract" Oct 31 00:20:27.434457 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:20:27.433913 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed46bf63-2f4f-4408-b5d3-7208ecffe89c" containerName="extract" Oct 31 00:20:27.436145 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:20:27.436124 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7fbb8789b-nj5rx" Oct 31 00:20:27.492540 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:20:27.492492 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 31 00:20:27.544803 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:20:27.544767 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7fbb8789b-nj5rx"] Oct 31 00:20:27.581580 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:20:27.581544 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 31 00:20:27.581580 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:20:27.581565 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 31 00:20:27.581837 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:20:27.581732 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 31 00:20:27.581837 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:20:27.581812 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-vk4k4\"" Oct 31 00:20:27.594578 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:20:27.594543 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 31 00:20:27.595313 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:20:27.595297 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/26883497-4872-400a-9f25-e9633d9ba15b-webhook-cert\") pod \"opendatahub-operator-controller-manager-7fbb8789b-nj5rx\" (UID: \"26883497-4872-400a-9f25-e9633d9ba15b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7fbb8789b-nj5rx" Oct 31 00:20:27.595416 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:20:27.595327 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/26883497-4872-400a-9f25-e9633d9ba15b-manifests\") pod \"opendatahub-operator-controller-manager-7fbb8789b-nj5rx\" (UID: \"26883497-4872-400a-9f25-e9633d9ba15b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7fbb8789b-nj5rx" Oct 31 00:20:27.595416 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:20:27.595355 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/26883497-4872-400a-9f25-e9633d9ba15b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7fbb8789b-nj5rx\" (UID: \"26883497-4872-400a-9f25-e9633d9ba15b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7fbb8789b-nj5rx" Oct 31 00:20:27.595416 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:20:27.595403 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/26883497-4872-400a-9f25-e9633d9ba15b-tmp\") pod \"opendatahub-operator-controller-manager-7fbb8789b-nj5rx\" (UID: \"26883497-4872-400a-9f25-e9633d9ba15b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7fbb8789b-nj5rx" Oct 31 00:20:27.595566 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:20:27.595542 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/26883497-4872-400a-9f25-e9633d9ba15b-metrics-cert\") pod \"opendatahub-operator-controller-manager-7fbb8789b-nj5rx\" (UID: \"26883497-4872-400a-9f25-e9633d9ba15b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7fbb8789b-nj5rx" Oct 31 00:20:27.595628 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:20:27.595611 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrqtf\" (UniqueName: \"kubernetes.io/projected/26883497-4872-400a-9f25-e9633d9ba15b-kube-api-access-lrqtf\") pod \"opendatahub-operator-controller-manager-7fbb8789b-nj5rx\" (UID: \"26883497-4872-400a-9f25-e9633d9ba15b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7fbb8789b-nj5rx" Oct 31 00:20:27.696480 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:20:27.696364 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/26883497-4872-400a-9f25-e9633d9ba15b-webhook-cert\") pod \"opendatahub-operator-controller-manager-7fbb8789b-nj5rx\" (UID: \"26883497-4872-400a-9f25-e9633d9ba15b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7fbb8789b-nj5rx" Oct 31 00:20:27.696480 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:20:27.696437 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/26883497-4872-400a-9f25-e9633d9ba15b-manifests\") pod \"opendatahub-operator-controller-manager-7fbb8789b-nj5rx\" (UID: \"26883497-4872-400a-9f25-e9633d9ba15b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7fbb8789b-nj5rx" Oct 31 00:20:27.696480 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:20:27.696472 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/26883497-4872-400a-9f25-e9633d9ba15b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7fbb8789b-nj5rx\" (UID: \"26883497-4872-400a-9f25-e9633d9ba15b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7fbb8789b-nj5rx" Oct 31 00:20:27.696722 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:20:27.696497 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/26883497-4872-400a-9f25-e9633d9ba15b-tmp\") pod \"opendatahub-operator-controller-manager-7fbb8789b-nj5rx\" (UID: \"26883497-4872-400a-9f25-e9633d9ba15b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7fbb8789b-nj5rx" Oct 31 00:20:27.696722 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:20:27.696521 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/26883497-4872-400a-9f25-e9633d9ba15b-metrics-cert\") pod \"opendatahub-operator-controller-manager-7fbb8789b-nj5rx\" (UID: \"26883497-4872-400a-9f25-e9633d9ba15b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7fbb8789b-nj5rx" Oct 31 00:20:27.696722 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:20:27.696691 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrqtf\" (UniqueName: \"kubernetes.io/projected/26883497-4872-400a-9f25-e9633d9ba15b-kube-api-access-lrqtf\") pod \"opendatahub-operator-controller-manager-7fbb8789b-nj5rx\" (UID: \"26883497-4872-400a-9f25-e9633d9ba15b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7fbb8789b-nj5rx" Oct 31 00:20:27.696861 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:20:27.696827 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/26883497-4872-400a-9f25-e9633d9ba15b-manifests\") pod \"opendatahub-operator-controller-manager-7fbb8789b-nj5rx\" (UID: \"26883497-4872-400a-9f25-e9633d9ba15b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7fbb8789b-nj5rx" Oct 31 00:20:27.699234 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:20:27.699202 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/26883497-4872-400a-9f25-e9633d9ba15b-tmp\") pod \"opendatahub-operator-controller-manager-7fbb8789b-nj5rx\" (UID: \"26883497-4872-400a-9f25-e9633d9ba15b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7fbb8789b-nj5rx" Oct 31 00:20:27.699432 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:20:27.699398 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/26883497-4872-400a-9f25-e9633d9ba15b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7fbb8789b-nj5rx\" (UID: \"26883497-4872-400a-9f25-e9633d9ba15b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7fbb8789b-nj5rx" Oct 31 00:20:27.699512 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:20:27.699448 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/26883497-4872-400a-9f25-e9633d9ba15b-metrics-cert\") pod \"opendatahub-operator-controller-manager-7fbb8789b-nj5rx\" (UID: \"26883497-4872-400a-9f25-e9633d9ba15b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7fbb8789b-nj5rx" Oct 31 00:20:27.699512 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:20:27.699456 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/26883497-4872-400a-9f25-e9633d9ba15b-webhook-cert\") pod \"opendatahub-operator-controller-manager-7fbb8789b-nj5rx\" (UID: \"26883497-4872-400a-9f25-e9633d9ba15b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7fbb8789b-nj5rx" Oct 31 00:20:27.759441 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:20:27.759400 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrqtf\" (UniqueName: \"kubernetes.io/projected/26883497-4872-400a-9f25-e9633d9ba15b-kube-api-access-lrqtf\") pod \"opendatahub-operator-controller-manager-7fbb8789b-nj5rx\" (UID: \"26883497-4872-400a-9f25-e9633d9ba15b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7fbb8789b-nj5rx" Oct 31 00:20:28.051231 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:20:28.051186 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7fbb8789b-nj5rx" Oct 31 00:20:28.627616 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:20:28.627583 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26883497_4872_400a_9f25_e9633d9ba15b.slice/crio-85a43d35e35fcc6c3d0a986e742760c1de0d9f13c9e32cfb1aaf3845009e1c9e WatchSource:0}: Error finding container 85a43d35e35fcc6c3d0a986e742760c1de0d9f13c9e32cfb1aaf3845009e1c9e: Status 404 returned error can't find the container with id 85a43d35e35fcc6c3d0a986e742760c1de0d9f13c9e32cfb1aaf3845009e1c9e Oct 31 00:20:28.628817 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:20:28.628791 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7fbb8789b-nj5rx"] Oct 31 00:20:28.629206 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:20:28.629188 5222 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 00:20:28.979076 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:20:28.978989 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7fbb8789b-nj5rx" event={"ID":"26883497-4872-400a-9f25-e9633d9ba15b","Type":"ContainerStarted","Data":"85a43d35e35fcc6c3d0a986e742760c1de0d9f13c9e32cfb1aaf3845009e1c9e"} Oct 31 00:20:33.000434 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:20:33.000309 5222 generic.go:356] "Generic (PLEG): container finished" podID="26883497-4872-400a-9f25-e9633d9ba15b" containerID="8044a90fc0b1f52cd0f96f7065c88dfccff35f1012cf73082ce5a6d5a98ae993" exitCode=0 Oct 31 00:20:33.000434 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:20:33.000402 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7fbb8789b-nj5rx" event={"ID":"26883497-4872-400a-9f25-e9633d9ba15b","Type":"ContainerDied","Data":"8044a90fc0b1f52cd0f96f7065c88dfccff35f1012cf73082ce5a6d5a98ae993"} Oct 31 00:20:35.010072 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:20:35.010033 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7fbb8789b-nj5rx" event={"ID":"26883497-4872-400a-9f25-e9633d9ba15b","Type":"ContainerStarted","Data":"38ca2333c5e72dcd53506d2c9016020d27daf35c987320c1dca767afa50327ba"} Oct 31 00:20:35.010527 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:20:35.010153 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7fbb8789b-nj5rx" Oct 31 00:20:35.305598 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:20:35.305541 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7fbb8789b-nj5rx" podStartSLOduration=3.355659616 podStartE2EDuration="8.305525979s" totalImagesPullingTime="4.480745679s" totalInitContainerRuntime="469.120684ms" isStatefulPod=false podCreationTimestamp="2026-10-31 00:20:27 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:20:35.253679288 +0000 UTC m=+327.724849037" watchObservedRunningTime="2026-10-31 00:20:35.305525979 +0000 UTC m=+327.776695725" Oct 31 00:20:35.889224 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:20:35.889189 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 00:20:46.016605 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:20:46.016577 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7fbb8789b-nj5rx" Oct 31 00:21:05.889931 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:21:05.889891 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 00:21:06.014153 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:21:06.014116 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-66d4777458-wxv4x"] Oct 31 00:21:06.019871 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:21:06.019844 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-66d4777458-wxv4x" Oct 31 00:21:06.079151 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:21:06.079119 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 31 00:21:06.129650 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:21:06.129619 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 31 00:21:06.129650 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:21:06.129646 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-r4rfq\"" Oct 31 00:21:06.130717 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:21:06.130698 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-66d4777458-wxv4x"] Oct 31 00:21:06.153299 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:21:06.153212 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a3d36c34-a283-4b19-8ac9-754fe86ff7d3-tmp\") pod \"kube-auth-proxy-66d4777458-wxv4x\" (UID: \"a3d36c34-a283-4b19-8ac9-754fe86ff7d3\") " pod="openshift-ingress/kube-auth-proxy-66d4777458-wxv4x" Oct 31 00:21:06.153299 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:21:06.153257 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzrpb\" (UniqueName: \"kubernetes.io/projected/a3d36c34-a283-4b19-8ac9-754fe86ff7d3-kube-api-access-rzrpb\") pod \"kube-auth-proxy-66d4777458-wxv4x\" (UID: \"a3d36c34-a283-4b19-8ac9-754fe86ff7d3\") " pod="openshift-ingress/kube-auth-proxy-66d4777458-wxv4x" Oct 31 00:21:06.153513 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:21:06.153412 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3d36c34-a283-4b19-8ac9-754fe86ff7d3-tls-certs\") pod \"kube-auth-proxy-66d4777458-wxv4x\" (UID: \"a3d36c34-a283-4b19-8ac9-754fe86ff7d3\") " pod="openshift-ingress/kube-auth-proxy-66d4777458-wxv4x" Oct 31 00:21:06.253859 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:21:06.253821 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a3d36c34-a283-4b19-8ac9-754fe86ff7d3-tmp\") pod \"kube-auth-proxy-66d4777458-wxv4x\" (UID: \"a3d36c34-a283-4b19-8ac9-754fe86ff7d3\") " pod="openshift-ingress/kube-auth-proxy-66d4777458-wxv4x" Oct 31 00:21:06.253859 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:21:06.253866 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzrpb\" (UniqueName: \"kubernetes.io/projected/a3d36c34-a283-4b19-8ac9-754fe86ff7d3-kube-api-access-rzrpb\") pod \"kube-auth-proxy-66d4777458-wxv4x\" (UID: \"a3d36c34-a283-4b19-8ac9-754fe86ff7d3\") " pod="openshift-ingress/kube-auth-proxy-66d4777458-wxv4x" Oct 31 00:21:06.254132 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:21:06.253912 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3d36c34-a283-4b19-8ac9-754fe86ff7d3-tls-certs\") pod \"kube-auth-proxy-66d4777458-wxv4x\" (UID: \"a3d36c34-a283-4b19-8ac9-754fe86ff7d3\") " pod="openshift-ingress/kube-auth-proxy-66d4777458-wxv4x" Oct 31 00:21:06.256424 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:21:06.256385 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a3d36c34-a283-4b19-8ac9-754fe86ff7d3-tmp\") pod \"kube-auth-proxy-66d4777458-wxv4x\" (UID: \"a3d36c34-a283-4b19-8ac9-754fe86ff7d3\") " pod="openshift-ingress/kube-auth-proxy-66d4777458-wxv4x" Oct 31 00:21:06.256722 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:21:06.256700 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3d36c34-a283-4b19-8ac9-754fe86ff7d3-tls-certs\") pod \"kube-auth-proxy-66d4777458-wxv4x\" (UID: \"a3d36c34-a283-4b19-8ac9-754fe86ff7d3\") " pod="openshift-ingress/kube-auth-proxy-66d4777458-wxv4x" Oct 31 00:21:06.322055 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:21:06.322012 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzrpb\" (UniqueName: \"kubernetes.io/projected/a3d36c34-a283-4b19-8ac9-754fe86ff7d3-kube-api-access-rzrpb\") pod \"kube-auth-proxy-66d4777458-wxv4x\" (UID: \"a3d36c34-a283-4b19-8ac9-754fe86ff7d3\") " pod="openshift-ingress/kube-auth-proxy-66d4777458-wxv4x" Oct 31 00:21:06.330747 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:21:06.330718 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-66d4777458-wxv4x" Oct 31 00:21:06.706916 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:21:06.706889 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3d36c34_a283_4b19_8ac9_754fe86ff7d3.slice/crio-e9f3a873e7c05877623ca0a22859e0d505928dddb93910bf4cf000d3ee1dd174 WatchSource:0}: Error finding container e9f3a873e7c05877623ca0a22859e0d505928dddb93910bf4cf000d3ee1dd174: Status 404 returned error can't find the container with id e9f3a873e7c05877623ca0a22859e0d505928dddb93910bf4cf000d3ee1dd174 Oct 31 00:21:06.707748 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:21:06.707726 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-66d4777458-wxv4x"] Oct 31 00:21:07.145196 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:21:07.145164 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-66d4777458-wxv4x" event={"ID":"a3d36c34-a283-4b19-8ac9-754fe86ff7d3","Type":"ContainerStarted","Data":"e9f3a873e7c05877623ca0a22859e0d505928dddb93910bf4cf000d3ee1dd174"} Oct 31 00:21:10.382044 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:21:10.382006 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-79tk8"] Oct 31 00:21:10.384618 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:21:10.384592 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-79tk8" Oct 31 00:21:10.441733 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:21:10.441635 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-79tk8"] Oct 31 00:21:10.443394 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:21:10.443342 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Oct 31 00:21:10.443641 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:21:10.443620 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 31 00:21:10.489606 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:21:10.489574 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-r5wr4\"" Oct 31 00:21:10.489606 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:21:10.489574 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 31 00:21:10.496072 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:21:10.496031 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4x9f6\" (UniqueName: \"kubernetes.io/projected/66c5518e-c6af-4298-aa66-5e1b6d6bc645-kube-api-access-4x9f6\") pod \"limitador-limitador-7c9b87d574-79tk8\" (UID: \"66c5518e-c6af-4298-aa66-5e1b6d6bc645\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-79tk8" Oct 31 00:21:10.496263 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:21:10.496083 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/66c5518e-c6af-4298-aa66-5e1b6d6bc645-config-file\") pod \"limitador-limitador-7c9b87d574-79tk8\" (UID: \"66c5518e-c6af-4298-aa66-5e1b6d6bc645\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-79tk8" Oct 31 00:21:10.597575 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:21:10.597534 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4x9f6\" (UniqueName: \"kubernetes.io/projected/66c5518e-c6af-4298-aa66-5e1b6d6bc645-kube-api-access-4x9f6\") pod \"limitador-limitador-7c9b87d574-79tk8\" (UID: \"66c5518e-c6af-4298-aa66-5e1b6d6bc645\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-79tk8" Oct 31 00:21:10.597763 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:21:10.597588 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/66c5518e-c6af-4298-aa66-5e1b6d6bc645-config-file\") pod \"limitador-limitador-7c9b87d574-79tk8\" (UID: \"66c5518e-c6af-4298-aa66-5e1b6d6bc645\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-79tk8" Oct 31 00:21:10.598435 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:21:10.598413 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/66c5518e-c6af-4298-aa66-5e1b6d6bc645-config-file\") pod \"limitador-limitador-7c9b87d574-79tk8\" (UID: \"66c5518e-c6af-4298-aa66-5e1b6d6bc645\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-79tk8" Oct 31 00:21:10.660660 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:21:10.660569 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4x9f6\" (UniqueName: \"kubernetes.io/projected/66c5518e-c6af-4298-aa66-5e1b6d6bc645-kube-api-access-4x9f6\") pod \"limitador-limitador-7c9b87d574-79tk8\" (UID: \"66c5518e-c6af-4298-aa66-5e1b6d6bc645\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-79tk8" Oct 31 00:21:10.701558 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:21:10.701526 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-79tk8" Oct 31 00:21:10.740785 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:21:10.740702 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-79tk8"] Oct 31 00:21:11.083642 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:21:11.083613 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-79tk8"] Oct 31 00:21:11.165097 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:21:11.165056 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-79tk8" event={"ID":"66c5518e-c6af-4298-aa66-5e1b6d6bc645","Type":"ContainerStarted","Data":"202fd06b72e2de6c2a94a2c672b7c5254a9a30c5a3a64b3f2542eb2048cc2a95"} Oct 31 00:21:11.670142 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:21:11.670114 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 31 00:21:12.170891 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:21:12.170855 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-66d4777458-wxv4x" event={"ID":"a3d36c34-a283-4b19-8ac9-754fe86ff7d3","Type":"ContainerStarted","Data":"4402b79b0058e6191de10abd5db4270d68d92958a0515611f591a9310510aecf"} Oct 31 00:21:12.291865 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:21:12.291809 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-66d4777458-wxv4x" podStartSLOduration=2.384147944 podStartE2EDuration="7.291792558s" totalImagesPullingTime="4.907644614s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:21:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:21:12.29062858 +0000 UTC m=+364.761798329" watchObservedRunningTime="2026-10-31 00:21:12.291792558 +0000 UTC m=+364.762962326" Oct 31 00:21:15.185408 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:21:15.185351 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-79tk8" event={"ID":"66c5518e-c6af-4298-aa66-5e1b6d6bc645","Type":"ContainerStarted","Data":"0aba3a3e5b0ac00c23c51d5e5cb74a77fbd8557e07b4542918ed49ebfbcefde3"} Oct 31 00:21:15.185798 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:21:15.185470 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-79tk8" Oct 31 00:21:15.317171 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:21:15.317121 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-79tk8" podStartSLOduration=1.989792265 podStartE2EDuration="5.317105537s" totalImagesPullingTime="3.327313272s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:21:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:21:15.316474457 +0000 UTC m=+367.787644207" watchObservedRunningTime="2026-10-31 00:21:15.317105537 +0000 UTC m=+367.788275286" Oct 31 00:21:26.190766 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:21:26.190739 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-79tk8" Oct 31 00:21:35.889715 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:21:35.889667 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 00:22:05.889974 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:22:05.889939 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 00:22:35.890211 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:22:35.890173 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 00:22:36.326318 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:22:36.326283 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8d22z"] Oct 31 00:22:36.329659 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:22:36.329641 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8d22z" Oct 31 00:22:36.435798 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:22:36.435768 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7f7ac90-6d52-40a2-873c-b878687b02ca-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8d22z\" (UID: \"e7f7ac90-6d52-40a2-873c-b878687b02ca\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8d22z" Oct 31 00:22:36.435994 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:22:36.435847 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7f7ac90-6d52-40a2-873c-b878687b02ca-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8d22z\" (UID: \"e7f7ac90-6d52-40a2-873c-b878687b02ca\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8d22z" Oct 31 00:22:36.435994 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:22:36.435876 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e7f7ac90-6d52-40a2-873c-b878687b02ca-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8d22z\" (UID: \"e7f7ac90-6d52-40a2-873c-b878687b02ca\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8d22z" Oct 31 00:22:36.435994 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:22:36.435909 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e7f7ac90-6d52-40a2-873c-b878687b02ca-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8d22z\" (UID: \"e7f7ac90-6d52-40a2-873c-b878687b02ca\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8d22z" Oct 31 00:22:36.436124 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:22:36.436089 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sthq6\" (UniqueName: \"kubernetes.io/projected/e7f7ac90-6d52-40a2-873c-b878687b02ca-kube-api-access-sthq6\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8d22z\" (UID: \"e7f7ac90-6d52-40a2-873c-b878687b02ca\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8d22z" Oct 31 00:22:36.436171 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:22:36.436134 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e7f7ac90-6d52-40a2-873c-b878687b02ca-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8d22z\" (UID: \"e7f7ac90-6d52-40a2-873c-b878687b02ca\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8d22z" Oct 31 00:22:36.436262 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:22:36.436239 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e7f7ac90-6d52-40a2-873c-b878687b02ca-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8d22z\" (UID: \"e7f7ac90-6d52-40a2-873c-b878687b02ca\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8d22z" Oct 31 00:22:36.439597 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:22:36.439569 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8d22z"] Oct 31 00:22:36.537543 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:22:36.537510 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e7f7ac90-6d52-40a2-873c-b878687b02ca-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8d22z\" (UID: \"e7f7ac90-6d52-40a2-873c-b878687b02ca\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8d22z" Oct 31 00:22:36.537753 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:22:36.537552 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7f7ac90-6d52-40a2-873c-b878687b02ca-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8d22z\" (UID: \"e7f7ac90-6d52-40a2-873c-b878687b02ca\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8d22z" Oct 31 00:22:36.537753 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:22:36.537596 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7f7ac90-6d52-40a2-873c-b878687b02ca-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8d22z\" (UID: \"e7f7ac90-6d52-40a2-873c-b878687b02ca\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8d22z" Oct 31 00:22:36.537753 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:22:36.537624 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e7f7ac90-6d52-40a2-873c-b878687b02ca-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8d22z\" (UID: \"e7f7ac90-6d52-40a2-873c-b878687b02ca\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8d22z" Oct 31 00:22:36.537753 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:22:36.537654 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e7f7ac90-6d52-40a2-873c-b878687b02ca-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8d22z\" (UID: \"e7f7ac90-6d52-40a2-873c-b878687b02ca\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8d22z" Oct 31 00:22:36.537753 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:22:36.537691 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sthq6\" (UniqueName: \"kubernetes.io/projected/e7f7ac90-6d52-40a2-873c-b878687b02ca-kube-api-access-sthq6\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8d22z\" (UID: \"e7f7ac90-6d52-40a2-873c-b878687b02ca\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8d22z" Oct 31 00:22:36.537753 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:22:36.537717 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e7f7ac90-6d52-40a2-873c-b878687b02ca-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8d22z\" (UID: \"e7f7ac90-6d52-40a2-873c-b878687b02ca\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8d22z" Oct 31 00:22:36.538508 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:22:36.538447 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e7f7ac90-6d52-40a2-873c-b878687b02ca-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8d22z\" (UID: \"e7f7ac90-6d52-40a2-873c-b878687b02ca\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8d22z" Oct 31 00:22:36.540214 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:22:36.540187 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e7f7ac90-6d52-40a2-873c-b878687b02ca-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8d22z\" (UID: \"e7f7ac90-6d52-40a2-873c-b878687b02ca\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8d22z" Oct 31 00:22:36.540314 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:22:36.540225 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e7f7ac90-6d52-40a2-873c-b878687b02ca-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8d22z\" (UID: \"e7f7ac90-6d52-40a2-873c-b878687b02ca\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8d22z" Oct 31 00:22:36.541148 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:22:36.541119 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7f7ac90-6d52-40a2-873c-b878687b02ca-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8d22z\" (UID: \"e7f7ac90-6d52-40a2-873c-b878687b02ca\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8d22z" Oct 31 00:22:36.541148 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:22:36.541134 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e7f7ac90-6d52-40a2-873c-b878687b02ca-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8d22z\" (UID: \"e7f7ac90-6d52-40a2-873c-b878687b02ca\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8d22z" Oct 31 00:22:36.601027 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:22:36.600957 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7f7ac90-6d52-40a2-873c-b878687b02ca-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8d22z\" (UID: \"e7f7ac90-6d52-40a2-873c-b878687b02ca\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8d22z" Oct 31 00:22:36.601027 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:22:36.600994 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sthq6\" (UniqueName: \"kubernetes.io/projected/e7f7ac90-6d52-40a2-873c-b878687b02ca-kube-api-access-sthq6\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8d22z\" (UID: \"e7f7ac90-6d52-40a2-873c-b878687b02ca\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8d22z" Oct 31 00:22:36.641302 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:22:36.641262 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8d22z" Oct 31 00:22:37.073202 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:22:37.073175 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7f7ac90_6d52_40a2_873c_b878687b02ca.slice/crio-0051b9aabbe4d0545eb9133a1c686c49814bd1c4f817fcc0caf019e38f468570 WatchSource:0}: Error finding container 0051b9aabbe4d0545eb9133a1c686c49814bd1c4f817fcc0caf019e38f468570: Status 404 returned error can't find the container with id 0051b9aabbe4d0545eb9133a1c686c49814bd1c4f817fcc0caf019e38f468570 Oct 31 00:22:37.073859 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:22:37.073834 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8d22z"] Oct 31 00:22:37.075291 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:22:37.075261 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:22:37.075424 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:22:37.075332 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:22:37.534453 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:22:37.534415 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8d22z" event={"ID":"e7f7ac90-6d52-40a2-873c-b878687b02ca","Type":"ContainerStarted","Data":"e246588a533778abb1c5a4d80202a72f4634d1427a9f3ba373bdd16adfec4c5e"} Oct 31 00:22:37.534656 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:22:37.534464 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8d22z" event={"ID":"e7f7ac90-6d52-40a2-873c-b878687b02ca","Type":"ContainerStarted","Data":"0051b9aabbe4d0545eb9133a1c686c49814bd1c4f817fcc0caf019e38f468570"} Oct 31 00:22:37.534656 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:22:37.534545 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8d22z" Oct 31 00:22:37.659511 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:22:37.659459 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8d22z" podStartSLOduration=1.6594448590000002 podStartE2EDuration="1.659444859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:22:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:22:37.65847931 +0000 UTC m=+450.129649170" watchObservedRunningTime="2026-10-31 00:22:37.659444859 +0000 UTC m=+450.130614607" Oct 31 00:22:38.541032 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:22:38.541003 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8d22z" Oct 31 00:22:38.769057 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:22:38.769023 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-qbvvz"] Oct 31 00:22:38.769290 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:22:38.769249 5222 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-qbvvz" podUID="2f75a02c-5239-4529-a375-bbcee7e5118d" containerName="discovery" containerID="cri-o://ffeb147b2eedd37bed4ea352d05c1cb76963feb2d9f4a93f0b91bf05c84c5c39" gracePeriod=30 Oct 31 00:22:39.019117 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:22:39.019092 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-qbvvz" Oct 31 00:22:39.060313 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:22:39.060285 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f75a02c-5239-4529-a375-bbcee7e5118d-cacerts\" (UniqueName: \"kubernetes.io/secret/2f75a02c-5239-4529-a375-bbcee7e5118d-cacerts\") pod \"2f75a02c-5239-4529-a375-bbcee7e5118d\" (UID: \"2f75a02c-5239-4529-a375-bbcee7e5118d\") " Oct 31 00:22:39.060510 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:22:39.060350 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f75a02c-5239-4529-a375-bbcee7e5118d-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2f75a02c-5239-4529-a375-bbcee7e5118d-istio-csr-ca-configmap\") pod \"2f75a02c-5239-4529-a375-bbcee7e5118d\" (UID: \"2f75a02c-5239-4529-a375-bbcee7e5118d\") " Oct 31 00:22:39.060510 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:22:39.060446 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f75a02c-5239-4529-a375-bbcee7e5118d-kube-api-access-n86sg\" (UniqueName: \"kubernetes.io/projected/2f75a02c-5239-4529-a375-bbcee7e5118d-kube-api-access-n86sg\") pod \"2f75a02c-5239-4529-a375-bbcee7e5118d\" (UID: \"2f75a02c-5239-4529-a375-bbcee7e5118d\") " Oct 31 00:22:39.060582 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:22:39.060529 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f75a02c-5239-4529-a375-bbcee7e5118d-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f75a02c-5239-4529-a375-bbcee7e5118d-local-certs\") pod \"2f75a02c-5239-4529-a375-bbcee7e5118d\" (UID: \"2f75a02c-5239-4529-a375-bbcee7e5118d\") " Oct 31 00:22:39.060582 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:22:39.060559 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f75a02c-5239-4529-a375-bbcee7e5118d-istio-token\" (UniqueName: \"kubernetes.io/projected/2f75a02c-5239-4529-a375-bbcee7e5118d-istio-token\") pod \"2f75a02c-5239-4529-a375-bbcee7e5118d\" (UID: \"2f75a02c-5239-4529-a375-bbcee7e5118d\") " Oct 31 00:22:39.060677 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:22:39.060596 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f75a02c-5239-4529-a375-bbcee7e5118d-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2f75a02c-5239-4529-a375-bbcee7e5118d-istio-kubeconfig\") pod \"2f75a02c-5239-4529-a375-bbcee7e5118d\" (UID: \"2f75a02c-5239-4529-a375-bbcee7e5118d\") " Oct 31 00:22:39.060677 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:22:39.060646 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f75a02c-5239-4529-a375-bbcee7e5118d-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2f75a02c-5239-4529-a375-bbcee7e5118d-istio-csr-dns-cert\") pod \"2f75a02c-5239-4529-a375-bbcee7e5118d\" (UID: \"2f75a02c-5239-4529-a375-bbcee7e5118d\") " Oct 31 00:22:39.061070 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:22:39.060804 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f75a02c-5239-4529-a375-bbcee7e5118d-istio-csr-ca-configmap" pod "2f75a02c-5239-4529-a375-bbcee7e5118d" (UID: "2f75a02c-5239-4529-a375-bbcee7e5118d"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:22:39.061070 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:22:39.060947 5222 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2f75a02c-5239-4529-a375-bbcee7e5118d-istio-csr-ca-configmap\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:22:39.063097 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:22:39.063065 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f75a02c-5239-4529-a375-bbcee7e5118d-istio-csr-dns-cert" pod "2f75a02c-5239-4529-a375-bbcee7e5118d" (UID: "2f75a02c-5239-4529-a375-bbcee7e5118d"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:22:39.063200 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:22:39.063096 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f75a02c-5239-4529-a375-bbcee7e5118d-istio-token" pod "2f75a02c-5239-4529-a375-bbcee7e5118d" (UID: "2f75a02c-5239-4529-a375-bbcee7e5118d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:22:39.063252 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:22:39.063211 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f75a02c-5239-4529-a375-bbcee7e5118d-kube-api-access-n86sg" pod "2f75a02c-5239-4529-a375-bbcee7e5118d" (UID: "2f75a02c-5239-4529-a375-bbcee7e5118d"). InnerVolumeSpecName "kube-api-access-n86sg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:22:39.063288 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:22:39.063271 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f75a02c-5239-4529-a375-bbcee7e5118d-istio-kubeconfig" pod "2f75a02c-5239-4529-a375-bbcee7e5118d" (UID: "2f75a02c-5239-4529-a375-bbcee7e5118d"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:22:39.063321 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:22:39.063286 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f75a02c-5239-4529-a375-bbcee7e5118d-cacerts" pod "2f75a02c-5239-4529-a375-bbcee7e5118d" (UID: "2f75a02c-5239-4529-a375-bbcee7e5118d"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:22:39.063394 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:22:39.063348 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f75a02c-5239-4529-a375-bbcee7e5118d-local-certs" pod "2f75a02c-5239-4529-a375-bbcee7e5118d" (UID: "2f75a02c-5239-4529-a375-bbcee7e5118d"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:22:39.161578 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:22:39.161494 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n86sg\" (UniqueName: \"kubernetes.io/projected/2f75a02c-5239-4529-a375-bbcee7e5118d-kube-api-access-n86sg\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:22:39.161578 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:22:39.161522 5222 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f75a02c-5239-4529-a375-bbcee7e5118d-local-certs\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:22:39.161578 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:22:39.161531 5222 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f75a02c-5239-4529-a375-bbcee7e5118d-istio-token\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:22:39.161578 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:22:39.161540 5222 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2f75a02c-5239-4529-a375-bbcee7e5118d-istio-kubeconfig\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:22:39.161578 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:22:39.161549 5222 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2f75a02c-5239-4529-a375-bbcee7e5118d-istio-csr-dns-cert\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:22:39.161578 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:22:39.161557 5222 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2f75a02c-5239-4529-a375-bbcee7e5118d-cacerts\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:22:39.545680 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:22:39.545641 5222 generic.go:356] "Generic (PLEG): container finished" podID="2f75a02c-5239-4529-a375-bbcee7e5118d" containerID="ffeb147b2eedd37bed4ea352d05c1cb76963feb2d9f4a93f0b91bf05c84c5c39" exitCode=0 Oct 31 00:22:39.546183 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:22:39.545693 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-qbvvz" Oct 31 00:22:39.546183 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:22:39.545733 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-qbvvz" event={"ID":"2f75a02c-5239-4529-a375-bbcee7e5118d","Type":"ContainerDied","Data":"ffeb147b2eedd37bed4ea352d05c1cb76963feb2d9f4a93f0b91bf05c84c5c39"} Oct 31 00:22:39.546183 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:22:39.545781 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-qbvvz" event={"ID":"2f75a02c-5239-4529-a375-bbcee7e5118d","Type":"ContainerDied","Data":"ac6eda2ddc393feceeb17b3671ed760f5213ade97b95e190937ee52aada8db88"} Oct 31 00:22:39.546183 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:22:39.545803 5222 scope.go:122] "RemoveContainer" containerID="ffeb147b2eedd37bed4ea352d05c1cb76963feb2d9f4a93f0b91bf05c84c5c39" Oct 31 00:22:39.554667 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:22:39.554646 5222 scope.go:122] "RemoveContainer" containerID="ffeb147b2eedd37bed4ea352d05c1cb76963feb2d9f4a93f0b91bf05c84c5c39" Oct 31 00:22:39.554948 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:22:39.554930 5222 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffeb147b2eedd37bed4ea352d05c1cb76963feb2d9f4a93f0b91bf05c84c5c39\": container with ID starting with ffeb147b2eedd37bed4ea352d05c1cb76963feb2d9f4a93f0b91bf05c84c5c39 not found: ID does not exist" containerID="ffeb147b2eedd37bed4ea352d05c1cb76963feb2d9f4a93f0b91bf05c84c5c39" Oct 31 00:22:39.555021 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:22:39.554962 5222 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffeb147b2eedd37bed4ea352d05c1cb76963feb2d9f4a93f0b91bf05c84c5c39"} err="failed to get container status \"ffeb147b2eedd37bed4ea352d05c1cb76963feb2d9f4a93f0b91bf05c84c5c39\": rpc error: code = NotFound desc = could not find container \"ffeb147b2eedd37bed4ea352d05c1cb76963feb2d9f4a93f0b91bf05c84c5c39\": container with ID starting with ffeb147b2eedd37bed4ea352d05c1cb76963feb2d9f4a93f0b91bf05c84c5c39 not found: ID does not exist" Oct 31 00:22:39.716338 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:22:39.716298 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-qbvvz"] Oct 31 00:22:39.767501 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:22:39.767465 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-qbvvz"] Oct 31 00:22:41.654741 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:22:41.654710 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f75a02c-5239-4529-a375-bbcee7e5118d" path="/var/lib/kubelet/pods/2f75a02c-5239-4529-a375-bbcee7e5118d/volumes" Oct 31 00:22:41.973638 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:22:41.973549 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-cc757"] Oct 31 00:22:41.973910 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:22:41.973897 5222 cpu_manager.go:414] "containerMap: removing container" podUID="2f75a02c-5239-4529-a375-bbcee7e5118d" containerName="discovery" Oct 31 00:22:41.973951 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:22:41.973911 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f75a02c-5239-4529-a375-bbcee7e5118d" containerName="discovery" Oct 31 00:22:41.973989 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:22:41.973977 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f75a02c-5239-4529-a375-bbcee7e5118d" containerName="discovery" Oct 31 00:22:41.978328 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:22:41.978307 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-cc757" Oct 31 00:22:42.031562 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:22:42.031522 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-cc757"] Oct 31 00:22:42.081140 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:22:42.081112 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 31 00:22:42.081321 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:22:42.081154 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 31 00:22:42.083245 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:22:42.083230 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Oct 31 00:22:42.083303 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:22:42.083270 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-j7g9w\"" Oct 31 00:22:42.083303 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:22:42.083290 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\"" Oct 31 00:22:42.088649 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:22:42.088627 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c31f203d-f391-446f-82c2-76dd9697bd43-cert\") pod \"odh-model-controller-769875db56-cc757\" (UID: \"c31f203d-f391-446f-82c2-76dd9697bd43\") " pod="opendatahub/odh-model-controller-769875db56-cc757" Oct 31 00:22:42.088708 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:22:42.088660 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c31f203d-f391-446f-82c2-76dd9697bd43-metrics-certs\") pod \"odh-model-controller-769875db56-cc757\" (UID: \"c31f203d-f391-446f-82c2-76dd9697bd43\") " pod="opendatahub/odh-model-controller-769875db56-cc757" Oct 31 00:22:42.088708 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:22:42.088679 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlfgl\" (UniqueName: \"kubernetes.io/projected/c31f203d-f391-446f-82c2-76dd9697bd43-kube-api-access-rlfgl\") pod \"odh-model-controller-769875db56-cc757\" (UID: \"c31f203d-f391-446f-82c2-76dd9697bd43\") " pod="opendatahub/odh-model-controller-769875db56-cc757" Oct 31 00:22:42.189974 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:22:42.189924 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c31f203d-f391-446f-82c2-76dd9697bd43-cert\") pod \"odh-model-controller-769875db56-cc757\" (UID: \"c31f203d-f391-446f-82c2-76dd9697bd43\") " pod="opendatahub/odh-model-controller-769875db56-cc757" Oct 31 00:22:42.189974 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:22:42.189979 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c31f203d-f391-446f-82c2-76dd9697bd43-metrics-certs\") pod \"odh-model-controller-769875db56-cc757\" (UID: \"c31f203d-f391-446f-82c2-76dd9697bd43\") " pod="opendatahub/odh-model-controller-769875db56-cc757" Oct 31 00:22:42.190211 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:22:42.190099 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlfgl\" (UniqueName: \"kubernetes.io/projected/c31f203d-f391-446f-82c2-76dd9697bd43-kube-api-access-rlfgl\") pod \"odh-model-controller-769875db56-cc757\" (UID: \"c31f203d-f391-446f-82c2-76dd9697bd43\") " pod="opendatahub/odh-model-controller-769875db56-cc757" Oct 31 00:22:42.192641 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:22:42.192608 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c31f203d-f391-446f-82c2-76dd9697bd43-cert\") pod \"odh-model-controller-769875db56-cc757\" (UID: \"c31f203d-f391-446f-82c2-76dd9697bd43\") " pod="opendatahub/odh-model-controller-769875db56-cc757" Oct 31 00:22:42.192641 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:22:42.192615 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c31f203d-f391-446f-82c2-76dd9697bd43-metrics-certs\") pod \"odh-model-controller-769875db56-cc757\" (UID: \"c31f203d-f391-446f-82c2-76dd9697bd43\") " pod="opendatahub/odh-model-controller-769875db56-cc757" Oct 31 00:22:42.252775 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:22:42.252695 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlfgl\" (UniqueName: \"kubernetes.io/projected/c31f203d-f391-446f-82c2-76dd9697bd43-kube-api-access-rlfgl\") pod \"odh-model-controller-769875db56-cc757\" (UID: \"c31f203d-f391-446f-82c2-76dd9697bd43\") " pod="opendatahub/odh-model-controller-769875db56-cc757" Oct 31 00:22:42.290765 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:22:42.290726 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-cc757" Oct 31 00:22:42.661176 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:22:42.661143 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc31f203d_f391_446f_82c2_76dd9697bd43.slice/crio-8ae8461339aa7fa11fb57fba975c40fefb4e5f7953b1db265f8cf45479a9dcc6 WatchSource:0}: Error finding container 8ae8461339aa7fa11fb57fba975c40fefb4e5f7953b1db265f8cf45479a9dcc6: Status 404 returned error can't find the container with id 8ae8461339aa7fa11fb57fba975c40fefb4e5f7953b1db265f8cf45479a9dcc6 Oct 31 00:22:42.662620 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:22:42.662597 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-cc757"] Oct 31 00:22:43.566699 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:22:43.566666 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-cc757" event={"ID":"c31f203d-f391-446f-82c2-76dd9697bd43","Type":"ContainerStarted","Data":"8ae8461339aa7fa11fb57fba975c40fefb4e5f7953b1db265f8cf45479a9dcc6"} Oct 31 00:22:47.594994 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:22:47.594958 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-cc757" event={"ID":"c31f203d-f391-446f-82c2-76dd9697bd43","Type":"ContainerStarted","Data":"a73e328e7930166d1ffd09f29aa88fb3a316ec74392c22acdb4592c985ea6ee2"} Oct 31 00:22:47.595423 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:22:47.595099 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-cc757" Oct 31 00:22:47.714886 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:22:47.714834 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-cc757" podStartSLOduration=2.37288575 podStartE2EDuration="6.714820742s" totalImagesPullingTime="4.341934992s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:22:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:22:47.713757208 +0000 UTC m=+460.184926957" watchObservedRunningTime="2026-10-31 00:22:47.714820742 +0000 UTC m=+460.185990488" Oct 31 00:22:51.826851 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:22:51.826820 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-79tk8"] Oct 31 00:22:58.601529 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:22:58.601498 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-cc757" Oct 31 00:23:05.889242 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:23:05.889198 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 00:23:21.412331 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:23:21.412288 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-856c7c87c8-lj5gw"] Oct 31 00:23:21.415736 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:23:21.415716 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-856c7c87c8-lj5gw" Oct 31 00:23:21.453724 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:23:21.453682 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/706fb85b-6592-4394-b572-6f8dbe9f2c4d-metrics-certs\") pod \"jobset-controller-manager-856c7c87c8-lj5gw\" (UID: \"706fb85b-6592-4394-b572-6f8dbe9f2c4d\") " pod="openshift-jobset-operator/jobset-controller-manager-856c7c87c8-lj5gw" Oct 31 00:23:21.453724 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:23:21.453721 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/706fb85b-6592-4394-b572-6f8dbe9f2c4d-manager-config\") pod \"jobset-controller-manager-856c7c87c8-lj5gw\" (UID: \"706fb85b-6592-4394-b572-6f8dbe9f2c4d\") " pod="openshift-jobset-operator/jobset-controller-manager-856c7c87c8-lj5gw" Oct 31 00:23:21.454001 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:23:21.453751 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/706fb85b-6592-4394-b572-6f8dbe9f2c4d-cert\") pod \"jobset-controller-manager-856c7c87c8-lj5gw\" (UID: \"706fb85b-6592-4394-b572-6f8dbe9f2c4d\") " pod="openshift-jobset-operator/jobset-controller-manager-856c7c87c8-lj5gw" Oct 31 00:23:21.454001 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:23:21.453841 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqnj9\" (UniqueName: \"kubernetes.io/projected/706fb85b-6592-4394-b572-6f8dbe9f2c4d-kube-api-access-rqnj9\") pod \"jobset-controller-manager-856c7c87c8-lj5gw\" (UID: \"706fb85b-6592-4394-b572-6f8dbe9f2c4d\") " pod="openshift-jobset-operator/jobset-controller-manager-856c7c87c8-lj5gw" Oct 31 00:23:21.469869 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:23:21.469834 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-856c7c87c8-lj5gw"] Oct 31 00:23:21.519857 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:23:21.519820 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Oct 31 00:23:21.519857 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:23:21.519835 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Oct 31 00:23:21.521951 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:23:21.521928 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\"" Oct 31 00:23:21.521951 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:23:21.521947 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\"" Oct 31 00:23:21.522111 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:23:21.521963 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-wqjrj\"" Oct 31 00:23:21.522111 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:23:21.521993 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\"" Oct 31 00:23:21.555356 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:23:21.555311 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqnj9\" (UniqueName: \"kubernetes.io/projected/706fb85b-6592-4394-b572-6f8dbe9f2c4d-kube-api-access-rqnj9\") pod \"jobset-controller-manager-856c7c87c8-lj5gw\" (UID: \"706fb85b-6592-4394-b572-6f8dbe9f2c4d\") " pod="openshift-jobset-operator/jobset-controller-manager-856c7c87c8-lj5gw" Oct 31 00:23:21.555590 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:23:21.555461 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/706fb85b-6592-4394-b572-6f8dbe9f2c4d-metrics-certs\") pod \"jobset-controller-manager-856c7c87c8-lj5gw\" (UID: \"706fb85b-6592-4394-b572-6f8dbe9f2c4d\") " pod="openshift-jobset-operator/jobset-controller-manager-856c7c87c8-lj5gw" Oct 31 00:23:21.555590 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:23:21.555488 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/706fb85b-6592-4394-b572-6f8dbe9f2c4d-manager-config\") pod \"jobset-controller-manager-856c7c87c8-lj5gw\" (UID: \"706fb85b-6592-4394-b572-6f8dbe9f2c4d\") " pod="openshift-jobset-operator/jobset-controller-manager-856c7c87c8-lj5gw" Oct 31 00:23:21.555590 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:23:21.555533 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/706fb85b-6592-4394-b572-6f8dbe9f2c4d-cert\") pod \"jobset-controller-manager-856c7c87c8-lj5gw\" (UID: \"706fb85b-6592-4394-b572-6f8dbe9f2c4d\") " pod="openshift-jobset-operator/jobset-controller-manager-856c7c87c8-lj5gw" Oct 31 00:23:21.556288 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:23:21.556260 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/706fb85b-6592-4394-b572-6f8dbe9f2c4d-manager-config\") pod \"jobset-controller-manager-856c7c87c8-lj5gw\" (UID: \"706fb85b-6592-4394-b572-6f8dbe9f2c4d\") " pod="openshift-jobset-operator/jobset-controller-manager-856c7c87c8-lj5gw" Oct 31 00:23:21.558228 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:23:21.558202 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/706fb85b-6592-4394-b572-6f8dbe9f2c4d-cert\") pod \"jobset-controller-manager-856c7c87c8-lj5gw\" (UID: \"706fb85b-6592-4394-b572-6f8dbe9f2c4d\") " pod="openshift-jobset-operator/jobset-controller-manager-856c7c87c8-lj5gw" Oct 31 00:23:21.558228 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:23:21.558215 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/706fb85b-6592-4394-b572-6f8dbe9f2c4d-metrics-certs\") pod \"jobset-controller-manager-856c7c87c8-lj5gw\" (UID: \"706fb85b-6592-4394-b572-6f8dbe9f2c4d\") " pod="openshift-jobset-operator/jobset-controller-manager-856c7c87c8-lj5gw" Oct 31 00:23:21.618034 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:23:21.617994 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqnj9\" (UniqueName: \"kubernetes.io/projected/706fb85b-6592-4394-b572-6f8dbe9f2c4d-kube-api-access-rqnj9\") pod \"jobset-controller-manager-856c7c87c8-lj5gw\" (UID: \"706fb85b-6592-4394-b572-6f8dbe9f2c4d\") " pod="openshift-jobset-operator/jobset-controller-manager-856c7c87c8-lj5gw" Oct 31 00:23:21.728455 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:23:21.728352 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-856c7c87c8-lj5gw" Oct 31 00:23:22.098021 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:23:22.097985 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod706fb85b_6592_4394_b572_6f8dbe9f2c4d.slice/crio-102c1bf271617ee57cfcb954bb57c3c4d8468f1855670f6fba347d6d4ed315e8 WatchSource:0}: Error finding container 102c1bf271617ee57cfcb954bb57c3c4d8468f1855670f6fba347d6d4ed315e8: Status 404 returned error can't find the container with id 102c1bf271617ee57cfcb954bb57c3c4d8468f1855670f6fba347d6d4ed315e8 Oct 31 00:23:22.098769 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:23:22.098744 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-856c7c87c8-lj5gw"] Oct 31 00:23:22.751438 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:23:22.751400 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-856c7c87c8-lj5gw" event={"ID":"706fb85b-6592-4394-b572-6f8dbe9f2c4d","Type":"ContainerStarted","Data":"102c1bf271617ee57cfcb954bb57c3c4d8468f1855670f6fba347d6d4ed315e8"} Oct 31 00:23:26.773166 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:23:26.773126 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-856c7c87c8-lj5gw" event={"ID":"706fb85b-6592-4394-b572-6f8dbe9f2c4d","Type":"ContainerStarted","Data":"bac1a2ccfc5bdbb6be44301b4fb8911d2b72193c8f016018a987b0a97e2fde33"} Oct 31 00:23:26.773717 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:23:26.773244 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-856c7c87c8-lj5gw" Oct 31 00:23:26.892829 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:23:26.892774 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-856c7c87c8-lj5gw" podStartSLOduration=1.544883963 podStartE2EDuration="5.892758803s" totalImagesPullingTime="4.34787484s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:23:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:23:26.891858248 +0000 UTC m=+499.363027996" watchObservedRunningTime="2026-10-31 00:23:26.892758803 +0000 UTC m=+499.363928551" Oct 31 00:23:35.889802 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:23:35.889754 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 00:23:37.782492 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:23:37.782460 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-856c7c87c8-lj5gw" Oct 31 00:24:05.889455 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:24:05.889325 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 00:24:35.889927 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:24:35.889888 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 00:24:57.703667 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:24:57.703631 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-79tk8"] Oct 31 00:25:05.889896 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:25:05.889859 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 00:25:07.712840 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:07.712816 5222 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h9hsf_0f8c3c03-a1fc-4376-98c6-feeeefa08068/ovn-acl-logging/0.log" Oct 31 00:25:07.713310 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:07.712973 5222 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h9hsf_0f8c3c03-a1fc-4376-98c6-feeeefa08068/ovn-acl-logging/0.log" Oct 31 00:25:09.779752 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:09.779714 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-79tk8"] Oct 31 00:25:16.212728 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:16.212688 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4hfsn"] Oct 31 00:25:16.216204 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:16.216186 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4hfsn" Oct 31 00:25:16.237814 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:16.237784 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00f550bd-1442-4662-bdc3-5714f0afda9d-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4hfsn\" (UID: \"00f550bd-1442-4662-bdc3-5714f0afda9d\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4hfsn" Oct 31 00:25:16.238005 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:16.237828 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsnhg\" (UniqueName: \"kubernetes.io/projected/00f550bd-1442-4662-bdc3-5714f0afda9d-kube-api-access-wsnhg\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4hfsn\" (UID: \"00f550bd-1442-4662-bdc3-5714f0afda9d\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4hfsn" Oct 31 00:25:16.238005 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:16.237909 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00f550bd-1442-4662-bdc3-5714f0afda9d-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4hfsn\" (UID: \"00f550bd-1442-4662-bdc3-5714f0afda9d\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4hfsn" Oct 31 00:25:16.238005 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:16.237978 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00f550bd-1442-4662-bdc3-5714f0afda9d-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4hfsn\" (UID: \"00f550bd-1442-4662-bdc3-5714f0afda9d\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4hfsn" Oct 31 00:25:16.238122 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:16.238013 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00f550bd-1442-4662-bdc3-5714f0afda9d-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4hfsn\" (UID: \"00f550bd-1442-4662-bdc3-5714f0afda9d\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4hfsn" Oct 31 00:25:16.238170 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:16.238111 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00f550bd-1442-4662-bdc3-5714f0afda9d-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4hfsn\" (UID: \"00f550bd-1442-4662-bdc3-5714f0afda9d\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4hfsn" Oct 31 00:25:16.270151 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:16.270118 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 31 00:25:16.321188 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:16.321156 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 31 00:25:16.321360 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:16.321216 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Oct 31 00:25:16.321360 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:16.321326 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-hg9vc\"" Oct 31 00:25:16.322708 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:16.322692 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4hfsn"] Oct 31 00:25:16.339593 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:16.339561 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00f550bd-1442-4662-bdc3-5714f0afda9d-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4hfsn\" (UID: \"00f550bd-1442-4662-bdc3-5714f0afda9d\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4hfsn" Oct 31 00:25:16.339778 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:16.339614 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00f550bd-1442-4662-bdc3-5714f0afda9d-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4hfsn\" (UID: \"00f550bd-1442-4662-bdc3-5714f0afda9d\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4hfsn" Oct 31 00:25:16.339778 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:16.339648 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00f550bd-1442-4662-bdc3-5714f0afda9d-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4hfsn\" (UID: \"00f550bd-1442-4662-bdc3-5714f0afda9d\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4hfsn" Oct 31 00:25:16.339778 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:16.339669 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsnhg\" (UniqueName: \"kubernetes.io/projected/00f550bd-1442-4662-bdc3-5714f0afda9d-kube-api-access-wsnhg\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4hfsn\" (UID: \"00f550bd-1442-4662-bdc3-5714f0afda9d\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4hfsn" Oct 31 00:25:16.339778 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:16.339695 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00f550bd-1442-4662-bdc3-5714f0afda9d-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4hfsn\" (UID: \"00f550bd-1442-4662-bdc3-5714f0afda9d\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4hfsn" Oct 31 00:25:16.339778 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:16.339745 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00f550bd-1442-4662-bdc3-5714f0afda9d-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4hfsn\" (UID: \"00f550bd-1442-4662-bdc3-5714f0afda9d\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4hfsn" Oct 31 00:25:16.340159 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:16.340130 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00f550bd-1442-4662-bdc3-5714f0afda9d-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4hfsn\" (UID: \"00f550bd-1442-4662-bdc3-5714f0afda9d\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4hfsn" Oct 31 00:25:16.340296 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:16.340166 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00f550bd-1442-4662-bdc3-5714f0afda9d-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4hfsn\" (UID: \"00f550bd-1442-4662-bdc3-5714f0afda9d\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4hfsn" Oct 31 00:25:16.340296 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:16.340201 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00f550bd-1442-4662-bdc3-5714f0afda9d-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4hfsn\" (UID: \"00f550bd-1442-4662-bdc3-5714f0afda9d\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4hfsn" Oct 31 00:25:16.342076 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:16.342054 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00f550bd-1442-4662-bdc3-5714f0afda9d-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4hfsn\" (UID: \"00f550bd-1442-4662-bdc3-5714f0afda9d\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4hfsn" Oct 31 00:25:16.342408 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:16.342385 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00f550bd-1442-4662-bdc3-5714f0afda9d-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4hfsn\" (UID: \"00f550bd-1442-4662-bdc3-5714f0afda9d\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4hfsn" Oct 31 00:25:16.403285 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:16.403252 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsnhg\" (UniqueName: \"kubernetes.io/projected/00f550bd-1442-4662-bdc3-5714f0afda9d-kube-api-access-wsnhg\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4hfsn\" (UID: \"00f550bd-1442-4662-bdc3-5714f0afda9d\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4hfsn" Oct 31 00:25:16.528719 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:16.528685 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4hfsn" Oct 31 00:25:16.902896 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:25:16.902851 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00f550bd_1442_4662_bdc3_5714f0afda9d.slice/crio-8ec54062158c5fad8561fe87b8aeb7a43c5847abb93e1619eba95024320b5987 WatchSource:0}: Error finding container 8ec54062158c5fad8561fe87b8aeb7a43c5847abb93e1619eba95024320b5987: Status 404 returned error can't find the container with id 8ec54062158c5fad8561fe87b8aeb7a43c5847abb93e1619eba95024320b5987 Oct 31 00:25:16.904119 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:16.904095 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4hfsn"] Oct 31 00:25:17.237603 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:17.237516 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4hfsn" event={"ID":"00f550bd-1442-4662-bdc3-5714f0afda9d","Type":"ContainerStarted","Data":"8ec54062158c5fad8561fe87b8aeb7a43c5847abb93e1619eba95024320b5987"} Oct 31 00:25:19.247559 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:19.247521 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4hfsn" event={"ID":"00f550bd-1442-4662-bdc3-5714f0afda9d","Type":"ContainerStarted","Data":"1b0d6eb557bc6a58deadd08af3a39b939d00ca479d9c53e536869b58b663792d"} Oct 31 00:25:19.374674 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:19.374571 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4hfsn" podStartSLOduration=1.193054013 podStartE2EDuration="3.374554619s" totalImagesPullingTime="2.181500606s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:25:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:25:19.373447679 +0000 UTC m=+611.844617427" watchObservedRunningTime="2026-10-31 00:25:19.374554619 +0000 UTC m=+611.845724369" Oct 31 00:25:19.528877 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:19.528838 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4hfsn" Oct 31 00:25:20.102881 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:20.102845 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nvtsz"] Oct 31 00:25:20.106385 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:20.106345 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nvtsz" Oct 31 00:25:20.163111 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:20.163081 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nvtsz"] Oct 31 00:25:20.175785 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:20.175745 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c357cb32-7e30-442c-b612-f46e4a7dbc5a-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-nvtsz\" (UID: \"c357cb32-7e30-442c-b612-f46e4a7dbc5a\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nvtsz" Oct 31 00:25:20.175949 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:20.175898 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c357cb32-7e30-442c-b612-f46e4a7dbc5a-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-nvtsz\" (UID: \"c357cb32-7e30-442c-b612-f46e4a7dbc5a\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nvtsz" Oct 31 00:25:20.176023 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:20.176002 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c357cb32-7e30-442c-b612-f46e4a7dbc5a-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-nvtsz\" (UID: \"c357cb32-7e30-442c-b612-f46e4a7dbc5a\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nvtsz" Oct 31 00:25:20.176064 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:20.176036 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c357cb32-7e30-442c-b612-f46e4a7dbc5a-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-nvtsz\" (UID: \"c357cb32-7e30-442c-b612-f46e4a7dbc5a\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nvtsz" Oct 31 00:25:20.176064 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:20.176054 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c357cb32-7e30-442c-b612-f46e4a7dbc5a-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-nvtsz\" (UID: \"c357cb32-7e30-442c-b612-f46e4a7dbc5a\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nvtsz" Oct 31 00:25:20.176167 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:20.176150 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4md8l\" (UniqueName: \"kubernetes.io/projected/c357cb32-7e30-442c-b612-f46e4a7dbc5a-kube-api-access-4md8l\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-nvtsz\" (UID: \"c357cb32-7e30-442c-b612-f46e4a7dbc5a\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nvtsz" Oct 31 00:25:20.211810 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:20.211778 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Oct 31 00:25:20.277257 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:20.277209 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c357cb32-7e30-442c-b612-f46e4a7dbc5a-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-nvtsz\" (UID: \"c357cb32-7e30-442c-b612-f46e4a7dbc5a\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nvtsz" Oct 31 00:25:20.277257 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:20.277253 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c357cb32-7e30-442c-b612-f46e4a7dbc5a-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-nvtsz\" (UID: \"c357cb32-7e30-442c-b612-f46e4a7dbc5a\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nvtsz" Oct 31 00:25:20.277842 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:20.277270 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c357cb32-7e30-442c-b612-f46e4a7dbc5a-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-nvtsz\" (UID: \"c357cb32-7e30-442c-b612-f46e4a7dbc5a\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nvtsz" Oct 31 00:25:20.277842 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:20.277296 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4md8l\" (UniqueName: \"kubernetes.io/projected/c357cb32-7e30-442c-b612-f46e4a7dbc5a-kube-api-access-4md8l\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-nvtsz\" (UID: \"c357cb32-7e30-442c-b612-f46e4a7dbc5a\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nvtsz" Oct 31 00:25:20.277842 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:20.277434 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c357cb32-7e30-442c-b612-f46e4a7dbc5a-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-nvtsz\" (UID: \"c357cb32-7e30-442c-b612-f46e4a7dbc5a\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nvtsz" Oct 31 00:25:20.277842 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:20.277542 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c357cb32-7e30-442c-b612-f46e4a7dbc5a-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-nvtsz\" (UID: \"c357cb32-7e30-442c-b612-f46e4a7dbc5a\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nvtsz" Oct 31 00:25:20.277842 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:20.277778 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c357cb32-7e30-442c-b612-f46e4a7dbc5a-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-nvtsz\" (UID: \"c357cb32-7e30-442c-b612-f46e4a7dbc5a\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nvtsz" Oct 31 00:25:20.277842 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:20.277796 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c357cb32-7e30-442c-b612-f46e4a7dbc5a-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-nvtsz\" (UID: \"c357cb32-7e30-442c-b612-f46e4a7dbc5a\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nvtsz" Oct 31 00:25:20.278030 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:20.277848 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c357cb32-7e30-442c-b612-f46e4a7dbc5a-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-nvtsz\" (UID: \"c357cb32-7e30-442c-b612-f46e4a7dbc5a\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nvtsz" Oct 31 00:25:20.279786 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:20.279762 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c357cb32-7e30-442c-b612-f46e4a7dbc5a-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-nvtsz\" (UID: \"c357cb32-7e30-442c-b612-f46e4a7dbc5a\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nvtsz" Oct 31 00:25:20.279920 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:20.279901 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c357cb32-7e30-442c-b612-f46e4a7dbc5a-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-nvtsz\" (UID: \"c357cb32-7e30-442c-b612-f46e4a7dbc5a\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nvtsz" Oct 31 00:25:20.340400 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:20.340332 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4md8l\" (UniqueName: \"kubernetes.io/projected/c357cb32-7e30-442c-b612-f46e4a7dbc5a-kube-api-access-4md8l\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-nvtsz\" (UID: \"c357cb32-7e30-442c-b612-f46e4a7dbc5a\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nvtsz" Oct 31 00:25:20.434576 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:20.434490 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nvtsz" Oct 31 00:25:20.808535 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:25:20.808504 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc357cb32_7e30_442c_b612_f46e4a7dbc5a.slice/crio-9973ef69561d7b6e6880304d0ece26095f4d76f1d51ca76b1a4cce9dfe5ff0ff WatchSource:0}: Error finding container 9973ef69561d7b6e6880304d0ece26095f4d76f1d51ca76b1a4cce9dfe5ff0ff: Status 404 returned error can't find the container with id 9973ef69561d7b6e6880304d0ece26095f4d76f1d51ca76b1a4cce9dfe5ff0ff Oct 31 00:25:20.809322 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:20.809300 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nvtsz"] Oct 31 00:25:21.258391 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:21.258285 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nvtsz" event={"ID":"c357cb32-7e30-442c-b612-f46e4a7dbc5a","Type":"ContainerStarted","Data":"9973ef69561d7b6e6880304d0ece26095f4d76f1d51ca76b1a4cce9dfe5ff0ff"} Oct 31 00:25:22.264253 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:22.264223 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nvtsz" event={"ID":"c357cb32-7e30-442c-b612-f46e4a7dbc5a","Type":"ContainerStarted","Data":"d1a9993e3a4a932a2776b168a4c0df4bdf9274b6581be87ee97394f46a8e3029"} Oct 31 00:25:22.384770 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:22.384714 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nvtsz" podStartSLOduration=1.816992484 podStartE2EDuration="2.384698162s" totalImagesPullingTime="567.705678ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:25:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:25:22.383683079 +0000 UTC m=+614.854852830" watchObservedRunningTime="2026-10-31 00:25:22.384698162 +0000 UTC m=+614.855867946" Oct 31 00:25:22.434832 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:22.434798 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nvtsz" Oct 31 00:25:22.920410 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:22.920361 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-79tk8"] Oct 31 00:25:26.529332 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:26.529284 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4hfsn" Oct 31 00:25:26.542216 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:26.542188 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4hfsn" Oct 31 00:25:27.297168 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:27.297132 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4hfsn" Oct 31 00:25:30.434653 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:30.434616 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nvtsz" Oct 31 00:25:30.446953 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:30.446924 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nvtsz" Oct 31 00:25:31.314877 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:31.314840 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nvtsz" Oct 31 00:25:31.527136 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:31.527099 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mk8f6"] Oct 31 00:25:31.531123 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:31.531099 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mk8f6" Oct 31 00:25:31.579849 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:31.579771 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a83f0bf9-4b49-4e5f-9e11-838a1307907e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-mk8f6\" (UID: \"a83f0bf9-4b49-4e5f-9e11-838a1307907e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mk8f6" Oct 31 00:25:31.579849 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:31.579815 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a83f0bf9-4b49-4e5f-9e11-838a1307907e-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-mk8f6\" (UID: \"a83f0bf9-4b49-4e5f-9e11-838a1307907e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mk8f6" Oct 31 00:25:31.579849 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:31.579831 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vl8hm\" (UniqueName: \"kubernetes.io/projected/a83f0bf9-4b49-4e5f-9e11-838a1307907e-kube-api-access-vl8hm\") pod \"maas-default-gateway-openshift-default-6886b96f9-mk8f6\" (UID: \"a83f0bf9-4b49-4e5f-9e11-838a1307907e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mk8f6" Oct 31 00:25:31.580073 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:31.579988 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a83f0bf9-4b49-4e5f-9e11-838a1307907e-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-mk8f6\" (UID: \"a83f0bf9-4b49-4e5f-9e11-838a1307907e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mk8f6" Oct 31 00:25:31.580073 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:31.580041 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a83f0bf9-4b49-4e5f-9e11-838a1307907e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-mk8f6\" (UID: \"a83f0bf9-4b49-4e5f-9e11-838a1307907e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mk8f6" Oct 31 00:25:31.580170 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:31.580078 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a83f0bf9-4b49-4e5f-9e11-838a1307907e-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-mk8f6\" (UID: \"a83f0bf9-4b49-4e5f-9e11-838a1307907e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mk8f6" Oct 31 00:25:31.580170 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:31.580096 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a83f0bf9-4b49-4e5f-9e11-838a1307907e-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-mk8f6\" (UID: \"a83f0bf9-4b49-4e5f-9e11-838a1307907e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mk8f6" Oct 31 00:25:31.580279 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:31.580162 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a83f0bf9-4b49-4e5f-9e11-838a1307907e-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-mk8f6\" (UID: \"a83f0bf9-4b49-4e5f-9e11-838a1307907e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mk8f6" Oct 31 00:25:31.580320 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:31.580301 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a83f0bf9-4b49-4e5f-9e11-838a1307907e-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-mk8f6\" (UID: \"a83f0bf9-4b49-4e5f-9e11-838a1307907e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mk8f6" Oct 31 00:25:31.621114 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:31.621077 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mk8f6"] Oct 31 00:25:31.667652 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:31.667611 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-fxh6q\"" Oct 31 00:25:31.681035 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:31.680997 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a83f0bf9-4b49-4e5f-9e11-838a1307907e-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-mk8f6\" (UID: \"a83f0bf9-4b49-4e5f-9e11-838a1307907e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mk8f6" Oct 31 00:25:31.681232 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:31.681052 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a83f0bf9-4b49-4e5f-9e11-838a1307907e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-mk8f6\" (UID: \"a83f0bf9-4b49-4e5f-9e11-838a1307907e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mk8f6" Oct 31 00:25:31.681232 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:31.681079 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a83f0bf9-4b49-4e5f-9e11-838a1307907e-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-mk8f6\" (UID: \"a83f0bf9-4b49-4e5f-9e11-838a1307907e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mk8f6" Oct 31 00:25:31.681232 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:31.681189 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vl8hm\" (UniqueName: \"kubernetes.io/projected/a83f0bf9-4b49-4e5f-9e11-838a1307907e-kube-api-access-vl8hm\") pod \"maas-default-gateway-openshift-default-6886b96f9-mk8f6\" (UID: \"a83f0bf9-4b49-4e5f-9e11-838a1307907e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mk8f6" Oct 31 00:25:31.681449 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:31.681238 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a83f0bf9-4b49-4e5f-9e11-838a1307907e-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-mk8f6\" (UID: \"a83f0bf9-4b49-4e5f-9e11-838a1307907e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mk8f6" Oct 31 00:25:31.681449 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:31.681427 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a83f0bf9-4b49-4e5f-9e11-838a1307907e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-mk8f6\" (UID: \"a83f0bf9-4b49-4e5f-9e11-838a1307907e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mk8f6" Oct 31 00:25:31.681565 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:31.681484 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a83f0bf9-4b49-4e5f-9e11-838a1307907e-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-mk8f6\" (UID: \"a83f0bf9-4b49-4e5f-9e11-838a1307907e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mk8f6" Oct 31 00:25:31.681565 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:31.681517 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a83f0bf9-4b49-4e5f-9e11-838a1307907e-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-mk8f6\" (UID: \"a83f0bf9-4b49-4e5f-9e11-838a1307907e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mk8f6" Oct 31 00:25:31.681565 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:31.681533 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a83f0bf9-4b49-4e5f-9e11-838a1307907e-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-mk8f6\" (UID: \"a83f0bf9-4b49-4e5f-9e11-838a1307907e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mk8f6" Oct 31 00:25:31.681565 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:31.681546 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a83f0bf9-4b49-4e5f-9e11-838a1307907e-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-mk8f6\" (UID: \"a83f0bf9-4b49-4e5f-9e11-838a1307907e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mk8f6" Oct 31 00:25:31.681753 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:31.681609 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a83f0bf9-4b49-4e5f-9e11-838a1307907e-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-mk8f6\" (UID: \"a83f0bf9-4b49-4e5f-9e11-838a1307907e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mk8f6" Oct 31 00:25:31.681753 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:31.681691 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a83f0bf9-4b49-4e5f-9e11-838a1307907e-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-mk8f6\" (UID: \"a83f0bf9-4b49-4e5f-9e11-838a1307907e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mk8f6" Oct 31 00:25:31.681956 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:31.681919 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a83f0bf9-4b49-4e5f-9e11-838a1307907e-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-mk8f6\" (UID: \"a83f0bf9-4b49-4e5f-9e11-838a1307907e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mk8f6" Oct 31 00:25:31.682144 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:31.682123 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a83f0bf9-4b49-4e5f-9e11-838a1307907e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-mk8f6\" (UID: \"a83f0bf9-4b49-4e5f-9e11-838a1307907e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mk8f6" Oct 31 00:25:31.683716 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:31.683698 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a83f0bf9-4b49-4e5f-9e11-838a1307907e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-mk8f6\" (UID: \"a83f0bf9-4b49-4e5f-9e11-838a1307907e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mk8f6" Oct 31 00:25:31.683812 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:31.683745 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a83f0bf9-4b49-4e5f-9e11-838a1307907e-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-mk8f6\" (UID: \"a83f0bf9-4b49-4e5f-9e11-838a1307907e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mk8f6" Oct 31 00:25:31.743517 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:31.743484 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a83f0bf9-4b49-4e5f-9e11-838a1307907e-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-mk8f6\" (UID: \"a83f0bf9-4b49-4e5f-9e11-838a1307907e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mk8f6" Oct 31 00:25:31.743657 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:31.743630 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vl8hm\" (UniqueName: \"kubernetes.io/projected/a83f0bf9-4b49-4e5f-9e11-838a1307907e-kube-api-access-vl8hm\") pod \"maas-default-gateway-openshift-default-6886b96f9-mk8f6\" (UID: \"a83f0bf9-4b49-4e5f-9e11-838a1307907e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mk8f6" Oct 31 00:25:31.843950 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:31.843860 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mk8f6" Oct 31 00:25:32.264469 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:25:32.264423 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda83f0bf9_4b49_4e5f_9e11_838a1307907e.slice/crio-2aa1b90556ac90a3e96b0ae1931b50723763b817b21b205b8819fad996481da0 WatchSource:0}: Error finding container 2aa1b90556ac90a3e96b0ae1931b50723763b817b21b205b8819fad996481da0: Status 404 returned error can't find the container with id 2aa1b90556ac90a3e96b0ae1931b50723763b817b21b205b8819fad996481da0 Oct 31 00:25:32.267504 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:32.265998 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mk8f6"] Oct 31 00:25:32.268655 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:32.268632 5222 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 00:25:32.308806 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:32.308765 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mk8f6" event={"ID":"a83f0bf9-4b49-4e5f-9e11-838a1307907e","Type":"ContainerStarted","Data":"2aa1b90556ac90a3e96b0ae1931b50723763b817b21b205b8819fad996481da0"} Oct 31 00:25:35.892338 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:25:35.892300 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 00:25:36.019519 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:36.019485 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-79tk8"] Oct 31 00:25:36.419927 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:36.419898 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:25:36.419995 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:36.419962 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:25:36.420029 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:36.420006 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:25:37.336330 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:37.336294 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mk8f6" event={"ID":"a83f0bf9-4b49-4e5f-9e11-838a1307907e","Type":"ContainerStarted","Data":"807b78c6972e752e3d52d4597ab34a5ca80314102260c84d495fda8a6f85c2aa"} Oct 31 00:25:37.529863 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:37.529801 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mk8f6" podStartSLOduration=2.379631503 podStartE2EDuration="6.529785447s" totalImagesPullingTime="4.150153944s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:25:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:25:37.528465615 +0000 UTC m=+629.999635358" watchObservedRunningTime="2026-10-31 00:25:37.529785447 +0000 UTC m=+630.000955254" Oct 31 00:25:37.856786 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:37.856747 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mk8f6" Oct 31 00:25:37.860545 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:37.860512 5222 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-mk8f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Oct 31 00:25:37.860811 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:37.860781 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mk8f6" podUID="a83f0bf9-4b49-4e5f-9e11-838a1307907e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Oct 31 00:25:38.857396 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:38.847793 5222 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-mk8f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Oct 31 00:25:38.857396 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:38.847874 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mk8f6" podUID="a83f0bf9-4b49-4e5f-9e11-838a1307907e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Oct 31 00:25:39.847884 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:39.847847 5222 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-mk8f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Oct 31 00:25:39.848179 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:39.848147 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mk8f6" podUID="a83f0bf9-4b49-4e5f-9e11-838a1307907e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Oct 31 00:25:40.858220 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:40.857761 5222 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-mk8f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Oct 31 00:25:40.858220 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:40.857839 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mk8f6" podUID="a83f0bf9-4b49-4e5f-9e11-838a1307907e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Oct 31 00:25:41.858398 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:41.855802 5222 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-mk8f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Oct 31 00:25:41.858398 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:41.855903 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mk8f6" podUID="a83f0bf9-4b49-4e5f-9e11-838a1307907e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Oct 31 00:25:42.844331 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:42.844298 5222 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-mk8f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Oct 31 00:25:42.844664 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:42.844354 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mk8f6" podUID="a83f0bf9-4b49-4e5f-9e11-838a1307907e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Oct 31 00:25:43.845128 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:43.845083 5222 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-mk8f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Oct 31 00:25:43.845681 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:43.845136 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mk8f6" podUID="a83f0bf9-4b49-4e5f-9e11-838a1307907e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Oct 31 00:25:44.849202 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:44.849121 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mk8f6" Oct 31 00:25:44.849668 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:44.849568 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mk8f6" Oct 31 00:25:44.850336 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:44.850311 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mk8f6" Oct 31 00:25:46.655957 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:46.655920 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ztfwh"] Oct 31 00:25:46.659550 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:46.659531 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ztfwh" Oct 31 00:25:46.750886 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:46.750851 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ztfwh"] Oct 31 00:25:46.848744 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:46.848710 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9980a823-4548-4604-ae1b-a9da1aa6d30a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-ztfwh\" (UID: \"9980a823-4548-4604-ae1b-a9da1aa6d30a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ztfwh" Oct 31 00:25:46.848931 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:46.848750 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9980a823-4548-4604-ae1b-a9da1aa6d30a-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-ztfwh\" (UID: \"9980a823-4548-4604-ae1b-a9da1aa6d30a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ztfwh" Oct 31 00:25:46.848931 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:46.848781 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9980a823-4548-4604-ae1b-a9da1aa6d30a-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-ztfwh\" (UID: \"9980a823-4548-4604-ae1b-a9da1aa6d30a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ztfwh" Oct 31 00:25:46.848999 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:46.848931 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbmq9\" (UniqueName: \"kubernetes.io/projected/9980a823-4548-4604-ae1b-a9da1aa6d30a-kube-api-access-vbmq9\") pod \"maas-default-gateway-openshift-default-6886b96f9-ztfwh\" (UID: \"9980a823-4548-4604-ae1b-a9da1aa6d30a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ztfwh" Oct 31 00:25:46.848999 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:46.848964 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9980a823-4548-4604-ae1b-a9da1aa6d30a-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-ztfwh\" (UID: \"9980a823-4548-4604-ae1b-a9da1aa6d30a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ztfwh" Oct 31 00:25:46.849063 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:46.849031 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9980a823-4548-4604-ae1b-a9da1aa6d30a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-ztfwh\" (UID: \"9980a823-4548-4604-ae1b-a9da1aa6d30a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ztfwh" Oct 31 00:25:46.849063 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:46.849049 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9980a823-4548-4604-ae1b-a9da1aa6d30a-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-ztfwh\" (UID: \"9980a823-4548-4604-ae1b-a9da1aa6d30a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ztfwh" Oct 31 00:25:46.849241 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:46.849220 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9980a823-4548-4604-ae1b-a9da1aa6d30a-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-ztfwh\" (UID: \"9980a823-4548-4604-ae1b-a9da1aa6d30a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ztfwh" Oct 31 00:25:46.849297 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:46.849272 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9980a823-4548-4604-ae1b-a9da1aa6d30a-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-ztfwh\" (UID: \"9980a823-4548-4604-ae1b-a9da1aa6d30a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ztfwh" Oct 31 00:25:46.950049 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:46.949956 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9980a823-4548-4604-ae1b-a9da1aa6d30a-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-ztfwh\" (UID: \"9980a823-4548-4604-ae1b-a9da1aa6d30a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ztfwh" Oct 31 00:25:46.950049 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:46.950008 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9980a823-4548-4604-ae1b-a9da1aa6d30a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-ztfwh\" (UID: \"9980a823-4548-4604-ae1b-a9da1aa6d30a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ztfwh" Oct 31 00:25:46.950049 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:46.950024 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9980a823-4548-4604-ae1b-a9da1aa6d30a-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-ztfwh\" (UID: \"9980a823-4548-4604-ae1b-a9da1aa6d30a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ztfwh" Oct 31 00:25:46.950309 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:46.950053 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9980a823-4548-4604-ae1b-a9da1aa6d30a-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-ztfwh\" (UID: \"9980a823-4548-4604-ae1b-a9da1aa6d30a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ztfwh" Oct 31 00:25:46.950309 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:46.950081 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9980a823-4548-4604-ae1b-a9da1aa6d30a-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-ztfwh\" (UID: \"9980a823-4548-4604-ae1b-a9da1aa6d30a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ztfwh" Oct 31 00:25:46.950309 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:46.950103 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9980a823-4548-4604-ae1b-a9da1aa6d30a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-ztfwh\" (UID: \"9980a823-4548-4604-ae1b-a9da1aa6d30a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ztfwh" Oct 31 00:25:46.950309 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:46.950181 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9980a823-4548-4604-ae1b-a9da1aa6d30a-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-ztfwh\" (UID: \"9980a823-4548-4604-ae1b-a9da1aa6d30a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ztfwh" Oct 31 00:25:46.950309 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:46.950259 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9980a823-4548-4604-ae1b-a9da1aa6d30a-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-ztfwh\" (UID: \"9980a823-4548-4604-ae1b-a9da1aa6d30a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ztfwh" Oct 31 00:25:46.950623 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:46.950340 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbmq9\" (UniqueName: \"kubernetes.io/projected/9980a823-4548-4604-ae1b-a9da1aa6d30a-kube-api-access-vbmq9\") pod \"maas-default-gateway-openshift-default-6886b96f9-ztfwh\" (UID: \"9980a823-4548-4604-ae1b-a9da1aa6d30a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ztfwh" Oct 31 00:25:46.950623 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:46.950533 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9980a823-4548-4604-ae1b-a9da1aa6d30a-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-ztfwh\" (UID: \"9980a823-4548-4604-ae1b-a9da1aa6d30a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ztfwh" Oct 31 00:25:46.950730 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:46.950652 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9980a823-4548-4604-ae1b-a9da1aa6d30a-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-ztfwh\" (UID: \"9980a823-4548-4604-ae1b-a9da1aa6d30a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ztfwh" Oct 31 00:25:46.950730 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:46.950661 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9980a823-4548-4604-ae1b-a9da1aa6d30a-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-ztfwh\" (UID: \"9980a823-4548-4604-ae1b-a9da1aa6d30a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ztfwh" Oct 31 00:25:46.950841 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:46.950804 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9980a823-4548-4604-ae1b-a9da1aa6d30a-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-ztfwh\" (UID: \"9980a823-4548-4604-ae1b-a9da1aa6d30a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ztfwh" Oct 31 00:25:46.950985 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:46.950964 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9980a823-4548-4604-ae1b-a9da1aa6d30a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-ztfwh\" (UID: \"9980a823-4548-4604-ae1b-a9da1aa6d30a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ztfwh" Oct 31 00:25:46.952506 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:46.952487 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9980a823-4548-4604-ae1b-a9da1aa6d30a-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-ztfwh\" (UID: \"9980a823-4548-4604-ae1b-a9da1aa6d30a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ztfwh" Oct 31 00:25:46.952908 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:46.952892 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9980a823-4548-4604-ae1b-a9da1aa6d30a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-ztfwh\" (UID: \"9980a823-4548-4604-ae1b-a9da1aa6d30a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ztfwh" Oct 31 00:25:47.013591 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:47.013562 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbmq9\" (UniqueName: \"kubernetes.io/projected/9980a823-4548-4604-ae1b-a9da1aa6d30a-kube-api-access-vbmq9\") pod \"maas-default-gateway-openshift-default-6886b96f9-ztfwh\" (UID: \"9980a823-4548-4604-ae1b-a9da1aa6d30a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ztfwh" Oct 31 00:25:47.013740 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:47.013565 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9980a823-4548-4604-ae1b-a9da1aa6d30a-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-ztfwh\" (UID: \"9980a823-4548-4604-ae1b-a9da1aa6d30a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ztfwh" Oct 31 00:25:47.272659 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:47.272623 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ztfwh" Oct 31 00:25:47.717854 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:47.717803 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:25:47.718436 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:47.718402 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:25:47.718629 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:47.718610 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:25:47.719178 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:47.717823 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ztfwh"] Oct 31 00:25:48.414672 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:48.414635 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ztfwh" event={"ID":"9980a823-4548-4604-ae1b-a9da1aa6d30a","Type":"ContainerStarted","Data":"7074fcaa39f01299f615d78cc7e76185ea3fa29ae56c620a2ed4b41a3f1e6d76"} Oct 31 00:25:48.415043 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:48.414699 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ztfwh" event={"ID":"9980a823-4548-4604-ae1b-a9da1aa6d30a","Type":"ContainerStarted","Data":"889bec921eb97972bc3421a636b9c37101590251cd860808ac37b0bcedaaadd7"} Oct 31 00:25:48.589276 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:48.589223 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ztfwh" podStartSLOduration=2.589210965 podStartE2EDuration="2.589210965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:25:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:25:48.588277201 +0000 UTC m=+641.059446949" watchObservedRunningTime="2026-10-31 00:25:48.589210965 +0000 UTC m=+641.060380714" Oct 31 00:25:49.273497 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:49.273459 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ztfwh" Oct 31 00:25:49.274835 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:49.274804 5222 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-ztfwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Oct 31 00:25:49.274988 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:49.274869 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ztfwh" podUID="9980a823-4548-4604-ae1b-a9da1aa6d30a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Oct 31 00:25:50.273300 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:50.273265 5222 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-ztfwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Oct 31 00:25:50.273503 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:50.273340 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ztfwh" podUID="9980a823-4548-4604-ae1b-a9da1aa6d30a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Oct 31 00:25:51.273524 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:51.273492 5222 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-ztfwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Oct 31 00:25:51.273892 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:51.273545 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ztfwh" podUID="9980a823-4548-4604-ae1b-a9da1aa6d30a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Oct 31 00:25:52.273705 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:52.273668 5222 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-ztfwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Oct 31 00:25:52.274134 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:52.273722 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ztfwh" podUID="9980a823-4548-4604-ae1b-a9da1aa6d30a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Oct 31 00:25:53.273682 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:53.273646 5222 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-ztfwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Oct 31 00:25:53.273832 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:53.273701 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ztfwh" podUID="9980a823-4548-4604-ae1b-a9da1aa6d30a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Oct 31 00:25:54.273922 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:54.273882 5222 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-ztfwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Oct 31 00:25:54.274314 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:54.273955 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ztfwh" podUID="9980a823-4548-4604-ae1b-a9da1aa6d30a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Oct 31 00:25:56.026966 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:56.026918 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ztfwh" Oct 31 00:25:56.027459 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:56.027298 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ztfwh" Oct 31 00:25:56.028544 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:25:56.028518 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ztfwh" Oct 31 00:26:05.889502 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:26:05.889463 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 00:26:35.889737 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:26:35.889695 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 00:26:52.528037 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:26:52.528005 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-cml7r"] Oct 31 00:26:52.531501 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:26:52.531480 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-cml7r" Oct 31 00:26:52.585770 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:26:52.585733 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-e8ba06-openshift-default-dockercfg-97jbg\"" Oct 31 00:26:52.639694 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:26:52.639655 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-cml7r"] Oct 31 00:26:52.687074 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:26:52.687038 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f85c9ba2-c597-4786-93a1-930609ff510a-istio-token\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-cml7r\" (UID: \"f85c9ba2-c597-4786-93a1-930609ff510a\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-cml7r" Oct 31 00:26:52.687257 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:26:52.687088 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f85c9ba2-c597-4786-93a1-930609ff510a-credential-socket\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-cml7r\" (UID: \"f85c9ba2-c597-4786-93a1-930609ff510a\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-cml7r" Oct 31 00:26:52.687257 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:26:52.687106 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f85c9ba2-c597-4786-93a1-930609ff510a-istiod-ca-cert\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-cml7r\" (UID: \"f85c9ba2-c597-4786-93a1-930609ff510a\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-cml7r" Oct 31 00:26:52.687257 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:26:52.687222 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f85c9ba2-c597-4786-93a1-930609ff510a-istio-podinfo\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-cml7r\" (UID: \"f85c9ba2-c597-4786-93a1-930609ff510a\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-cml7r" Oct 31 00:26:52.687472 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:26:52.687257 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f85c9ba2-c597-4786-93a1-930609ff510a-istio-envoy\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-cml7r\" (UID: \"f85c9ba2-c597-4786-93a1-930609ff510a\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-cml7r" Oct 31 00:26:52.687472 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:26:52.687422 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f85c9ba2-c597-4786-93a1-930609ff510a-istio-data\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-cml7r\" (UID: \"f85c9ba2-c597-4786-93a1-930609ff510a\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-cml7r" Oct 31 00:26:52.687472 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:26:52.687462 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f85c9ba2-c597-4786-93a1-930609ff510a-workload-socket\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-cml7r\" (UID: \"f85c9ba2-c597-4786-93a1-930609ff510a\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-cml7r" Oct 31 00:26:52.687612 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:26:52.687523 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p72mr\" (UniqueName: \"kubernetes.io/projected/f85c9ba2-c597-4786-93a1-930609ff510a-kube-api-access-p72mr\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-cml7r\" (UID: \"f85c9ba2-c597-4786-93a1-930609ff510a\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-cml7r" Oct 31 00:26:52.687612 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:26:52.687594 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f85c9ba2-c597-4786-93a1-930609ff510a-workload-certs\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-cml7r\" (UID: \"f85c9ba2-c597-4786-93a1-930609ff510a\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-cml7r" Oct 31 00:26:52.788718 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:26:52.788623 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f85c9ba2-c597-4786-93a1-930609ff510a-istio-data\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-cml7r\" (UID: \"f85c9ba2-c597-4786-93a1-930609ff510a\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-cml7r" Oct 31 00:26:52.788718 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:26:52.788667 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f85c9ba2-c597-4786-93a1-930609ff510a-workload-socket\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-cml7r\" (UID: \"f85c9ba2-c597-4786-93a1-930609ff510a\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-cml7r" Oct 31 00:26:52.788718 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:26:52.788703 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p72mr\" (UniqueName: \"kubernetes.io/projected/f85c9ba2-c597-4786-93a1-930609ff510a-kube-api-access-p72mr\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-cml7r\" (UID: \"f85c9ba2-c597-4786-93a1-930609ff510a\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-cml7r" Oct 31 00:26:52.788990 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:26:52.788744 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f85c9ba2-c597-4786-93a1-930609ff510a-workload-certs\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-cml7r\" (UID: \"f85c9ba2-c597-4786-93a1-930609ff510a\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-cml7r" Oct 31 00:26:52.788990 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:26:52.788820 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f85c9ba2-c597-4786-93a1-930609ff510a-istio-token\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-cml7r\" (UID: \"f85c9ba2-c597-4786-93a1-930609ff510a\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-cml7r" Oct 31 00:26:52.788990 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:26:52.788869 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f85c9ba2-c597-4786-93a1-930609ff510a-credential-socket\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-cml7r\" (UID: \"f85c9ba2-c597-4786-93a1-930609ff510a\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-cml7r" Oct 31 00:26:52.788990 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:26:52.788892 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f85c9ba2-c597-4786-93a1-930609ff510a-istiod-ca-cert\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-cml7r\" (UID: \"f85c9ba2-c597-4786-93a1-930609ff510a\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-cml7r" Oct 31 00:26:52.788990 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:26:52.788919 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f85c9ba2-c597-4786-93a1-930609ff510a-istio-podinfo\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-cml7r\" (UID: \"f85c9ba2-c597-4786-93a1-930609ff510a\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-cml7r" Oct 31 00:26:52.788990 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:26:52.788949 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f85c9ba2-c597-4786-93a1-930609ff510a-istio-envoy\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-cml7r\" (UID: \"f85c9ba2-c597-4786-93a1-930609ff510a\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-cml7r" Oct 31 00:26:52.789288 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:26:52.789062 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f85c9ba2-c597-4786-93a1-930609ff510a-workload-socket\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-cml7r\" (UID: \"f85c9ba2-c597-4786-93a1-930609ff510a\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-cml7r" Oct 31 00:26:52.789288 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:26:52.789116 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f85c9ba2-c597-4786-93a1-930609ff510a-istio-data\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-cml7r\" (UID: \"f85c9ba2-c597-4786-93a1-930609ff510a\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-cml7r" Oct 31 00:26:52.789288 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:26:52.789174 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f85c9ba2-c597-4786-93a1-930609ff510a-workload-certs\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-cml7r\" (UID: \"f85c9ba2-c597-4786-93a1-930609ff510a\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-cml7r" Oct 31 00:26:52.789465 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:26:52.789353 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f85c9ba2-c597-4786-93a1-930609ff510a-credential-socket\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-cml7r\" (UID: \"f85c9ba2-c597-4786-93a1-930609ff510a\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-cml7r" Oct 31 00:26:52.789654 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:26:52.789635 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f85c9ba2-c597-4786-93a1-930609ff510a-istiod-ca-cert\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-cml7r\" (UID: \"f85c9ba2-c597-4786-93a1-930609ff510a\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-cml7r" Oct 31 00:26:52.791499 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:26:52.791479 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f85c9ba2-c597-4786-93a1-930609ff510a-istio-envoy\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-cml7r\" (UID: \"f85c9ba2-c597-4786-93a1-930609ff510a\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-cml7r" Oct 31 00:26:52.791667 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:26:52.791645 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f85c9ba2-c597-4786-93a1-930609ff510a-istio-podinfo\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-cml7r\" (UID: \"f85c9ba2-c597-4786-93a1-930609ff510a\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-cml7r" Oct 31 00:26:52.856128 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:26:52.856100 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f85c9ba2-c597-4786-93a1-930609ff510a-istio-token\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-cml7r\" (UID: \"f85c9ba2-c597-4786-93a1-930609ff510a\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-cml7r" Oct 31 00:26:52.856327 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:26:52.856304 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p72mr\" (UniqueName: \"kubernetes.io/projected/f85c9ba2-c597-4786-93a1-930609ff510a-kube-api-access-p72mr\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-cml7r\" (UID: \"f85c9ba2-c597-4786-93a1-930609ff510a\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-cml7r" Oct 31 00:26:53.144813 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:26:53.144727 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-cml7r" Oct 31 00:26:53.571914 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:26:53.571884 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf85c9ba2_c597_4786_93a1_930609ff510a.slice/crio-2af46126587414886cedc5d7166cc1452d249bc8e0d2628ef533fefdefa8dccc WatchSource:0}: Error finding container 2af46126587414886cedc5d7166cc1452d249bc8e0d2628ef533fefdefa8dccc: Status 404 returned error can't find the container with id 2af46126587414886cedc5d7166cc1452d249bc8e0d2628ef533fefdefa8dccc Oct 31 00:26:53.573591 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:26:53.573524 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-cml7r"] Oct 31 00:26:53.574303 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:26:53.574273 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:26:53.574350 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:26:53.574339 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:26:53.574425 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:26:53.574405 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:26:53.751106 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:26:53.751073 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-cml7r" event={"ID":"f85c9ba2-c597-4786-93a1-930609ff510a","Type":"ContainerStarted","Data":"34125c454b0f15ed31fe01b3cb49afe5730ab4bc1d4edef5b8f7784a20daa122"} Oct 31 00:26:53.751305 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:26:53.751130 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-cml7r" event={"ID":"f85c9ba2-c597-4786-93a1-930609ff510a","Type":"ContainerStarted","Data":"2af46126587414886cedc5d7166cc1452d249bc8e0d2628ef533fefdefa8dccc"} Oct 31 00:26:53.817644 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:26:53.817615 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-2jrfg"] Oct 31 00:26:53.821259 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:26:53.821200 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-2jrfg" Oct 31 00:26:53.876883 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:26:53.876851 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-982ae3-openshift-default-dockercfg-cwtrb\"" Oct 31 00:26:53.877603 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:26:53.877564 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-cml7r" podStartSLOduration=1.877549837 podStartE2EDuration="1.877549837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:26:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:26:53.876554004 +0000 UTC m=+706.347723754" watchObservedRunningTime="2026-10-31 00:26:53.877549837 +0000 UTC m=+706.348719589" Oct 31 00:26:53.887289 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:26:53.887259 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-2jrfg"] Oct 31 00:26:53.899560 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:26:53.899522 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fe80038-c64b-4114-ac0f-337a29f20204-credential-socket\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-2jrfg\" (UID: \"2fe80038-c64b-4114-ac0f-337a29f20204\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-2jrfg" Oct 31 00:26:53.899721 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:26:53.899569 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fe80038-c64b-4114-ac0f-337a29f20204-workload-certs\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-2jrfg\" (UID: \"2fe80038-c64b-4114-ac0f-337a29f20204\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-2jrfg" Oct 31 00:26:53.899721 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:26:53.899625 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fe80038-c64b-4114-ac0f-337a29f20204-istio-podinfo\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-2jrfg\" (UID: \"2fe80038-c64b-4114-ac0f-337a29f20204\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-2jrfg" Oct 31 00:26:53.899721 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:26:53.899644 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mrhl\" (UniqueName: \"kubernetes.io/projected/2fe80038-c64b-4114-ac0f-337a29f20204-kube-api-access-9mrhl\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-2jrfg\" (UID: \"2fe80038-c64b-4114-ac0f-337a29f20204\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-2jrfg" Oct 31 00:26:53.899721 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:26:53.899675 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fe80038-c64b-4114-ac0f-337a29f20204-istiod-ca-cert\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-2jrfg\" (UID: \"2fe80038-c64b-4114-ac0f-337a29f20204\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-2jrfg" Oct 31 00:26:53.899721 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:26:53.899707 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fe80038-c64b-4114-ac0f-337a29f20204-workload-socket\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-2jrfg\" (UID: \"2fe80038-c64b-4114-ac0f-337a29f20204\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-2jrfg" Oct 31 00:26:53.899907 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:26:53.899759 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fe80038-c64b-4114-ac0f-337a29f20204-istio-data\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-2jrfg\" (UID: \"2fe80038-c64b-4114-ac0f-337a29f20204\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-2jrfg" Oct 31 00:26:53.899907 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:26:53.899802 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fe80038-c64b-4114-ac0f-337a29f20204-istio-token\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-2jrfg\" (UID: \"2fe80038-c64b-4114-ac0f-337a29f20204\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-2jrfg" Oct 31 00:26:53.899907 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:26:53.899844 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fe80038-c64b-4114-ac0f-337a29f20204-istio-envoy\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-2jrfg\" (UID: \"2fe80038-c64b-4114-ac0f-337a29f20204\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-2jrfg" Oct 31 00:26:54.002206 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:26:54.002172 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fe80038-c64b-4114-ac0f-337a29f20204-istio-envoy\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-2jrfg\" (UID: \"2fe80038-c64b-4114-ac0f-337a29f20204\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-2jrfg" Oct 31 00:26:54.002498 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:26:54.002222 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fe80038-c64b-4114-ac0f-337a29f20204-credential-socket\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-2jrfg\" (UID: \"2fe80038-c64b-4114-ac0f-337a29f20204\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-2jrfg" Oct 31 00:26:54.002498 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:26:54.002261 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fe80038-c64b-4114-ac0f-337a29f20204-workload-certs\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-2jrfg\" (UID: \"2fe80038-c64b-4114-ac0f-337a29f20204\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-2jrfg" Oct 31 00:26:54.002498 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:26:54.002308 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fe80038-c64b-4114-ac0f-337a29f20204-istio-podinfo\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-2jrfg\" (UID: \"2fe80038-c64b-4114-ac0f-337a29f20204\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-2jrfg" Oct 31 00:26:54.002676 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:26:54.002497 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mrhl\" (UniqueName: \"kubernetes.io/projected/2fe80038-c64b-4114-ac0f-337a29f20204-kube-api-access-9mrhl\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-2jrfg\" (UID: \"2fe80038-c64b-4114-ac0f-337a29f20204\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-2jrfg" Oct 31 00:26:54.002676 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:26:54.002532 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fe80038-c64b-4114-ac0f-337a29f20204-istiod-ca-cert\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-2jrfg\" (UID: \"2fe80038-c64b-4114-ac0f-337a29f20204\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-2jrfg" Oct 31 00:26:54.002676 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:26:54.002576 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fe80038-c64b-4114-ac0f-337a29f20204-workload-socket\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-2jrfg\" (UID: \"2fe80038-c64b-4114-ac0f-337a29f20204\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-2jrfg" Oct 31 00:26:54.002676 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:26:54.002650 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fe80038-c64b-4114-ac0f-337a29f20204-istio-data\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-2jrfg\" (UID: \"2fe80038-c64b-4114-ac0f-337a29f20204\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-2jrfg" Oct 31 00:26:54.002857 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:26:54.002694 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fe80038-c64b-4114-ac0f-337a29f20204-istio-token\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-2jrfg\" (UID: \"2fe80038-c64b-4114-ac0f-337a29f20204\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-2jrfg" Oct 31 00:26:54.004186 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:26:54.003925 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fe80038-c64b-4114-ac0f-337a29f20204-workload-certs\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-2jrfg\" (UID: \"2fe80038-c64b-4114-ac0f-337a29f20204\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-2jrfg" Oct 31 00:26:54.004186 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:26:54.004007 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fe80038-c64b-4114-ac0f-337a29f20204-istio-data\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-2jrfg\" (UID: \"2fe80038-c64b-4114-ac0f-337a29f20204\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-2jrfg" Oct 31 00:26:54.004186 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:26:54.004057 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fe80038-c64b-4114-ac0f-337a29f20204-workload-socket\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-2jrfg\" (UID: \"2fe80038-c64b-4114-ac0f-337a29f20204\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-2jrfg" Oct 31 00:26:54.004641 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:26:54.004220 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fe80038-c64b-4114-ac0f-337a29f20204-credential-socket\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-2jrfg\" (UID: \"2fe80038-c64b-4114-ac0f-337a29f20204\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-2jrfg" Oct 31 00:26:54.004737 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:26:54.004702 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fe80038-c64b-4114-ac0f-337a29f20204-istiod-ca-cert\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-2jrfg\" (UID: \"2fe80038-c64b-4114-ac0f-337a29f20204\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-2jrfg" Oct 31 00:26:54.006621 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:26:54.006601 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fe80038-c64b-4114-ac0f-337a29f20204-istio-podinfo\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-2jrfg\" (UID: \"2fe80038-c64b-4114-ac0f-337a29f20204\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-2jrfg" Oct 31 00:26:54.011979 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:26:54.011956 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fe80038-c64b-4114-ac0f-337a29f20204-istio-envoy\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-2jrfg\" (UID: \"2fe80038-c64b-4114-ac0f-337a29f20204\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-2jrfg" Oct 31 00:26:54.065270 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:26:54.065237 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fe80038-c64b-4114-ac0f-337a29f20204-istio-token\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-2jrfg\" (UID: \"2fe80038-c64b-4114-ac0f-337a29f20204\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-2jrfg" Oct 31 00:26:54.065895 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:26:54.065864 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mrhl\" (UniqueName: \"kubernetes.io/projected/2fe80038-c64b-4114-ac0f-337a29f20204-kube-api-access-9mrhl\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-2jrfg\" (UID: \"2fe80038-c64b-4114-ac0f-337a29f20204\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-2jrfg" Oct 31 00:26:54.136866 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:26:54.136781 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-2jrfg" Oct 31 00:26:54.145061 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:26:54.145027 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-cml7r" Oct 31 00:26:54.150653 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:26:54.150630 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-cml7r" Oct 31 00:26:54.559402 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:26:54.559351 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fe80038_c64b_4114_ac0f_337a29f20204.slice/crio-9fdc833f4f2eebc9497590c6056fd9220fba2864b1dd01d2a321360cb5a2ff4d WatchSource:0}: Error finding container 9fdc833f4f2eebc9497590c6056fd9220fba2864b1dd01d2a321360cb5a2ff4d: Status 404 returned error can't find the container with id 9fdc833f4f2eebc9497590c6056fd9220fba2864b1dd01d2a321360cb5a2ff4d Oct 31 00:26:54.562099 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:26:54.562059 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-2jrfg"] Oct 31 00:26:54.567587 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:26:54.567407 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:26:54.567587 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:26:54.567547 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:26:54.567728 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:26:54.567638 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:26:54.758991 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:26:54.758953 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-2jrfg" event={"ID":"2fe80038-c64b-4114-ac0f-337a29f20204","Type":"ContainerStarted","Data":"51103936220c641723e6faed9902d3b5829355f32f6ab65da69cb81f5503a3e1"} Oct 31 00:26:54.759522 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:26:54.759017 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-2jrfg" event={"ID":"2fe80038-c64b-4114-ac0f-337a29f20204","Type":"ContainerStarted","Data":"9fdc833f4f2eebc9497590c6056fd9220fba2864b1dd01d2a321360cb5a2ff4d"} Oct 31 00:26:54.759522 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:26:54.759163 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-cml7r" Oct 31 00:26:54.760309 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:26:54.760289 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-cml7r" Oct 31 00:26:54.882829 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:26:54.882709 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-2jrfg" podStartSLOduration=1.882694466 podStartE2EDuration="1.882694466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:26:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:26:54.881725086 +0000 UTC m=+707.352894836" watchObservedRunningTime="2026-10-31 00:26:54.882694466 +0000 UTC m=+707.353864212" Oct 31 00:26:55.138573 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:26:55.137491 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-2jrfg" Oct 31 00:26:55.146001 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:26:55.145973 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-2jrfg" Oct 31 00:26:55.764116 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:26:55.764084 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-2jrfg" Oct 31 00:26:55.765116 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:26:55.765097 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-2jrfg" Oct 31 00:27:02.214310 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:02.214272 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-e8ba06-7b548bc5f-m9r6m"] Oct 31 00:27:02.217967 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:02.217943 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e8ba06-7b548bc5f-m9r6m" Oct 31 00:27:02.298438 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:02.298403 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06704c9d-24a7-4e63-bdd3-b7ee85866fbf-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-e8ba06-7b548bc5f-m9r6m\" (UID: \"06704c9d-24a7-4e63-bdd3-b7ee85866fbf\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e8ba06-7b548bc5f-m9r6m" Oct 31 00:27:02.298438 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:02.298442 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkckd\" (UniqueName: \"kubernetes.io/projected/06704c9d-24a7-4e63-bdd3-b7ee85866fbf-kube-api-access-tkckd\") pod \"payload-pre-processing-e2e-worker-w2-e8ba06-7b548bc5f-m9r6m\" (UID: \"06704c9d-24a7-4e63-bdd3-b7ee85866fbf\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e8ba06-7b548bc5f-m9r6m" Oct 31 00:27:02.322970 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:02.322939 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-e8ba06-dockercfg-4n6bm\"" Oct 31 00:27:02.322970 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:02.322955 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-e8ba06\"" Oct 31 00:27:02.324652 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:02.324625 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-e8ba06-7b548bc5f-m9r6m"] Oct 31 00:27:02.386842 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:02.386798 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-e8ba06-bb9fb667f-x6fzk"] Oct 31 00:27:02.390931 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:02.390904 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-e8ba06-bb9fb667f-x6fzk" Oct 31 00:27:02.398869 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:02.398841 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/207c55f2-0d5c-497c-b021-c6ba4939370b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-e8ba06-bb9fb667f-x6fzk\" (UID: \"207c55f2-0d5c-497c-b021-c6ba4939370b\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-e8ba06-bb9fb667f-x6fzk" Oct 31 00:27:02.399041 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:02.398878 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jx7sm\" (UniqueName: \"kubernetes.io/projected/207c55f2-0d5c-497c-b021-c6ba4939370b-kube-api-access-jx7sm\") pod \"payload-processing-e2e-worker-w2-e8ba06-bb9fb667f-x6fzk\" (UID: \"207c55f2-0d5c-497c-b021-c6ba4939370b\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-e8ba06-bb9fb667f-x6fzk" Oct 31 00:27:02.399041 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:02.398912 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06704c9d-24a7-4e63-bdd3-b7ee85866fbf-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-e8ba06-7b548bc5f-m9r6m\" (UID: \"06704c9d-24a7-4e63-bdd3-b7ee85866fbf\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e8ba06-7b548bc5f-m9r6m" Oct 31 00:27:02.399041 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:02.398961 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkckd\" (UniqueName: \"kubernetes.io/projected/06704c9d-24a7-4e63-bdd3-b7ee85866fbf-kube-api-access-tkckd\") pod \"payload-pre-processing-e2e-worker-w2-e8ba06-7b548bc5f-m9r6m\" (UID: \"06704c9d-24a7-4e63-bdd3-b7ee85866fbf\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e8ba06-7b548bc5f-m9r6m" Oct 31 00:27:02.399859 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:02.399813 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06704c9d-24a7-4e63-bdd3-b7ee85866fbf-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-e8ba06-7b548bc5f-m9r6m\" (UID: \"06704c9d-24a7-4e63-bdd3-b7ee85866fbf\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e8ba06-7b548bc5f-m9r6m" Oct 31 00:27:02.419683 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:02.419648 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-e8ba06-bb9fb667f-x6fzk"] Oct 31 00:27:02.461544 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:02.461504 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkckd\" (UniqueName: \"kubernetes.io/projected/06704c9d-24a7-4e63-bdd3-b7ee85866fbf-kube-api-access-tkckd\") pod \"payload-pre-processing-e2e-worker-w2-e8ba06-7b548bc5f-m9r6m\" (UID: \"06704c9d-24a7-4e63-bdd3-b7ee85866fbf\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e8ba06-7b548bc5f-m9r6m" Oct 31 00:27:02.499852 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:02.499755 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/207c55f2-0d5c-497c-b021-c6ba4939370b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-e8ba06-bb9fb667f-x6fzk\" (UID: \"207c55f2-0d5c-497c-b021-c6ba4939370b\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-e8ba06-bb9fb667f-x6fzk" Oct 31 00:27:02.499852 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:02.499803 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jx7sm\" (UniqueName: \"kubernetes.io/projected/207c55f2-0d5c-497c-b021-c6ba4939370b-kube-api-access-jx7sm\") pod \"payload-processing-e2e-worker-w2-e8ba06-bb9fb667f-x6fzk\" (UID: \"207c55f2-0d5c-497c-b021-c6ba4939370b\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-e8ba06-bb9fb667f-x6fzk" Oct 31 00:27:02.500501 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:02.500481 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/207c55f2-0d5c-497c-b021-c6ba4939370b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-e8ba06-bb9fb667f-x6fzk\" (UID: \"207c55f2-0d5c-497c-b021-c6ba4939370b\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-e8ba06-bb9fb667f-x6fzk" Oct 31 00:27:02.531955 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:02.531920 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e8ba06-7b548bc5f-m9r6m" Oct 31 00:27:02.562637 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:02.562610 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jx7sm\" (UniqueName: \"kubernetes.io/projected/207c55f2-0d5c-497c-b021-c6ba4939370b-kube-api-access-jx7sm\") pod \"payload-processing-e2e-worker-w2-e8ba06-bb9fb667f-x6fzk\" (UID: \"207c55f2-0d5c-497c-b021-c6ba4939370b\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-e8ba06-bb9fb667f-x6fzk" Oct 31 00:27:02.707775 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:02.707746 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-e8ba06-bb9fb667f-x6fzk" Oct 31 00:27:02.929965 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:27:02.929934 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06704c9d_24a7_4e63_bdd3_b7ee85866fbf.slice/crio-753a75e65d00462b2fbf77c8b79f49a15d2b5fc6d3383f9d688dbc503ef2c511 WatchSource:0}: Error finding container 753a75e65d00462b2fbf77c8b79f49a15d2b5fc6d3383f9d688dbc503ef2c511: Status 404 returned error can't find the container with id 753a75e65d00462b2fbf77c8b79f49a15d2b5fc6d3383f9d688dbc503ef2c511 Oct 31 00:27:02.932222 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:02.932199 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-e8ba06-7b548bc5f-m9r6m"] Oct 31 00:27:03.093207 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:27:03.093181 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod207c55f2_0d5c_497c_b021_c6ba4939370b.slice/crio-4c59f8ad7928bc750731457b3190269fb3bdfcb3c6e0e672bf1690a8c1cdc1e4 WatchSource:0}: Error finding container 4c59f8ad7928bc750731457b3190269fb3bdfcb3c6e0e672bf1690a8c1cdc1e4: Status 404 returned error can't find the container with id 4c59f8ad7928bc750731457b3190269fb3bdfcb3c6e0e672bf1690a8c1cdc1e4 Oct 31 00:27:03.094206 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:03.094096 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-e8ba06-bb9fb667f-x6fzk"] Oct 31 00:27:03.804679 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:03.804638 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e8ba06-7b548bc5f-m9r6m" event={"ID":"06704c9d-24a7-4e63-bdd3-b7ee85866fbf","Type":"ContainerStarted","Data":"753a75e65d00462b2fbf77c8b79f49a15d2b5fc6d3383f9d688dbc503ef2c511"} Oct 31 00:27:03.805932 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:03.805902 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-e8ba06-bb9fb667f-x6fzk" event={"ID":"207c55f2-0d5c-497c-b021-c6ba4939370b","Type":"ContainerStarted","Data":"4c59f8ad7928bc750731457b3190269fb3bdfcb3c6e0e672bf1690a8c1cdc1e4"} Oct 31 00:27:05.900010 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:27:05.899964 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 00:27:06.847934 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:06.847601 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-e8ba06-bb9fb667f-x6fzk" event={"ID":"207c55f2-0d5c-497c-b021-c6ba4939370b","Type":"ContainerStarted","Data":"924123f35d24276665ed606867ca065a9f35f091fa65355a50a823c42cb1d316"} Oct 31 00:27:06.847934 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:06.847691 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-e8ba06-bb9fb667f-x6fzk" Oct 31 00:27:06.850346 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:06.849958 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e8ba06-7b548bc5f-m9r6m" event={"ID":"06704c9d-24a7-4e63-bdd3-b7ee85866fbf","Type":"ContainerStarted","Data":"c7ad93d2942a5d855c70743f3b243a5ccbfe1d8a5b8e0972bed3dee955d231f6"} Oct 31 00:27:06.850346 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:06.850236 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e8ba06-7b548bc5f-m9r6m" Oct 31 00:27:06.978944 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:06.978889 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-e8ba06-bb9fb667f-x6fzk" podStartSLOduration=1.9784517 podStartE2EDuration="4.978875247s" totalImagesPullingTime="3.000423547s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:27:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:27:06.977610516 +0000 UTC m=+719.448780266" watchObservedRunningTime="2026-10-31 00:27:06.978875247 +0000 UTC m=+719.450044997" Oct 31 00:27:07.105619 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:07.105517 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e8ba06-7b548bc5f-m9r6m" podStartSLOduration=1.947169717 podStartE2EDuration="5.105502578s" totalImagesPullingTime="3.158332861s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:27:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:27:07.105167722 +0000 UTC m=+719.576337471" watchObservedRunningTime="2026-10-31 00:27:07.105502578 +0000 UTC m=+719.576672327" Oct 31 00:27:07.148967 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:07.148940 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-1e5e75-ff6df9bb9-d888l"] Oct 31 00:27:07.152668 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:07.152642 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-1e5e75-ff6df9bb9-d888l" Oct 31 00:27:07.213319 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:07.213284 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-1e5e75-ff6df9bb9-d888l"] Oct 31 00:27:07.214347 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:07.214323 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-1e5e75\"" Oct 31 00:27:07.214516 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:07.214487 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-1e5e75-dockercfg-2p47m\"" Oct 31 00:27:07.250310 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:07.250276 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/795c26a3-f65f-40f0-9cd6-503ab681a866-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-1e5e75-ff6df9bb9-d888l\" (UID: \"795c26a3-f65f-40f0-9cd6-503ab681a866\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-1e5e75-ff6df9bb9-d888l" Oct 31 00:27:07.250498 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:07.250322 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcv6x\" (UniqueName: \"kubernetes.io/projected/795c26a3-f65f-40f0-9cd6-503ab681a866-kube-api-access-jcv6x\") pod \"payload-processing-e2e-worker-w3-1e5e75-ff6df9bb9-d888l\" (UID: \"795c26a3-f65f-40f0-9cd6-503ab681a866\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-1e5e75-ff6df9bb9-d888l" Oct 31 00:27:07.352102 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:07.351700 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcv6x\" (UniqueName: \"kubernetes.io/projected/795c26a3-f65f-40f0-9cd6-503ab681a866-kube-api-access-jcv6x\") pod \"payload-processing-e2e-worker-w3-1e5e75-ff6df9bb9-d888l\" (UID: \"795c26a3-f65f-40f0-9cd6-503ab681a866\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-1e5e75-ff6df9bb9-d888l" Oct 31 00:27:07.352433 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:07.352293 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/795c26a3-f65f-40f0-9cd6-503ab681a866-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-1e5e75-ff6df9bb9-d888l\" (UID: \"795c26a3-f65f-40f0-9cd6-503ab681a866\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-1e5e75-ff6df9bb9-d888l" Oct 31 00:27:07.353249 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:07.353228 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/795c26a3-f65f-40f0-9cd6-503ab681a866-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-1e5e75-ff6df9bb9-d888l\" (UID: \"795c26a3-f65f-40f0-9cd6-503ab681a866\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-1e5e75-ff6df9bb9-d888l" Oct 31 00:27:07.418515 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:07.418441 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcv6x\" (UniqueName: \"kubernetes.io/projected/795c26a3-f65f-40f0-9cd6-503ab681a866-kube-api-access-jcv6x\") pod \"payload-processing-e2e-worker-w3-1e5e75-ff6df9bb9-d888l\" (UID: \"795c26a3-f65f-40f0-9cd6-503ab681a866\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-1e5e75-ff6df9bb9-d888l" Oct 31 00:27:07.465462 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:07.465427 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-1e5e75-ff6df9bb9-d888l" Oct 31 00:27:07.866765 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:07.866735 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-1e5e75-ff6df9bb9-d888l"] Oct 31 00:27:08.862126 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:08.862087 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-1e5e75-ff6df9bb9-d888l" event={"ID":"795c26a3-f65f-40f0-9cd6-503ab681a866","Type":"ContainerStarted","Data":"d5a9a484f5eb5ed7a74f93d682c027ec376ca05bf3215dc72e775878918b66f3"} Oct 31 00:27:08.862672 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:08.862151 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-1e5e75-ff6df9bb9-d888l" event={"ID":"795c26a3-f65f-40f0-9cd6-503ab681a866","Type":"ContainerStarted","Data":"6828b5ac8b242aa183bdc0c2415b2de10679848aa9e5f1ad54b055690a7b1be3"} Oct 31 00:27:08.862672 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:08.862197 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-1e5e75-ff6df9bb9-d888l" Oct 31 00:27:08.988152 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:08.988101 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-1e5e75-ff6df9bb9-d888l" podStartSLOduration=1.988089377 podStartE2EDuration="1.988089377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:27:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:27:08.986564339 +0000 UTC m=+721.457734089" watchObservedRunningTime="2026-10-31 00:27:08.988089377 +0000 UTC m=+721.459259125" Oct 31 00:27:09.456037 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:09.456002 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-a6f4a5-7666bb86c-glmj2"] Oct 31 00:27:09.459838 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:09.459816 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a6f4a5-7666bb86c-glmj2" Oct 31 00:27:09.516849 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:09.516803 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-a6f4a5-dockercfg-bdfk2\"" Oct 31 00:27:09.517455 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:09.517429 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-a6f4a5-7666bb86c-glmj2"] Oct 31 00:27:09.531577 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:09.531543 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-a6f4a5\"" Oct 31 00:27:09.574198 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:09.574161 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91cb96ab-e55e-463f-9fe0-059e65184a2c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-a6f4a5-7666bb86c-glmj2\" (UID: \"91cb96ab-e55e-463f-9fe0-059e65184a2c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a6f4a5-7666bb86c-glmj2" Oct 31 00:27:09.574423 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:09.574329 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tvjl\" (UniqueName: \"kubernetes.io/projected/91cb96ab-e55e-463f-9fe0-059e65184a2c-kube-api-access-9tvjl\") pod \"payload-pre-processing-e2e-worker-w6-a6f4a5-7666bb86c-glmj2\" (UID: \"91cb96ab-e55e-463f-9fe0-059e65184a2c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a6f4a5-7666bb86c-glmj2" Oct 31 00:27:09.619031 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:09.618996 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-a6f4a5-7b4d7b4dcc-v2hts"] Oct 31 00:27:09.622726 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:09.622701 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-a6f4a5-7b4d7b4dcc-v2hts" Oct 31 00:27:09.675625 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:09.675588 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tvjl\" (UniqueName: \"kubernetes.io/projected/91cb96ab-e55e-463f-9fe0-059e65184a2c-kube-api-access-9tvjl\") pod \"payload-pre-processing-e2e-worker-w6-a6f4a5-7666bb86c-glmj2\" (UID: \"91cb96ab-e55e-463f-9fe0-059e65184a2c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a6f4a5-7666bb86c-glmj2" Oct 31 00:27:09.675883 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:09.675640 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bcm2\" (UniqueName: \"kubernetes.io/projected/37865b42-ea0f-4744-ab63-9dbc9b7f74b1-kube-api-access-9bcm2\") pod \"payload-processing-e2e-worker-w6-a6f4a5-7b4d7b4dcc-v2hts\" (UID: \"37865b42-ea0f-4744-ab63-9dbc9b7f74b1\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-a6f4a5-7b4d7b4dcc-v2hts" Oct 31 00:27:09.675883 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:09.675780 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37865b42-ea0f-4744-ab63-9dbc9b7f74b1-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-a6f4a5-7b4d7b4dcc-v2hts\" (UID: \"37865b42-ea0f-4744-ab63-9dbc9b7f74b1\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-a6f4a5-7b4d7b4dcc-v2hts" Oct 31 00:27:09.675883 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:09.675849 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91cb96ab-e55e-463f-9fe0-059e65184a2c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-a6f4a5-7666bb86c-glmj2\" (UID: \"91cb96ab-e55e-463f-9fe0-059e65184a2c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a6f4a5-7666bb86c-glmj2" Oct 31 00:27:09.676878 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:09.676850 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91cb96ab-e55e-463f-9fe0-059e65184a2c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-a6f4a5-7666bb86c-glmj2\" (UID: \"91cb96ab-e55e-463f-9fe0-059e65184a2c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a6f4a5-7666bb86c-glmj2" Oct 31 00:27:09.678314 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:09.678291 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-a6f4a5-7b4d7b4dcc-v2hts"] Oct 31 00:27:09.740868 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:09.740774 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tvjl\" (UniqueName: \"kubernetes.io/projected/91cb96ab-e55e-463f-9fe0-059e65184a2c-kube-api-access-9tvjl\") pod \"payload-pre-processing-e2e-worker-w6-a6f4a5-7666bb86c-glmj2\" (UID: \"91cb96ab-e55e-463f-9fe0-059e65184a2c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a6f4a5-7666bb86c-glmj2" Oct 31 00:27:09.773853 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:09.773804 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a6f4a5-7666bb86c-glmj2" Oct 31 00:27:09.777341 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:09.777294 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37865b42-ea0f-4744-ab63-9dbc9b7f74b1-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-a6f4a5-7b4d7b4dcc-v2hts\" (UID: \"37865b42-ea0f-4744-ab63-9dbc9b7f74b1\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-a6f4a5-7b4d7b4dcc-v2hts" Oct 31 00:27:09.777554 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:09.777488 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bcm2\" (UniqueName: \"kubernetes.io/projected/37865b42-ea0f-4744-ab63-9dbc9b7f74b1-kube-api-access-9bcm2\") pod \"payload-processing-e2e-worker-w6-a6f4a5-7b4d7b4dcc-v2hts\" (UID: \"37865b42-ea0f-4744-ab63-9dbc9b7f74b1\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-a6f4a5-7b4d7b4dcc-v2hts" Oct 31 00:27:09.778886 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:09.778858 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37865b42-ea0f-4744-ab63-9dbc9b7f74b1-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-a6f4a5-7b4d7b4dcc-v2hts\" (UID: \"37865b42-ea0f-4744-ab63-9dbc9b7f74b1\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-a6f4a5-7b4d7b4dcc-v2hts" Oct 31 00:27:09.840977 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:09.840946 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bcm2\" (UniqueName: \"kubernetes.io/projected/37865b42-ea0f-4744-ab63-9dbc9b7f74b1-kube-api-access-9bcm2\") pod \"payload-processing-e2e-worker-w6-a6f4a5-7b4d7b4dcc-v2hts\" (UID: \"37865b42-ea0f-4744-ab63-9dbc9b7f74b1\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-a6f4a5-7b4d7b4dcc-v2hts" Oct 31 00:27:09.945566 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:09.945528 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-a6f4a5-7b4d7b4dcc-v2hts" Oct 31 00:27:10.147456 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:27:10.147425 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91cb96ab_e55e_463f_9fe0_059e65184a2c.slice/crio-ddf9fe68f8abf47d3cb3cc8d5a48b70d927a8426631e4d8f62b221bb9d5a3342 WatchSource:0}: Error finding container ddf9fe68f8abf47d3cb3cc8d5a48b70d927a8426631e4d8f62b221bb9d5a3342: Status 404 returned error can't find the container with id ddf9fe68f8abf47d3cb3cc8d5a48b70d927a8426631e4d8f62b221bb9d5a3342 Oct 31 00:27:10.149961 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:10.149938 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-a6f4a5-7666bb86c-glmj2"] Oct 31 00:27:10.316309 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:27:10.316282 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37865b42_ea0f_4744_ab63_9dbc9b7f74b1.slice/crio-626e7bdc6144bc68cbfd611456fbebd0124d0f86806d10239747ed076fd151b4 WatchSource:0}: Error finding container 626e7bdc6144bc68cbfd611456fbebd0124d0f86806d10239747ed076fd151b4: Status 404 returned error can't find the container with id 626e7bdc6144bc68cbfd611456fbebd0124d0f86806d10239747ed076fd151b4 Oct 31 00:27:10.317508 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:10.317484 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-a6f4a5-7b4d7b4dcc-v2hts"] Oct 31 00:27:10.880038 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:10.879997 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-a6f4a5-7b4d7b4dcc-v2hts" event={"ID":"37865b42-ea0f-4744-ab63-9dbc9b7f74b1","Type":"ContainerStarted","Data":"0c2a1b91b0df790654027c7e739688f777a630888871a77c41e3fa155ef1407f"} Oct 31 00:27:10.880244 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:10.880072 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-a6f4a5-7b4d7b4dcc-v2hts" event={"ID":"37865b42-ea0f-4744-ab63-9dbc9b7f74b1","Type":"ContainerStarted","Data":"626e7bdc6144bc68cbfd611456fbebd0124d0f86806d10239747ed076fd151b4"} Oct 31 00:27:10.880244 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:10.880123 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-a6f4a5-7b4d7b4dcc-v2hts" Oct 31 00:27:10.881729 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:10.881706 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a6f4a5-7666bb86c-glmj2" event={"ID":"91cb96ab-e55e-463f-9fe0-059e65184a2c","Type":"ContainerStarted","Data":"1383fa36b66f6c16b4a2c7b462b5504bf6c56f6de4fa8d25b95849d2ff1a4800"} Oct 31 00:27:10.881850 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:10.881743 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a6f4a5-7666bb86c-glmj2" event={"ID":"91cb96ab-e55e-463f-9fe0-059e65184a2c","Type":"ContainerStarted","Data":"ddf9fe68f8abf47d3cb3cc8d5a48b70d927a8426631e4d8f62b221bb9d5a3342"} Oct 31 00:27:10.881850 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:10.881762 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a6f4a5-7666bb86c-glmj2" Oct 31 00:27:10.999970 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:10.999918 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-a6f4a5-7b4d7b4dcc-v2hts" podStartSLOduration=1.999903701 podStartE2EDuration="1.999903701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:27:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:27:10.998255745 +0000 UTC m=+723.469425493" watchObservedRunningTime="2026-10-31 00:27:10.999903701 +0000 UTC m=+723.471073450" Oct 31 00:27:12.195207 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:12.195145 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a6f4a5-7666bb86c-glmj2" podStartSLOduration=3.195121881 podStartE2EDuration="3.195121881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:27:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:27:11.131663608 +0000 UTC m=+723.602833358" watchObservedRunningTime="2026-10-31 00:27:12.195121881 +0000 UTC m=+724.666291631" Oct 31 00:27:12.196640 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:12.196622 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-79tk8"] Oct 31 00:27:16.630552 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:16.630452 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-412743-6d5cb5bd7f-pzb7l"] Oct 31 00:27:16.634603 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:16.634579 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-412743-6d5cb5bd7f-pzb7l" Oct 31 00:27:16.710962 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:16.699021 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-412743-6d5cb5bd7f-pzb7l"] Oct 31 00:27:16.747452 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:16.747415 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-412743\"" Oct 31 00:27:16.747655 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:16.747509 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-412743-dockercfg-58jvh\"" Oct 31 00:27:16.756872 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:16.756836 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vbf4\" (UniqueName: \"kubernetes.io/projected/9dbd9108-ac33-45a0-9eed-ef9b81755398-kube-api-access-9vbf4\") pod \"payload-pre-processing-e2e-worker-w5-412743-6d5cb5bd7f-pzb7l\" (UID: \"9dbd9108-ac33-45a0-9eed-ef9b81755398\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-412743-6d5cb5bd7f-pzb7l" Oct 31 00:27:16.757056 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:16.756922 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9dbd9108-ac33-45a0-9eed-ef9b81755398-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-412743-6d5cb5bd7f-pzb7l\" (UID: \"9dbd9108-ac33-45a0-9eed-ef9b81755398\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-412743-6d5cb5bd7f-pzb7l" Oct 31 00:27:16.771060 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:16.771028 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-412743-688d66585f-vk5r7"] Oct 31 00:27:16.774646 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:16.774625 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-412743-688d66585f-vk5r7" Oct 31 00:27:16.829647 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:16.829611 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-412743-688d66585f-vk5r7"] Oct 31 00:27:16.858520 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:16.858476 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vbf4\" (UniqueName: \"kubernetes.io/projected/9dbd9108-ac33-45a0-9eed-ef9b81755398-kube-api-access-9vbf4\") pod \"payload-pre-processing-e2e-worker-w5-412743-6d5cb5bd7f-pzb7l\" (UID: \"9dbd9108-ac33-45a0-9eed-ef9b81755398\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-412743-6d5cb5bd7f-pzb7l" Oct 31 00:27:16.858520 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:16.858526 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a99cbf31-f3f2-4d16-a665-58f0f5af77f0-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-412743-688d66585f-vk5r7\" (UID: \"a99cbf31-f3f2-4d16-a665-58f0f5af77f0\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-412743-688d66585f-vk5r7" Oct 31 00:27:16.858789 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:16.858574 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9dbd9108-ac33-45a0-9eed-ef9b81755398-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-412743-6d5cb5bd7f-pzb7l\" (UID: \"9dbd9108-ac33-45a0-9eed-ef9b81755398\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-412743-6d5cb5bd7f-pzb7l" Oct 31 00:27:16.858789 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:16.858603 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79v4x\" (UniqueName: \"kubernetes.io/projected/a99cbf31-f3f2-4d16-a665-58f0f5af77f0-kube-api-access-79v4x\") pod \"payload-processing-e2e-worker-w5-412743-688d66585f-vk5r7\" (UID: \"a99cbf31-f3f2-4d16-a665-58f0f5af77f0\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-412743-688d66585f-vk5r7" Oct 31 00:27:16.859318 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:16.859299 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9dbd9108-ac33-45a0-9eed-ef9b81755398-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-412743-6d5cb5bd7f-pzb7l\" (UID: \"9dbd9108-ac33-45a0-9eed-ef9b81755398\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-412743-6d5cb5bd7f-pzb7l" Oct 31 00:27:16.923206 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:16.923129 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vbf4\" (UniqueName: \"kubernetes.io/projected/9dbd9108-ac33-45a0-9eed-ef9b81755398-kube-api-access-9vbf4\") pod \"payload-pre-processing-e2e-worker-w5-412743-6d5cb5bd7f-pzb7l\" (UID: \"9dbd9108-ac33-45a0-9eed-ef9b81755398\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-412743-6d5cb5bd7f-pzb7l" Oct 31 00:27:16.948108 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:16.948067 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-412743-6d5cb5bd7f-pzb7l" Oct 31 00:27:16.959626 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:16.959594 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a99cbf31-f3f2-4d16-a665-58f0f5af77f0-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-412743-688d66585f-vk5r7\" (UID: \"a99cbf31-f3f2-4d16-a665-58f0f5af77f0\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-412743-688d66585f-vk5r7" Oct 31 00:27:16.959760 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:16.959688 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79v4x\" (UniqueName: \"kubernetes.io/projected/a99cbf31-f3f2-4d16-a665-58f0f5af77f0-kube-api-access-79v4x\") pod \"payload-processing-e2e-worker-w5-412743-688d66585f-vk5r7\" (UID: \"a99cbf31-f3f2-4d16-a665-58f0f5af77f0\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-412743-688d66585f-vk5r7" Oct 31 00:27:16.960491 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:16.960465 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a99cbf31-f3f2-4d16-a665-58f0f5af77f0-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-412743-688d66585f-vk5r7\" (UID: \"a99cbf31-f3f2-4d16-a665-58f0f5af77f0\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-412743-688d66585f-vk5r7" Oct 31 00:27:17.022503 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:17.022438 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79v4x\" (UniqueName: \"kubernetes.io/projected/a99cbf31-f3f2-4d16-a665-58f0f5af77f0-kube-api-access-79v4x\") pod \"payload-processing-e2e-worker-w5-412743-688d66585f-vk5r7\" (UID: \"a99cbf31-f3f2-4d16-a665-58f0f5af77f0\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-412743-688d66585f-vk5r7" Oct 31 00:27:17.089527 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:17.089482 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-412743-688d66585f-vk5r7" Oct 31 00:27:17.318310 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:27:17.318281 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dbd9108_ac33_45a0_9eed_ef9b81755398.slice/crio-18353d6fa5213acc2dee87b0262306f9097e146d70c1e0891135e82b948eddab WatchSource:0}: Error finding container 18353d6fa5213acc2dee87b0262306f9097e146d70c1e0891135e82b948eddab: Status 404 returned error can't find the container with id 18353d6fa5213acc2dee87b0262306f9097e146d70c1e0891135e82b948eddab Oct 31 00:27:17.319760 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:17.319733 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-412743-6d5cb5bd7f-pzb7l"] Oct 31 00:27:17.465586 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:27:17.465557 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda99cbf31_f3f2_4d16_a665_58f0f5af77f0.slice/crio-7b37376c0e917b8e04dcfa56028b71bda9770e1f3e0725d89b826da1d5294220 WatchSource:0}: Error finding container 7b37376c0e917b8e04dcfa56028b71bda9770e1f3e0725d89b826da1d5294220: Status 404 returned error can't find the container with id 7b37376c0e917b8e04dcfa56028b71bda9770e1f3e0725d89b826da1d5294220 Oct 31 00:27:17.467062 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:17.467038 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-412743-688d66585f-vk5r7"] Oct 31 00:27:17.860122 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:17.860098 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-e8ba06-bb9fb667f-x6fzk" Oct 31 00:27:17.860602 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:17.860162 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e8ba06-7b548bc5f-m9r6m" Oct 31 00:27:17.927433 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:17.927394 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-412743-6d5cb5bd7f-pzb7l" event={"ID":"9dbd9108-ac33-45a0-9eed-ef9b81755398","Type":"ContainerStarted","Data":"c28d50fb6af5559cb39a55913a3db1d9d96be0c43e6c5ab96f9c35b102cbecf4"} Oct 31 00:27:17.927621 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:17.927453 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-412743-6d5cb5bd7f-pzb7l" event={"ID":"9dbd9108-ac33-45a0-9eed-ef9b81755398","Type":"ContainerStarted","Data":"18353d6fa5213acc2dee87b0262306f9097e146d70c1e0891135e82b948eddab"} Oct 31 00:27:17.927621 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:17.927478 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-412743-6d5cb5bd7f-pzb7l" Oct 31 00:27:17.929180 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:17.929152 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-412743-688d66585f-vk5r7" event={"ID":"a99cbf31-f3f2-4d16-a665-58f0f5af77f0","Type":"ContainerStarted","Data":"66d2dcee34673511c9d5bdcc236e9cac8f2e3852170c01d565e384bf000e246a"} Oct 31 00:27:17.929316 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:17.929210 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-412743-688d66585f-vk5r7" Oct 31 00:27:17.929316 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:17.929238 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-412743-688d66585f-vk5r7" event={"ID":"a99cbf31-f3f2-4d16-a665-58f0f5af77f0","Type":"ContainerStarted","Data":"7b37376c0e917b8e04dcfa56028b71bda9770e1f3e0725d89b826da1d5294220"} Oct 31 00:27:18.335545 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:18.335471 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-412743-6d5cb5bd7f-pzb7l" podStartSLOduration=2.33544943 podStartE2EDuration="2.33544943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:27:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:27:18.335146748 +0000 UTC m=+730.806316498" watchObservedRunningTime="2026-10-31 00:27:18.33544943 +0000 UTC m=+730.806619179" Oct 31 00:27:18.337665 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:18.337596 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w5-412743-688d66585f-vk5r7" podStartSLOduration=2.337577977 podStartE2EDuration="2.337577977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:27:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:27:18.216839025 +0000 UTC m=+730.688008773" watchObservedRunningTime="2026-10-31 00:27:18.337577977 +0000 UTC m=+730.808747727" Oct 31 00:27:19.869876 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:19.869838 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-1e5e75-ff6df9bb9-d888l" Oct 31 00:27:21.024428 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:21.024392 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-79tk8"] Oct 31 00:27:21.344280 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:21.344187 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-da9d07-69bb55d779-hg9l2"] Oct 31 00:27:21.347935 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:21.347916 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-da9d07-69bb55d779-hg9l2" Oct 31 00:27:21.408436 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:21.408401 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6s8ml\" (UniqueName: \"kubernetes.io/projected/1b4de77a-88d7-466f-a8a4-37c224ab4a3c-kube-api-access-6s8ml\") pod \"payload-pre-processing-e2e-worker-w4-da9d07-69bb55d779-hg9l2\" (UID: \"1b4de77a-88d7-466f-a8a4-37c224ab4a3c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-da9d07-69bb55d779-hg9l2" Oct 31 00:27:21.408651 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:21.408523 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b4de77a-88d7-466f-a8a4-37c224ab4a3c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-da9d07-69bb55d779-hg9l2\" (UID: \"1b4de77a-88d7-466f-a8a4-37c224ab4a3c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-da9d07-69bb55d779-hg9l2" Oct 31 00:27:21.420674 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:21.420643 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-da9d07-69bb55d779-hg9l2"] Oct 31 00:27:21.459191 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:21.459145 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-da9d07\"" Oct 31 00:27:21.459191 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:21.459165 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-da9d07-dockercfg-x25ms\"" Oct 31 00:27:21.500870 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:21.500842 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-da9d07-5848fd449-s4wwm"] Oct 31 00:27:21.506263 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:21.506237 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-da9d07-5848fd449-s4wwm" Oct 31 00:27:21.509108 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:21.509082 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b4de77a-88d7-466f-a8a4-37c224ab4a3c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-da9d07-69bb55d779-hg9l2\" (UID: \"1b4de77a-88d7-466f-a8a4-37c224ab4a3c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-da9d07-69bb55d779-hg9l2" Oct 31 00:27:21.509246 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:21.509185 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6s8ml\" (UniqueName: \"kubernetes.io/projected/1b4de77a-88d7-466f-a8a4-37c224ab4a3c-kube-api-access-6s8ml\") pod \"payload-pre-processing-e2e-worker-w4-da9d07-69bb55d779-hg9l2\" (UID: \"1b4de77a-88d7-466f-a8a4-37c224ab4a3c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-da9d07-69bb55d779-hg9l2" Oct 31 00:27:21.509991 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:21.509966 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b4de77a-88d7-466f-a8a4-37c224ab4a3c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-da9d07-69bb55d779-hg9l2\" (UID: \"1b4de77a-88d7-466f-a8a4-37c224ab4a3c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-da9d07-69bb55d779-hg9l2" Oct 31 00:27:21.572840 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:21.561092 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-da9d07-5848fd449-s4wwm"] Oct 31 00:27:21.588978 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:21.588935 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6s8ml\" (UniqueName: \"kubernetes.io/projected/1b4de77a-88d7-466f-a8a4-37c224ab4a3c-kube-api-access-6s8ml\") pod \"payload-pre-processing-e2e-worker-w4-da9d07-69bb55d779-hg9l2\" (UID: \"1b4de77a-88d7-466f-a8a4-37c224ab4a3c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-da9d07-69bb55d779-hg9l2" Oct 31 00:27:21.610670 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:21.610190 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc1b882b-b51a-4d28-a9ee-35362dd92d69-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-da9d07-5848fd449-s4wwm\" (UID: \"bc1b882b-b51a-4d28-a9ee-35362dd92d69\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-da9d07-5848fd449-s4wwm" Oct 31 00:27:21.610670 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:21.610269 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdlk6\" (UniqueName: \"kubernetes.io/projected/bc1b882b-b51a-4d28-a9ee-35362dd92d69-kube-api-access-rdlk6\") pod \"payload-processing-e2e-worker-w4-da9d07-5848fd449-s4wwm\" (UID: \"bc1b882b-b51a-4d28-a9ee-35362dd92d69\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-da9d07-5848fd449-s4wwm" Oct 31 00:27:21.663489 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:21.663450 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-da9d07-69bb55d779-hg9l2" Oct 31 00:27:21.711880 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:21.711838 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc1b882b-b51a-4d28-a9ee-35362dd92d69-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-da9d07-5848fd449-s4wwm\" (UID: \"bc1b882b-b51a-4d28-a9ee-35362dd92d69\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-da9d07-5848fd449-s4wwm" Oct 31 00:27:21.712046 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:21.711934 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdlk6\" (UniqueName: \"kubernetes.io/projected/bc1b882b-b51a-4d28-a9ee-35362dd92d69-kube-api-access-rdlk6\") pod \"payload-processing-e2e-worker-w4-da9d07-5848fd449-s4wwm\" (UID: \"bc1b882b-b51a-4d28-a9ee-35362dd92d69\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-da9d07-5848fd449-s4wwm" Oct 31 00:27:21.714758 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:21.714733 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc1b882b-b51a-4d28-a9ee-35362dd92d69-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-da9d07-5848fd449-s4wwm\" (UID: \"bc1b882b-b51a-4d28-a9ee-35362dd92d69\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-da9d07-5848fd449-s4wwm" Oct 31 00:27:21.775114 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:21.775067 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdlk6\" (UniqueName: \"kubernetes.io/projected/bc1b882b-b51a-4d28-a9ee-35362dd92d69-kube-api-access-rdlk6\") pod \"payload-processing-e2e-worker-w4-da9d07-5848fd449-s4wwm\" (UID: \"bc1b882b-b51a-4d28-a9ee-35362dd92d69\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-da9d07-5848fd449-s4wwm" Oct 31 00:27:21.823855 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:21.823817 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-da9d07-5848fd449-s4wwm" Oct 31 00:27:21.889686 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:21.889612 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-a6f4a5-7b4d7b4dcc-v2hts" Oct 31 00:27:21.889686 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:21.889685 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a6f4a5-7666bb86c-glmj2" Oct 31 00:27:22.037338 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:22.037305 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-da9d07-69bb55d779-hg9l2"] Oct 31 00:27:22.073016 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:27:22.072983 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b4de77a_88d7_466f_a8a4_37c224ab4a3c.slice/crio-ac620df1fa3e62df015b12095fd4fa332bf0e49ec55441a6889bdce7ae348894 WatchSource:0}: Error finding container ac620df1fa3e62df015b12095fd4fa332bf0e49ec55441a6889bdce7ae348894: Status 404 returned error can't find the container with id ac620df1fa3e62df015b12095fd4fa332bf0e49ec55441a6889bdce7ae348894 Oct 31 00:27:22.230236 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:27:22.230202 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc1b882b_b51a_4d28_a9ee_35362dd92d69.slice/crio-862ab2b8a85cdafc54e1333db78abf7056b30b930de2b3116028825629410af9 WatchSource:0}: Error finding container 862ab2b8a85cdafc54e1333db78abf7056b30b930de2b3116028825629410af9: Status 404 returned error can't find the container with id 862ab2b8a85cdafc54e1333db78abf7056b30b930de2b3116028825629410af9 Oct 31 00:27:22.231969 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:22.231941 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-da9d07-5848fd449-s4wwm"] Oct 31 00:27:22.977395 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:22.977334 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-da9d07-69bb55d779-hg9l2" event={"ID":"1b4de77a-88d7-466f-a8a4-37c224ab4a3c","Type":"ContainerStarted","Data":"9e0e113793b61d5918a3a4d25a7690379e8f177c50465b69b9ff577361cbdd12"} Oct 31 00:27:22.977600 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:22.977447 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-da9d07-69bb55d779-hg9l2" event={"ID":"1b4de77a-88d7-466f-a8a4-37c224ab4a3c","Type":"ContainerStarted","Data":"ac620df1fa3e62df015b12095fd4fa332bf0e49ec55441a6889bdce7ae348894"} Oct 31 00:27:22.977600 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:22.977486 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-da9d07-69bb55d779-hg9l2" Oct 31 00:27:22.978927 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:22.978900 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-da9d07-5848fd449-s4wwm" event={"ID":"bc1b882b-b51a-4d28-a9ee-35362dd92d69","Type":"ContainerStarted","Data":"b0bb94e6b83687089012e92190f7d97b981d645b7eb64472835acd134c959db8"} Oct 31 00:27:22.979052 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:22.978940 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-da9d07-5848fd449-s4wwm" event={"ID":"bc1b882b-b51a-4d28-a9ee-35362dd92d69","Type":"ContainerStarted","Data":"862ab2b8a85cdafc54e1333db78abf7056b30b930de2b3116028825629410af9"} Oct 31 00:27:22.979052 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:22.979029 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-da9d07-5848fd449-s4wwm" Oct 31 00:27:23.095864 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:23.095810 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-da9d07-69bb55d779-hg9l2" podStartSLOduration=2.095797618 podStartE2EDuration="2.095797618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:27:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:27:23.09466382 +0000 UTC m=+735.565833569" watchObservedRunningTime="2026-10-31 00:27:23.095797618 +0000 UTC m=+735.566967368" Oct 31 00:27:23.215477 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:23.215402 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-da9d07-5848fd449-s4wwm" podStartSLOduration=2.215380137 podStartE2EDuration="2.215380137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:27:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:27:23.214044259 +0000 UTC m=+735.685214008" watchObservedRunningTime="2026-10-31 00:27:23.215380137 +0000 UTC m=+735.686549879" Oct 31 00:27:23.764785 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:23.764752 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-101b9e-688fd6f5d7-7nrcm"] Oct 31 00:27:23.768328 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:23.768306 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-101b9e-688fd6f5d7-7nrcm" Oct 31 00:27:23.821989 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:23.821937 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-101b9e\"" Oct 31 00:27:23.822172 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:23.822022 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-101b9e-dockercfg-mmz4v\"" Oct 31 00:27:23.822605 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:23.822583 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-101b9e-688fd6f5d7-7nrcm"] Oct 31 00:27:23.835900 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:23.835715 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5wks\" (UniqueName: \"kubernetes.io/projected/38bd486f-e3b6-4d6e-8ca2-6bb7591b9ba3-kube-api-access-m5wks\") pod \"payload-pre-processing-e2e-worker-w0-101b9e-688fd6f5d7-7nrcm\" (UID: \"38bd486f-e3b6-4d6e-8ca2-6bb7591b9ba3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-101b9e-688fd6f5d7-7nrcm" Oct 31 00:27:23.835900 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:23.835863 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38bd486f-e3b6-4d6e-8ca2-6bb7591b9ba3-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-101b9e-688fd6f5d7-7nrcm\" (UID: \"38bd486f-e3b6-4d6e-8ca2-6bb7591b9ba3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-101b9e-688fd6f5d7-7nrcm" Oct 31 00:27:23.936910 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:23.936872 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38bd486f-e3b6-4d6e-8ca2-6bb7591b9ba3-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-101b9e-688fd6f5d7-7nrcm\" (UID: \"38bd486f-e3b6-4d6e-8ca2-6bb7591b9ba3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-101b9e-688fd6f5d7-7nrcm" Oct 31 00:27:23.937129 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:23.937001 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5wks\" (UniqueName: \"kubernetes.io/projected/38bd486f-e3b6-4d6e-8ca2-6bb7591b9ba3-kube-api-access-m5wks\") pod \"payload-pre-processing-e2e-worker-w0-101b9e-688fd6f5d7-7nrcm\" (UID: \"38bd486f-e3b6-4d6e-8ca2-6bb7591b9ba3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-101b9e-688fd6f5d7-7nrcm" Oct 31 00:27:23.949397 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:23.937840 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38bd486f-e3b6-4d6e-8ca2-6bb7591b9ba3-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-101b9e-688fd6f5d7-7nrcm\" (UID: \"38bd486f-e3b6-4d6e-8ca2-6bb7591b9ba3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-101b9e-688fd6f5d7-7nrcm" Oct 31 00:27:24.022028 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:24.021988 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5wks\" (UniqueName: \"kubernetes.io/projected/38bd486f-e3b6-4d6e-8ca2-6bb7591b9ba3-kube-api-access-m5wks\") pod \"payload-pre-processing-e2e-worker-w0-101b9e-688fd6f5d7-7nrcm\" (UID: \"38bd486f-e3b6-4d6e-8ca2-6bb7591b9ba3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-101b9e-688fd6f5d7-7nrcm" Oct 31 00:27:24.081918 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:24.081882 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-101b9e-688fd6f5d7-7nrcm" Oct 31 00:27:24.475842 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:27:24.475810 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38bd486f_e3b6_4d6e_8ca2_6bb7591b9ba3.slice/crio-878e7588f72a91ab37d334938f568904f73a9549c4abd20e93e642b550a52017 WatchSource:0}: Error finding container 878e7588f72a91ab37d334938f568904f73a9549c4abd20e93e642b550a52017: Status 404 returned error can't find the container with id 878e7588f72a91ab37d334938f568904f73a9549c4abd20e93e642b550a52017 Oct 31 00:27:24.477826 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:24.477800 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-101b9e-688fd6f5d7-7nrcm"] Oct 31 00:27:24.991266 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:24.991227 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-101b9e-688fd6f5d7-7nrcm" event={"ID":"38bd486f-e3b6-4d6e-8ca2-6bb7591b9ba3","Type":"ContainerStarted","Data":"52aae2a664052461da0e54e45e312a372bc73cf5d96aeeb28fb5a40434b06f65"} Oct 31 00:27:24.991493 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:24.991283 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-101b9e-688fd6f5d7-7nrcm" event={"ID":"38bd486f-e3b6-4d6e-8ca2-6bb7591b9ba3","Type":"ContainerStarted","Data":"878e7588f72a91ab37d334938f568904f73a9549c4abd20e93e642b550a52017"} Oct 31 00:27:24.991493 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:24.991312 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-101b9e-688fd6f5d7-7nrcm" Oct 31 00:27:25.124884 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:25.124818 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-101b9e-688fd6f5d7-7nrcm" podStartSLOduration=2.124801933 podStartE2EDuration="2.124801933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:27:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:27:25.12421562 +0000 UTC m=+737.595385369" watchObservedRunningTime="2026-10-31 00:27:25.124801933 +0000 UTC m=+737.595971685" Oct 31 00:27:28.937532 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:28.937501 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-412743-688d66585f-vk5r7" Oct 31 00:27:28.938007 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:28.937586 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-412743-6d5cb5bd7f-pzb7l" Oct 31 00:27:31.023338 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:31.023300 5222 generic.go:356] "Generic (PLEG): container finished" podID="2fe80038-c64b-4114-ac0f-337a29f20204" containerID="51103936220c641723e6faed9902d3b5829355f32f6ab65da69cb81f5503a3e1" exitCode=0 Oct 31 00:27:31.023849 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:31.023387 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-2jrfg" event={"ID":"2fe80038-c64b-4114-ac0f-337a29f20204","Type":"ContainerDied","Data":"51103936220c641723e6faed9902d3b5829355f32f6ab65da69cb81f5503a3e1"} Oct 31 00:27:31.023849 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:31.023778 5222 scope.go:122] "RemoveContainer" containerID="51103936220c641723e6faed9902d3b5829355f32f6ab65da69cb81f5503a3e1" Oct 31 00:27:31.025174 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:31.025145 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:27:31.025228 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:31.025214 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:27:31.025262 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:31.025253 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:27:32.030270 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:32.030234 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-2jrfg" event={"ID":"2fe80038-c64b-4114-ac0f-337a29f20204","Type":"ContainerStarted","Data":"fb22614abc89ee4bce35a3f610f5394eac15c2e48293e5d248f3b217798685ca"} Oct 31 00:27:32.153819 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:32.153785 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-2jrfg" Oct 31 00:27:32.154207 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:32.154100 5222 patch_prober.go:29] interesting pod/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-2jrfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Oct 31 00:27:32.154429 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:32.154403 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-2jrfg" podUID="2fe80038-c64b-4114-ac0f-337a29f20204" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Oct 31 00:27:33.137598 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:33.137562 5222 patch_prober.go:29] interesting pod/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-2jrfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Oct 31 00:27:33.159553 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:33.159504 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-2jrfg" podUID="2fe80038-c64b-4114-ac0f-337a29f20204" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Oct 31 00:27:33.987306 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:33.987278 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-da9d07-5848fd449-s4wwm" Oct 31 00:27:33.987503 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:33.987415 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-da9d07-69bb55d779-hg9l2" Oct 31 00:27:34.163406 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:34.159811 5222 patch_prober.go:29] interesting pod/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-2jrfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Oct 31 00:27:34.163406 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:34.159874 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-2jrfg" podUID="2fe80038-c64b-4114-ac0f-337a29f20204" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Oct 31 00:27:35.158396 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:35.143502 5222 patch_prober.go:29] interesting pod/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-2jrfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Oct 31 00:27:35.158396 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:35.143573 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-2jrfg" podUID="2fe80038-c64b-4114-ac0f-337a29f20204" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Oct 31 00:27:35.890982 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:27:35.890946 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 00:27:35.999893 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:35.999864 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-101b9e-688fd6f5d7-7nrcm" Oct 31 00:27:36.143625 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:36.143526 5222 patch_prober.go:29] interesting pod/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-2jrfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Oct 31 00:27:36.143875 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:36.143845 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-2jrfg" podUID="2fe80038-c64b-4114-ac0f-337a29f20204" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Oct 31 00:27:36.720804 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:36.720764 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-79tk8"] Oct 31 00:27:37.138103 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:37.138058 5222 patch_prober.go:29] interesting pod/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-2jrfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Oct 31 00:27:37.138527 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:37.138125 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-2jrfg" podUID="2fe80038-c64b-4114-ac0f-337a29f20204" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Oct 31 00:27:38.159397 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:38.153765 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-2jrfg" Oct 31 00:27:38.159397 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:38.155147 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-2jrfg" Oct 31 00:27:38.159397 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:38.158628 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-2jrfg" Oct 31 00:27:46.648792 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:46.648704 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-79tk8"] Oct 31 00:27:47.134426 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:47.134393 5222 generic.go:356] "Generic (PLEG): container finished" podID="f85c9ba2-c597-4786-93a1-930609ff510a" containerID="34125c454b0f15ed31fe01b3cb49afe5730ab4bc1d4edef5b8f7784a20daa122" exitCode=0 Oct 31 00:27:47.134627 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:47.134464 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-cml7r" event={"ID":"f85c9ba2-c597-4786-93a1-930609ff510a","Type":"ContainerDied","Data":"34125c454b0f15ed31fe01b3cb49afe5730ab4bc1d4edef5b8f7784a20daa122"} Oct 31 00:27:47.134890 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:47.134873 5222 scope.go:122] "RemoveContainer" containerID="34125c454b0f15ed31fe01b3cb49afe5730ab4bc1d4edef5b8f7784a20daa122" Oct 31 00:27:47.136125 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:47.136094 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:27:47.136235 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:47.136178 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:27:47.136235 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:47.136228 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:27:48.143816 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:48.143777 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-cml7r" event={"ID":"f85c9ba2-c597-4786-93a1-930609ff510a","Type":"ContainerStarted","Data":"1b5d3d06724d455f00d307b92c2c6a4b5554da8c4d8238c849ad38eec38a7fee"} Oct 31 00:27:48.146445 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:48.145981 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-cml7r" Oct 31 00:27:48.146445 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:48.146077 5222 patch_prober.go:29] interesting pod/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-cml7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Oct 31 00:27:48.146445 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:48.146208 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-cml7r" podUID="f85c9ba2-c597-4786-93a1-930609ff510a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Oct 31 00:27:49.160395 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:49.151810 5222 patch_prober.go:29] interesting pod/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-cml7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Oct 31 00:27:49.160395 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:49.151893 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-cml7r" podUID="f85c9ba2-c597-4786-93a1-930609ff510a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Oct 31 00:27:50.159394 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:50.158844 5222 patch_prober.go:29] interesting pod/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-cml7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Oct 31 00:27:50.159394 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:50.158906 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-cml7r" podUID="f85c9ba2-c597-4786-93a1-930609ff510a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Oct 31 00:27:51.145889 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:51.145844 5222 patch_prober.go:29] interesting pod/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-cml7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Oct 31 00:27:51.146576 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:51.146541 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-cml7r" podUID="f85c9ba2-c597-4786-93a1-930609ff510a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Oct 31 00:27:52.157912 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:52.157873 5222 patch_prober.go:29] interesting pod/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-cml7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Oct 31 00:27:52.158562 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:52.158531 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-cml7r" podUID="f85c9ba2-c597-4786-93a1-930609ff510a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Oct 31 00:27:53.146018 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:53.145965 5222 patch_prober.go:29] interesting pod/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-cml7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Oct 31 00:27:53.146265 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:53.146042 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-cml7r" podUID="f85c9ba2-c597-4786-93a1-930609ff510a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Oct 31 00:27:54.149466 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:54.149431 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-cml7r" Oct 31 00:27:54.149868 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:54.149713 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-cml7r" Oct 31 00:27:54.150515 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:54.150495 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-cml7r" Oct 31 00:27:56.416066 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:56.416029 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-79tk8"] Oct 31 00:27:56.966869 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:56.966835 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-1e5e75/simulator-w3-1e5e75-kserve-6cd984c46b-tjz8p"] Oct 31 00:27:56.973346 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:56.973325 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-1e5e75/simulator-w3-1e5e75-kserve-6cd984c46b-tjz8p" Oct 31 00:27:57.028946 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:57.028917 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-1e5e75/simulator-w3-1e5e75-kserve-6cd984c46b-tjz8p"] Oct 31 00:27:57.063265 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:57.063223 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6158bb1-fa2f-4d00-8159-d1620958c7c0-dshm\") pod \"simulator-w3-1e5e75-kserve-6cd984c46b-tjz8p\" (UID: \"a6158bb1-fa2f-4d00-8159-d1620958c7c0\") " pod="e2e-models-e2e-worker-w3-1e5e75/simulator-w3-1e5e75-kserve-6cd984c46b-tjz8p" Oct 31 00:27:57.063559 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:57.063536 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6158bb1-fa2f-4d00-8159-d1620958c7c0-tmp-dir\") pod \"simulator-w3-1e5e75-kserve-6cd984c46b-tjz8p\" (UID: \"a6158bb1-fa2f-4d00-8159-d1620958c7c0\") " pod="e2e-models-e2e-worker-w3-1e5e75/simulator-w3-1e5e75-kserve-6cd984c46b-tjz8p" Oct 31 00:27:57.063720 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:57.063706 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6158bb1-fa2f-4d00-8159-d1620958c7c0-home\") pod \"simulator-w3-1e5e75-kserve-6cd984c46b-tjz8p\" (UID: \"a6158bb1-fa2f-4d00-8159-d1620958c7c0\") " pod="e2e-models-e2e-worker-w3-1e5e75/simulator-w3-1e5e75-kserve-6cd984c46b-tjz8p" Oct 31 00:27:57.063836 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:57.063822 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6158bb1-fa2f-4d00-8159-d1620958c7c0-model-cache\") pod \"simulator-w3-1e5e75-kserve-6cd984c46b-tjz8p\" (UID: \"a6158bb1-fa2f-4d00-8159-d1620958c7c0\") " pod="e2e-models-e2e-worker-w3-1e5e75/simulator-w3-1e5e75-kserve-6cd984c46b-tjz8p" Oct 31 00:27:57.063925 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:57.063913 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6158bb1-fa2f-4d00-8159-d1620958c7c0-tls-certs\") pod \"simulator-w3-1e5e75-kserve-6cd984c46b-tjz8p\" (UID: \"a6158bb1-fa2f-4d00-8159-d1620958c7c0\") " pod="e2e-models-e2e-worker-w3-1e5e75/simulator-w3-1e5e75-kserve-6cd984c46b-tjz8p" Oct 31 00:27:57.064043 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:57.064029 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdw2b\" (UniqueName: \"kubernetes.io/projected/a6158bb1-fa2f-4d00-8159-d1620958c7c0-kube-api-access-vdw2b\") pod \"simulator-w3-1e5e75-kserve-6cd984c46b-tjz8p\" (UID: \"a6158bb1-fa2f-4d00-8159-d1620958c7c0\") " pod="e2e-models-e2e-worker-w3-1e5e75/simulator-w3-1e5e75-kserve-6cd984c46b-tjz8p" Oct 31 00:27:57.078217 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:57.078180 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-1e5e75\"/\"default-dockercfg-jzlwr\"" Oct 31 00:27:57.078217 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:57.078180 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-1e5e75\"/\"simulator-w3-1e5e75-kserve-self-signed-certs\"" Oct 31 00:27:57.079910 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:57.079891 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-1e5e75\"/\"openshift-service-ca.crt\"" Oct 31 00:27:57.079995 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:57.079920 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-1e5e75\"/\"kube-root-ca.crt\"" Oct 31 00:27:57.165151 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:57.165113 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6158bb1-fa2f-4d00-8159-d1620958c7c0-home\") pod \"simulator-w3-1e5e75-kserve-6cd984c46b-tjz8p\" (UID: \"a6158bb1-fa2f-4d00-8159-d1620958c7c0\") " pod="e2e-models-e2e-worker-w3-1e5e75/simulator-w3-1e5e75-kserve-6cd984c46b-tjz8p" Oct 31 00:27:57.165346 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:57.165171 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6158bb1-fa2f-4d00-8159-d1620958c7c0-model-cache\") pod \"simulator-w3-1e5e75-kserve-6cd984c46b-tjz8p\" (UID: \"a6158bb1-fa2f-4d00-8159-d1620958c7c0\") " pod="e2e-models-e2e-worker-w3-1e5e75/simulator-w3-1e5e75-kserve-6cd984c46b-tjz8p" Oct 31 00:27:57.165346 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:57.165197 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6158bb1-fa2f-4d00-8159-d1620958c7c0-tls-certs\") pod \"simulator-w3-1e5e75-kserve-6cd984c46b-tjz8p\" (UID: \"a6158bb1-fa2f-4d00-8159-d1620958c7c0\") " pod="e2e-models-e2e-worker-w3-1e5e75/simulator-w3-1e5e75-kserve-6cd984c46b-tjz8p" Oct 31 00:27:57.165346 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:57.165224 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdw2b\" (UniqueName: \"kubernetes.io/projected/a6158bb1-fa2f-4d00-8159-d1620958c7c0-kube-api-access-vdw2b\") pod \"simulator-w3-1e5e75-kserve-6cd984c46b-tjz8p\" (UID: \"a6158bb1-fa2f-4d00-8159-d1620958c7c0\") " pod="e2e-models-e2e-worker-w3-1e5e75/simulator-w3-1e5e75-kserve-6cd984c46b-tjz8p" Oct 31 00:27:57.165346 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:57.165336 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6158bb1-fa2f-4d00-8159-d1620958c7c0-dshm\") pod \"simulator-w3-1e5e75-kserve-6cd984c46b-tjz8p\" (UID: \"a6158bb1-fa2f-4d00-8159-d1620958c7c0\") " pod="e2e-models-e2e-worker-w3-1e5e75/simulator-w3-1e5e75-kserve-6cd984c46b-tjz8p" Oct 31 00:27:57.165618 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:57.165359 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6158bb1-fa2f-4d00-8159-d1620958c7c0-tmp-dir\") pod \"simulator-w3-1e5e75-kserve-6cd984c46b-tjz8p\" (UID: \"a6158bb1-fa2f-4d00-8159-d1620958c7c0\") " pod="e2e-models-e2e-worker-w3-1e5e75/simulator-w3-1e5e75-kserve-6cd984c46b-tjz8p" Oct 31 00:27:57.165618 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:57.165605 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6158bb1-fa2f-4d00-8159-d1620958c7c0-home\") pod \"simulator-w3-1e5e75-kserve-6cd984c46b-tjz8p\" (UID: \"a6158bb1-fa2f-4d00-8159-d1620958c7c0\") " pod="e2e-models-e2e-worker-w3-1e5e75/simulator-w3-1e5e75-kserve-6cd984c46b-tjz8p" Oct 31 00:27:57.165708 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:57.165631 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6158bb1-fa2f-4d00-8159-d1620958c7c0-model-cache\") pod \"simulator-w3-1e5e75-kserve-6cd984c46b-tjz8p\" (UID: \"a6158bb1-fa2f-4d00-8159-d1620958c7c0\") " pod="e2e-models-e2e-worker-w3-1e5e75/simulator-w3-1e5e75-kserve-6cd984c46b-tjz8p" Oct 31 00:27:57.165762 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:57.165745 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6158bb1-fa2f-4d00-8159-d1620958c7c0-tmp-dir\") pod \"simulator-w3-1e5e75-kserve-6cd984c46b-tjz8p\" (UID: \"a6158bb1-fa2f-4d00-8159-d1620958c7c0\") " pod="e2e-models-e2e-worker-w3-1e5e75/simulator-w3-1e5e75-kserve-6cd984c46b-tjz8p" Oct 31 00:27:57.167660 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:57.167641 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6158bb1-fa2f-4d00-8159-d1620958c7c0-dshm\") pod \"simulator-w3-1e5e75-kserve-6cd984c46b-tjz8p\" (UID: \"a6158bb1-fa2f-4d00-8159-d1620958c7c0\") " pod="e2e-models-e2e-worker-w3-1e5e75/simulator-w3-1e5e75-kserve-6cd984c46b-tjz8p" Oct 31 00:27:57.167809 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:57.167793 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6158bb1-fa2f-4d00-8159-d1620958c7c0-tls-certs\") pod \"simulator-w3-1e5e75-kserve-6cd984c46b-tjz8p\" (UID: \"a6158bb1-fa2f-4d00-8159-d1620958c7c0\") " pod="e2e-models-e2e-worker-w3-1e5e75/simulator-w3-1e5e75-kserve-6cd984c46b-tjz8p" Oct 31 00:27:57.227875 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:57.227796 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdw2b\" (UniqueName: \"kubernetes.io/projected/a6158bb1-fa2f-4d00-8159-d1620958c7c0-kube-api-access-vdw2b\") pod \"simulator-w3-1e5e75-kserve-6cd984c46b-tjz8p\" (UID: \"a6158bb1-fa2f-4d00-8159-d1620958c7c0\") " pod="e2e-models-e2e-worker-w3-1e5e75/simulator-w3-1e5e75-kserve-6cd984c46b-tjz8p" Oct 31 00:27:57.285869 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:57.285831 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-1e5e75/simulator-w3-1e5e75-kserve-6cd984c46b-tjz8p" Oct 31 00:27:57.666592 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:27:57.666560 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda6158bb1_fa2f_4d00_8159_d1620958c7c0.slice/crio-a2401b91de8f7e3fc737242074ecc205e2db0cde73808518f890a1c3256bd21d WatchSource:0}: Error finding container a2401b91de8f7e3fc737242074ecc205e2db0cde73808518f890a1c3256bd21d: Status 404 returned error can't find the container with id a2401b91de8f7e3fc737242074ecc205e2db0cde73808518f890a1c3256bd21d Oct 31 00:27:57.668691 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:57.668666 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-1e5e75/simulator-w3-1e5e75-kserve-6cd984c46b-tjz8p"] Oct 31 00:27:58.215293 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:58.215259 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-1e5e75/simulator-w3-1e5e75-kserve-6cd984c46b-tjz8p" event={"ID":"a6158bb1-fa2f-4d00-8159-d1620958c7c0","Type":"ContainerStarted","Data":"bc10714b685a61f0b635f904d557f09c68243916c086bdb8ecaaf41ebc808f1f"} Oct 31 00:27:58.215293 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:58.215298 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-1e5e75/simulator-w3-1e5e75-kserve-6cd984c46b-tjz8p" event={"ID":"a6158bb1-fa2f-4d00-8159-d1620958c7c0","Type":"ContainerStarted","Data":"a2401b91de8f7e3fc737242074ecc205e2db0cde73808518f890a1c3256bd21d"} Oct 31 00:27:58.286152 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:58.286108 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-1e5e75/simulator-w3-1e5e75-kserve-6cd984c46b-tjz8p" Oct 31 00:27:58.336203 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:27:58.336157 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-1e5e75/simulator-w3-1e5e75-kserve-6cd984c46b-tjz8p" podStartSLOduration=2.3361435249999998 podStartE2EDuration="2.336143525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:27:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:27:58.335832124 +0000 UTC m=+770.807001873" watchObservedRunningTime="2026-10-31 00:27:58.336143525 +0000 UTC m=+770.807313274" Oct 31 00:28:05.889778 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:28:05.889742 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 00:28:06.637124 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:28:06.637091 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-79tk8"] Oct 31 00:28:07.286466 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:28:07.286438 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-1e5e75/simulator-w3-1e5e75-kserve-6cd984c46b-tjz8p" Oct 31 00:28:07.299288 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:28:07.299259 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-1e5e75/simulator-w3-1e5e75-kserve-6cd984c46b-tjz8p" Oct 31 00:28:08.286432 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:28:08.286397 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-1e5e75/simulator-w3-1e5e75-kserve-6cd984c46b-tjz8p" Oct 31 00:28:18.505774 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:28:18.505735 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-a6f4a5/simulator-w6-a6f4a5-kserve-7769689d48-bpkc6"] Oct 31 00:28:18.509533 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:28:18.509508 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-a6f4a5/simulator-w6-a6f4a5-kserve-7769689d48-bpkc6" Oct 31 00:28:18.591215 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:28:18.591178 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24079bcb-658a-4109-9b14-a167a38cfac7-tmp-dir\") pod \"simulator-w6-a6f4a5-kserve-7769689d48-bpkc6\" (UID: \"24079bcb-658a-4109-9b14-a167a38cfac7\") " pod="e2e-models-e2e-worker-w6-a6f4a5/simulator-w6-a6f4a5-kserve-7769689d48-bpkc6" Oct 31 00:28:18.591422 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:28:18.591239 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24079bcb-658a-4109-9b14-a167a38cfac7-tls-certs\") pod \"simulator-w6-a6f4a5-kserve-7769689d48-bpkc6\" (UID: \"24079bcb-658a-4109-9b14-a167a38cfac7\") " pod="e2e-models-e2e-worker-w6-a6f4a5/simulator-w6-a6f4a5-kserve-7769689d48-bpkc6" Oct 31 00:28:18.591422 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:28:18.591260 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24079bcb-658a-4109-9b14-a167a38cfac7-model-cache\") pod \"simulator-w6-a6f4a5-kserve-7769689d48-bpkc6\" (UID: \"24079bcb-658a-4109-9b14-a167a38cfac7\") " pod="e2e-models-e2e-worker-w6-a6f4a5/simulator-w6-a6f4a5-kserve-7769689d48-bpkc6" Oct 31 00:28:18.591518 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:28:18.591433 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24079bcb-658a-4109-9b14-a167a38cfac7-dshm\") pod \"simulator-w6-a6f4a5-kserve-7769689d48-bpkc6\" (UID: \"24079bcb-658a-4109-9b14-a167a38cfac7\") " pod="e2e-models-e2e-worker-w6-a6f4a5/simulator-w6-a6f4a5-kserve-7769689d48-bpkc6" Oct 31 00:28:18.591518 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:28:18.591462 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24079bcb-658a-4109-9b14-a167a38cfac7-home\") pod \"simulator-w6-a6f4a5-kserve-7769689d48-bpkc6\" (UID: \"24079bcb-658a-4109-9b14-a167a38cfac7\") " pod="e2e-models-e2e-worker-w6-a6f4a5/simulator-w6-a6f4a5-kserve-7769689d48-bpkc6" Oct 31 00:28:18.591593 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:28:18.591535 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49zss\" (UniqueName: \"kubernetes.io/projected/24079bcb-658a-4109-9b14-a167a38cfac7-kube-api-access-49zss\") pod \"simulator-w6-a6f4a5-kserve-7769689d48-bpkc6\" (UID: \"24079bcb-658a-4109-9b14-a167a38cfac7\") " pod="e2e-models-e2e-worker-w6-a6f4a5/simulator-w6-a6f4a5-kserve-7769689d48-bpkc6" Oct 31 00:28:18.612447 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:28:18.612409 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-a6f4a5\"/\"kube-root-ca.crt\"" Oct 31 00:28:18.614579 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:28:18.614555 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-a6f4a5\"/\"openshift-service-ca.crt\"" Oct 31 00:28:18.614579 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:28:18.614556 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-a6f4a5\"/\"default-dockercfg-n9zmp\"" Oct 31 00:28:18.614767 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:28:18.614556 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-a6f4a5\"/\"simulator-w6-a6f4a5-kserve-self-signed-certs\"" Oct 31 00:28:18.616318 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:28:18.616298 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-a6f4a5/simulator-w6-a6f4a5-kserve-7769689d48-bpkc6"] Oct 31 00:28:18.692605 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:28:18.692570 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24079bcb-658a-4109-9b14-a167a38cfac7-dshm\") pod \"simulator-w6-a6f4a5-kserve-7769689d48-bpkc6\" (UID: \"24079bcb-658a-4109-9b14-a167a38cfac7\") " pod="e2e-models-e2e-worker-w6-a6f4a5/simulator-w6-a6f4a5-kserve-7769689d48-bpkc6" Oct 31 00:28:18.692605 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:28:18.692608 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24079bcb-658a-4109-9b14-a167a38cfac7-home\") pod \"simulator-w6-a6f4a5-kserve-7769689d48-bpkc6\" (UID: \"24079bcb-658a-4109-9b14-a167a38cfac7\") " pod="e2e-models-e2e-worker-w6-a6f4a5/simulator-w6-a6f4a5-kserve-7769689d48-bpkc6" Oct 31 00:28:18.692835 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:28:18.692646 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49zss\" (UniqueName: \"kubernetes.io/projected/24079bcb-658a-4109-9b14-a167a38cfac7-kube-api-access-49zss\") pod \"simulator-w6-a6f4a5-kserve-7769689d48-bpkc6\" (UID: \"24079bcb-658a-4109-9b14-a167a38cfac7\") " pod="e2e-models-e2e-worker-w6-a6f4a5/simulator-w6-a6f4a5-kserve-7769689d48-bpkc6" Oct 31 00:28:18.692835 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:28:18.692691 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24079bcb-658a-4109-9b14-a167a38cfac7-tmp-dir\") pod \"simulator-w6-a6f4a5-kserve-7769689d48-bpkc6\" (UID: \"24079bcb-658a-4109-9b14-a167a38cfac7\") " pod="e2e-models-e2e-worker-w6-a6f4a5/simulator-w6-a6f4a5-kserve-7769689d48-bpkc6" Oct 31 00:28:18.692835 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:28:18.692735 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24079bcb-658a-4109-9b14-a167a38cfac7-tls-certs\") pod \"simulator-w6-a6f4a5-kserve-7769689d48-bpkc6\" (UID: \"24079bcb-658a-4109-9b14-a167a38cfac7\") " pod="e2e-models-e2e-worker-w6-a6f4a5/simulator-w6-a6f4a5-kserve-7769689d48-bpkc6" Oct 31 00:28:18.692835 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:28:18.692753 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24079bcb-658a-4109-9b14-a167a38cfac7-model-cache\") pod \"simulator-w6-a6f4a5-kserve-7769689d48-bpkc6\" (UID: \"24079bcb-658a-4109-9b14-a167a38cfac7\") " pod="e2e-models-e2e-worker-w6-a6f4a5/simulator-w6-a6f4a5-kserve-7769689d48-bpkc6" Oct 31 00:28:18.693091 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:28:18.693069 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24079bcb-658a-4109-9b14-a167a38cfac7-home\") pod \"simulator-w6-a6f4a5-kserve-7769689d48-bpkc6\" (UID: \"24079bcb-658a-4109-9b14-a167a38cfac7\") " pod="e2e-models-e2e-worker-w6-a6f4a5/simulator-w6-a6f4a5-kserve-7769689d48-bpkc6" Oct 31 00:28:18.693143 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:28:18.693069 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24079bcb-658a-4109-9b14-a167a38cfac7-tmp-dir\") pod \"simulator-w6-a6f4a5-kserve-7769689d48-bpkc6\" (UID: \"24079bcb-658a-4109-9b14-a167a38cfac7\") " pod="e2e-models-e2e-worker-w6-a6f4a5/simulator-w6-a6f4a5-kserve-7769689d48-bpkc6" Oct 31 00:28:18.693204 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:28:18.693170 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24079bcb-658a-4109-9b14-a167a38cfac7-model-cache\") pod \"simulator-w6-a6f4a5-kserve-7769689d48-bpkc6\" (UID: \"24079bcb-658a-4109-9b14-a167a38cfac7\") " pod="e2e-models-e2e-worker-w6-a6f4a5/simulator-w6-a6f4a5-kserve-7769689d48-bpkc6" Oct 31 00:28:18.695906 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:28:18.695884 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24079bcb-658a-4109-9b14-a167a38cfac7-dshm\") pod \"simulator-w6-a6f4a5-kserve-7769689d48-bpkc6\" (UID: \"24079bcb-658a-4109-9b14-a167a38cfac7\") " pod="e2e-models-e2e-worker-w6-a6f4a5/simulator-w6-a6f4a5-kserve-7769689d48-bpkc6" Oct 31 00:28:18.696044 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:28:18.696025 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24079bcb-658a-4109-9b14-a167a38cfac7-tls-certs\") pod \"simulator-w6-a6f4a5-kserve-7769689d48-bpkc6\" (UID: \"24079bcb-658a-4109-9b14-a167a38cfac7\") " pod="e2e-models-e2e-worker-w6-a6f4a5/simulator-w6-a6f4a5-kserve-7769689d48-bpkc6" Oct 31 00:28:18.755322 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:28:18.755290 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49zss\" (UniqueName: \"kubernetes.io/projected/24079bcb-658a-4109-9b14-a167a38cfac7-kube-api-access-49zss\") pod \"simulator-w6-a6f4a5-kserve-7769689d48-bpkc6\" (UID: \"24079bcb-658a-4109-9b14-a167a38cfac7\") " pod="e2e-models-e2e-worker-w6-a6f4a5/simulator-w6-a6f4a5-kserve-7769689d48-bpkc6" Oct 31 00:28:18.821317 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:28:18.821278 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-a6f4a5/simulator-w6-a6f4a5-kserve-7769689d48-bpkc6" Oct 31 00:28:19.235930 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:28:19.235899 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod24079bcb_658a_4109_9b14_a167a38cfac7.slice/crio-68b003aaf4f5c90c9ca8fa5ac14c30e7a0a3fecb255c9479a00ac4f201090478 WatchSource:0}: Error finding container 68b003aaf4f5c90c9ca8fa5ac14c30e7a0a3fecb255c9479a00ac4f201090478: Status 404 returned error can't find the container with id 68b003aaf4f5c90c9ca8fa5ac14c30e7a0a3fecb255c9479a00ac4f201090478 Oct 31 00:28:19.237811 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:28:19.237784 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-a6f4a5/simulator-w6-a6f4a5-kserve-7769689d48-bpkc6"] Oct 31 00:28:19.334270 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:28:19.334238 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-a6f4a5/simulator-w6-a6f4a5-kserve-7769689d48-bpkc6" event={"ID":"24079bcb-658a-4109-9b14-a167a38cfac7","Type":"ContainerStarted","Data":"98c8701665839fb9b5694c7d1cb27b0fe5491f47e419c2d4032a60210b5adf5f"} Oct 31 00:28:19.334270 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:28:19.334276 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-a6f4a5/simulator-w6-a6f4a5-kserve-7769689d48-bpkc6" event={"ID":"24079bcb-658a-4109-9b14-a167a38cfac7","Type":"ContainerStarted","Data":"68b003aaf4f5c90c9ca8fa5ac14c30e7a0a3fecb255c9479a00ac4f201090478"} Oct 31 00:28:19.456784 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:28:19.456724 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-a6f4a5/simulator-w6-a6f4a5-kserve-7769689d48-bpkc6" podStartSLOduration=1.4567055039999999 podStartE2EDuration="1.456705504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:28:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:28:19.455096094 +0000 UTC m=+791.926265843" watchObservedRunningTime="2026-10-31 00:28:19.456705504 +0000 UTC m=+791.927875253" Oct 31 00:28:19.822215 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:28:19.822180 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a6f4a5/simulator-w6-a6f4a5-kserve-7769689d48-bpkc6" Oct 31 00:28:25.468454 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:28:25.468409 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-79tk8"] Oct 31 00:28:28.821936 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:28:28.821893 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-a6f4a5/simulator-w6-a6f4a5-kserve-7769689d48-bpkc6" Oct 31 00:28:28.834909 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:28:28.834881 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-a6f4a5/simulator-w6-a6f4a5-kserve-7769689d48-bpkc6" Oct 31 00:28:29.406685 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:28:29.406656 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a6f4a5/simulator-w6-a6f4a5-kserve-7769689d48-bpkc6" Oct 31 00:28:35.889782 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:28:35.889743 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 00:28:38.975729 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:28:38.975697 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-1e5e75/premium-w3-1e5e75-kserve-7865bd6955-55dbx"] Oct 31 00:28:38.980352 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:28:38.980334 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-1e5e75/premium-w3-1e5e75-kserve-7865bd6955-55dbx" Oct 31 00:28:39.034325 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:28:39.034285 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-1e5e75/premium-w3-1e5e75-kserve-7865bd6955-55dbx"] Oct 31 00:28:39.083123 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:28:39.083095 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-1e5e75\"/\"premium-w3-1e5e75-kserve-self-signed-certs\"" Oct 31 00:28:39.126446 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:28:39.126109 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f72e496-0ec8-44f4-a295-4d985d1afb92-dshm\") pod \"premium-w3-1e5e75-kserve-7865bd6955-55dbx\" (UID: \"8f72e496-0ec8-44f4-a295-4d985d1afb92\") " pod="e2e-models-e2e-worker-w3-1e5e75/premium-w3-1e5e75-kserve-7865bd6955-55dbx" Oct 31 00:28:39.126446 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:28:39.126199 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f72e496-0ec8-44f4-a295-4d985d1afb92-tmp-dir\") pod \"premium-w3-1e5e75-kserve-7865bd6955-55dbx\" (UID: \"8f72e496-0ec8-44f4-a295-4d985d1afb92\") " pod="e2e-models-e2e-worker-w3-1e5e75/premium-w3-1e5e75-kserve-7865bd6955-55dbx" Oct 31 00:28:39.126446 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:28:39.126321 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f72e496-0ec8-44f4-a295-4d985d1afb92-tls-certs\") pod \"premium-w3-1e5e75-kserve-7865bd6955-55dbx\" (UID: \"8f72e496-0ec8-44f4-a295-4d985d1afb92\") " pod="e2e-models-e2e-worker-w3-1e5e75/premium-w3-1e5e75-kserve-7865bd6955-55dbx" Oct 31 00:28:39.126446 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:28:39.126408 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f72e496-0ec8-44f4-a295-4d985d1afb92-model-cache\") pod \"premium-w3-1e5e75-kserve-7865bd6955-55dbx\" (UID: \"8f72e496-0ec8-44f4-a295-4d985d1afb92\") " pod="e2e-models-e2e-worker-w3-1e5e75/premium-w3-1e5e75-kserve-7865bd6955-55dbx" Oct 31 00:28:39.126751 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:28:39.126471 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f72e496-0ec8-44f4-a295-4d985d1afb92-home\") pod \"premium-w3-1e5e75-kserve-7865bd6955-55dbx\" (UID: \"8f72e496-0ec8-44f4-a295-4d985d1afb92\") " pod="e2e-models-e2e-worker-w3-1e5e75/premium-w3-1e5e75-kserve-7865bd6955-55dbx" Oct 31 00:28:39.126751 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:28:39.126528 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2bg6\" (UniqueName: \"kubernetes.io/projected/8f72e496-0ec8-44f4-a295-4d985d1afb92-kube-api-access-j2bg6\") pod \"premium-w3-1e5e75-kserve-7865bd6955-55dbx\" (UID: \"8f72e496-0ec8-44f4-a295-4d985d1afb92\") " pod="e2e-models-e2e-worker-w3-1e5e75/premium-w3-1e5e75-kserve-7865bd6955-55dbx" Oct 31 00:28:39.227569 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:28:39.227478 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f72e496-0ec8-44f4-a295-4d985d1afb92-home\") pod \"premium-w3-1e5e75-kserve-7865bd6955-55dbx\" (UID: \"8f72e496-0ec8-44f4-a295-4d985d1afb92\") " pod="e2e-models-e2e-worker-w3-1e5e75/premium-w3-1e5e75-kserve-7865bd6955-55dbx" Oct 31 00:28:39.227569 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:28:39.227551 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2bg6\" (UniqueName: \"kubernetes.io/projected/8f72e496-0ec8-44f4-a295-4d985d1afb92-kube-api-access-j2bg6\") pod \"premium-w3-1e5e75-kserve-7865bd6955-55dbx\" (UID: \"8f72e496-0ec8-44f4-a295-4d985d1afb92\") " pod="e2e-models-e2e-worker-w3-1e5e75/premium-w3-1e5e75-kserve-7865bd6955-55dbx" Oct 31 00:28:39.227800 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:28:39.227627 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f72e496-0ec8-44f4-a295-4d985d1afb92-dshm\") pod \"premium-w3-1e5e75-kserve-7865bd6955-55dbx\" (UID: \"8f72e496-0ec8-44f4-a295-4d985d1afb92\") " pod="e2e-models-e2e-worker-w3-1e5e75/premium-w3-1e5e75-kserve-7865bd6955-55dbx" Oct 31 00:28:39.227800 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:28:39.227685 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f72e496-0ec8-44f4-a295-4d985d1afb92-tmp-dir\") pod \"premium-w3-1e5e75-kserve-7865bd6955-55dbx\" (UID: \"8f72e496-0ec8-44f4-a295-4d985d1afb92\") " pod="e2e-models-e2e-worker-w3-1e5e75/premium-w3-1e5e75-kserve-7865bd6955-55dbx" Oct 31 00:28:39.227800 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:28:39.227728 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f72e496-0ec8-44f4-a295-4d985d1afb92-tls-certs\") pod \"premium-w3-1e5e75-kserve-7865bd6955-55dbx\" (UID: \"8f72e496-0ec8-44f4-a295-4d985d1afb92\") " pod="e2e-models-e2e-worker-w3-1e5e75/premium-w3-1e5e75-kserve-7865bd6955-55dbx" Oct 31 00:28:39.227800 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:28:39.227779 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f72e496-0ec8-44f4-a295-4d985d1afb92-model-cache\") pod \"premium-w3-1e5e75-kserve-7865bd6955-55dbx\" (UID: \"8f72e496-0ec8-44f4-a295-4d985d1afb92\") " pod="e2e-models-e2e-worker-w3-1e5e75/premium-w3-1e5e75-kserve-7865bd6955-55dbx" Oct 31 00:28:39.228061 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:28:39.228018 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f72e496-0ec8-44f4-a295-4d985d1afb92-home\") pod \"premium-w3-1e5e75-kserve-7865bd6955-55dbx\" (UID: \"8f72e496-0ec8-44f4-a295-4d985d1afb92\") " pod="e2e-models-e2e-worker-w3-1e5e75/premium-w3-1e5e75-kserve-7865bd6955-55dbx" Oct 31 00:28:39.228129 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:28:39.228074 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f72e496-0ec8-44f4-a295-4d985d1afb92-model-cache\") pod \"premium-w3-1e5e75-kserve-7865bd6955-55dbx\" (UID: \"8f72e496-0ec8-44f4-a295-4d985d1afb92\") " pod="e2e-models-e2e-worker-w3-1e5e75/premium-w3-1e5e75-kserve-7865bd6955-55dbx" Oct 31 00:28:39.228194 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:28:39.228167 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f72e496-0ec8-44f4-a295-4d985d1afb92-tmp-dir\") pod \"premium-w3-1e5e75-kserve-7865bd6955-55dbx\" (UID: \"8f72e496-0ec8-44f4-a295-4d985d1afb92\") " pod="e2e-models-e2e-worker-w3-1e5e75/premium-w3-1e5e75-kserve-7865bd6955-55dbx" Oct 31 00:28:39.230094 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:28:39.230071 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f72e496-0ec8-44f4-a295-4d985d1afb92-dshm\") pod \"premium-w3-1e5e75-kserve-7865bd6955-55dbx\" (UID: \"8f72e496-0ec8-44f4-a295-4d985d1afb92\") " pod="e2e-models-e2e-worker-w3-1e5e75/premium-w3-1e5e75-kserve-7865bd6955-55dbx" Oct 31 00:28:39.231125 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:28:39.231103 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f72e496-0ec8-44f4-a295-4d985d1afb92-tls-certs\") pod \"premium-w3-1e5e75-kserve-7865bd6955-55dbx\" (UID: \"8f72e496-0ec8-44f4-a295-4d985d1afb92\") " pod="e2e-models-e2e-worker-w3-1e5e75/premium-w3-1e5e75-kserve-7865bd6955-55dbx" Oct 31 00:28:39.290510 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:28:39.290482 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2bg6\" (UniqueName: \"kubernetes.io/projected/8f72e496-0ec8-44f4-a295-4d985d1afb92-kube-api-access-j2bg6\") pod \"premium-w3-1e5e75-kserve-7865bd6955-55dbx\" (UID: \"8f72e496-0ec8-44f4-a295-4d985d1afb92\") " pod="e2e-models-e2e-worker-w3-1e5e75/premium-w3-1e5e75-kserve-7865bd6955-55dbx" Oct 31 00:28:39.348395 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:28:39.330932 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-1e5e75/premium-w3-1e5e75-kserve-7865bd6955-55dbx" Oct 31 00:28:39.635537 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:28:39.635504 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-79tk8"] Oct 31 00:28:39.746043 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:28:39.745999 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8f72e496_0ec8_44f4_a295_4d985d1afb92.slice/crio-033b4a77e7e8bef6e4060a0a0ec85886470509a8fc0fc5bbf4fe44859ec8f023 WatchSource:0}: Error finding container 033b4a77e7e8bef6e4060a0a0ec85886470509a8fc0fc5bbf4fe44859ec8f023: Status 404 returned error can't find the container with id 033b4a77e7e8bef6e4060a0a0ec85886470509a8fc0fc5bbf4fe44859ec8f023 Oct 31 00:28:39.748349 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:28:39.748324 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-1e5e75/premium-w3-1e5e75-kserve-7865bd6955-55dbx"] Oct 31 00:28:40.456302 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:28:40.456260 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-1e5e75/premium-w3-1e5e75-kserve-7865bd6955-55dbx" event={"ID":"8f72e496-0ec8-44f4-a295-4d985d1afb92","Type":"ContainerStarted","Data":"230a69db020e8a9fb62039a1b66c1a57252bb1377aa5dfbceab46d4b5be220a5"} Oct 31 00:28:40.456302 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:28:40.456309 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-1e5e75/premium-w3-1e5e75-kserve-7865bd6955-55dbx" event={"ID":"8f72e496-0ec8-44f4-a295-4d985d1afb92","Type":"ContainerStarted","Data":"033b4a77e7e8bef6e4060a0a0ec85886470509a8fc0fc5bbf4fe44859ec8f023"} Oct 31 00:28:40.576744 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:28:40.576689 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-1e5e75/premium-w3-1e5e75-kserve-7865bd6955-55dbx" podStartSLOduration=2.5766750529999998 podStartE2EDuration="2.576675053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:28:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:28:40.575488118 +0000 UTC m=+813.046657880" watchObservedRunningTime="2026-10-31 00:28:40.576675053 +0000 UTC m=+813.047844803" Oct 31 00:28:41.334600 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:28:41.334563 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-1e5e75/premium-w3-1e5e75-kserve-7865bd6955-55dbx" Oct 31 00:28:49.332446 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:28:49.332347 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-1e5e75/premium-w3-1e5e75-kserve-7865bd6955-55dbx" Oct 31 00:28:49.345110 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:28:49.345078 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-1e5e75/premium-w3-1e5e75-kserve-7865bd6955-55dbx" Oct 31 00:28:49.516488 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:28:49.516449 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-1e5e75/premium-w3-1e5e75-kserve-7865bd6955-55dbx" Oct 31 00:28:52.491417 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:28:52.491384 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-79tk8"] Oct 31 00:28:56.545807 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:28:56.545774 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-da9d07/simulator-w4-da9d07-kserve-7c7588b894-snqq4"] Oct 31 00:28:56.549795 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:28:56.549777 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-da9d07/simulator-w4-da9d07-kserve-7c7588b894-snqq4" Oct 31 00:28:56.604184 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:28:56.604150 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d37ba9a-a648-44b2-aa04-4acf2b503a08-tmp-dir\") pod \"simulator-w4-da9d07-kserve-7c7588b894-snqq4\" (UID: \"3d37ba9a-a648-44b2-aa04-4acf2b503a08\") " pod="e2e-models-e2e-worker-w4-da9d07/simulator-w4-da9d07-kserve-7c7588b894-snqq4" Oct 31 00:28:56.604410 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:28:56.604218 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8rjn\" (UniqueName: \"kubernetes.io/projected/3d37ba9a-a648-44b2-aa04-4acf2b503a08-kube-api-access-q8rjn\") pod \"simulator-w4-da9d07-kserve-7c7588b894-snqq4\" (UID: \"3d37ba9a-a648-44b2-aa04-4acf2b503a08\") " pod="e2e-models-e2e-worker-w4-da9d07/simulator-w4-da9d07-kserve-7c7588b894-snqq4" Oct 31 00:28:56.604410 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:28:56.604224 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-da9d07\"/\"simulator-w4-da9d07-kserve-self-signed-certs\"" Oct 31 00:28:56.604410 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:28:56.604399 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d37ba9a-a648-44b2-aa04-4acf2b503a08-model-cache\") pod \"simulator-w4-da9d07-kserve-7c7588b894-snqq4\" (UID: \"3d37ba9a-a648-44b2-aa04-4acf2b503a08\") " pod="e2e-models-e2e-worker-w4-da9d07/simulator-w4-da9d07-kserve-7c7588b894-snqq4" Oct 31 00:28:56.604567 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:28:56.604461 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d37ba9a-a648-44b2-aa04-4acf2b503a08-tls-certs\") pod \"simulator-w4-da9d07-kserve-7c7588b894-snqq4\" (UID: \"3d37ba9a-a648-44b2-aa04-4acf2b503a08\") " pod="e2e-models-e2e-worker-w4-da9d07/simulator-w4-da9d07-kserve-7c7588b894-snqq4" Oct 31 00:28:56.604567 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:28:56.604516 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d37ba9a-a648-44b2-aa04-4acf2b503a08-home\") pod \"simulator-w4-da9d07-kserve-7c7588b894-snqq4\" (UID: \"3d37ba9a-a648-44b2-aa04-4acf2b503a08\") " pod="e2e-models-e2e-worker-w4-da9d07/simulator-w4-da9d07-kserve-7c7588b894-snqq4" Oct 31 00:28:56.604567 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:28:56.604549 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d37ba9a-a648-44b2-aa04-4acf2b503a08-dshm\") pod \"simulator-w4-da9d07-kserve-7c7588b894-snqq4\" (UID: \"3d37ba9a-a648-44b2-aa04-4acf2b503a08\") " pod="e2e-models-e2e-worker-w4-da9d07/simulator-w4-da9d07-kserve-7c7588b894-snqq4" Oct 31 00:28:56.652845 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:28:56.652811 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-da9d07\"/\"openshift-service-ca.crt\"" Oct 31 00:28:56.653037 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:28:56.652875 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-da9d07\"/\"default-dockercfg-zzljb\"" Oct 31 00:28:56.655572 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:28:56.655545 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-da9d07\"/\"kube-root-ca.crt\"" Oct 31 00:28:56.657594 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:28:56.657574 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-da9d07/simulator-w4-da9d07-kserve-7c7588b894-snqq4"] Oct 31 00:28:56.712940 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:28:56.712754 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8rjn\" (UniqueName: \"kubernetes.io/projected/3d37ba9a-a648-44b2-aa04-4acf2b503a08-kube-api-access-q8rjn\") pod \"simulator-w4-da9d07-kserve-7c7588b894-snqq4\" (UID: \"3d37ba9a-a648-44b2-aa04-4acf2b503a08\") " pod="e2e-models-e2e-worker-w4-da9d07/simulator-w4-da9d07-kserve-7c7588b894-snqq4" Oct 31 00:28:56.713536 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:28:56.713509 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d37ba9a-a648-44b2-aa04-4acf2b503a08-model-cache\") pod \"simulator-w4-da9d07-kserve-7c7588b894-snqq4\" (UID: \"3d37ba9a-a648-44b2-aa04-4acf2b503a08\") " pod="e2e-models-e2e-worker-w4-da9d07/simulator-w4-da9d07-kserve-7c7588b894-snqq4" Oct 31 00:28:56.716060 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:28:56.715165 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d37ba9a-a648-44b2-aa04-4acf2b503a08-tls-certs\") pod \"simulator-w4-da9d07-kserve-7c7588b894-snqq4\" (UID: \"3d37ba9a-a648-44b2-aa04-4acf2b503a08\") " pod="e2e-models-e2e-worker-w4-da9d07/simulator-w4-da9d07-kserve-7c7588b894-snqq4" Oct 31 00:28:56.716060 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:28:56.715260 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d37ba9a-a648-44b2-aa04-4acf2b503a08-home\") pod \"simulator-w4-da9d07-kserve-7c7588b894-snqq4\" (UID: \"3d37ba9a-a648-44b2-aa04-4acf2b503a08\") " pod="e2e-models-e2e-worker-w4-da9d07/simulator-w4-da9d07-kserve-7c7588b894-snqq4" Oct 31 00:28:56.716060 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:28:56.715303 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d37ba9a-a648-44b2-aa04-4acf2b503a08-dshm\") pod \"simulator-w4-da9d07-kserve-7c7588b894-snqq4\" (UID: \"3d37ba9a-a648-44b2-aa04-4acf2b503a08\") " pod="e2e-models-e2e-worker-w4-da9d07/simulator-w4-da9d07-kserve-7c7588b894-snqq4" Oct 31 00:28:56.716060 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:28:56.715400 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d37ba9a-a648-44b2-aa04-4acf2b503a08-tmp-dir\") pod \"simulator-w4-da9d07-kserve-7c7588b894-snqq4\" (UID: \"3d37ba9a-a648-44b2-aa04-4acf2b503a08\") " pod="e2e-models-e2e-worker-w4-da9d07/simulator-w4-da9d07-kserve-7c7588b894-snqq4" Oct 31 00:28:56.716940 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:28:56.716905 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d37ba9a-a648-44b2-aa04-4acf2b503a08-model-cache\") pod \"simulator-w4-da9d07-kserve-7c7588b894-snqq4\" (UID: \"3d37ba9a-a648-44b2-aa04-4acf2b503a08\") " pod="e2e-models-e2e-worker-w4-da9d07/simulator-w4-da9d07-kserve-7c7588b894-snqq4" Oct 31 00:28:56.719415 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:28:56.717610 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d37ba9a-a648-44b2-aa04-4acf2b503a08-home\") pod \"simulator-w4-da9d07-kserve-7c7588b894-snqq4\" (UID: \"3d37ba9a-a648-44b2-aa04-4acf2b503a08\") " pod="e2e-models-e2e-worker-w4-da9d07/simulator-w4-da9d07-kserve-7c7588b894-snqq4" Oct 31 00:28:56.719415 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:28:56.717710 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d37ba9a-a648-44b2-aa04-4acf2b503a08-tmp-dir\") pod \"simulator-w4-da9d07-kserve-7c7588b894-snqq4\" (UID: \"3d37ba9a-a648-44b2-aa04-4acf2b503a08\") " pod="e2e-models-e2e-worker-w4-da9d07/simulator-w4-da9d07-kserve-7c7588b894-snqq4" Oct 31 00:28:56.719592 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:28:56.719427 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d37ba9a-a648-44b2-aa04-4acf2b503a08-dshm\") pod \"simulator-w4-da9d07-kserve-7c7588b894-snqq4\" (UID: \"3d37ba9a-a648-44b2-aa04-4acf2b503a08\") " pod="e2e-models-e2e-worker-w4-da9d07/simulator-w4-da9d07-kserve-7c7588b894-snqq4" Oct 31 00:28:56.720559 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:28:56.720536 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d37ba9a-a648-44b2-aa04-4acf2b503a08-tls-certs\") pod \"simulator-w4-da9d07-kserve-7c7588b894-snqq4\" (UID: \"3d37ba9a-a648-44b2-aa04-4acf2b503a08\") " pod="e2e-models-e2e-worker-w4-da9d07/simulator-w4-da9d07-kserve-7c7588b894-snqq4" Oct 31 00:28:56.778583 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:28:56.778442 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8rjn\" (UniqueName: \"kubernetes.io/projected/3d37ba9a-a648-44b2-aa04-4acf2b503a08-kube-api-access-q8rjn\") pod \"simulator-w4-da9d07-kserve-7c7588b894-snqq4\" (UID: \"3d37ba9a-a648-44b2-aa04-4acf2b503a08\") " pod="e2e-models-e2e-worker-w4-da9d07/simulator-w4-da9d07-kserve-7c7588b894-snqq4" Oct 31 00:28:56.862873 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:28:56.862777 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-da9d07/simulator-w4-da9d07-kserve-7c7588b894-snqq4" Oct 31 00:28:57.296088 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:28:57.296052 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3d37ba9a_a648_44b2_aa04_4acf2b503a08.slice/crio-75e6583c7520fa014f61f618fefb4a19255efdd97ca1fac2ed6e6e2bcf74c93e WatchSource:0}: Error finding container 75e6583c7520fa014f61f618fefb4a19255efdd97ca1fac2ed6e6e2bcf74c93e: Status 404 returned error can't find the container with id 75e6583c7520fa014f61f618fefb4a19255efdd97ca1fac2ed6e6e2bcf74c93e Oct 31 00:28:57.298717 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:28:57.298696 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-da9d07/simulator-w4-da9d07-kserve-7c7588b894-snqq4"] Oct 31 00:28:57.548448 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:28:57.548336 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-da9d07/simulator-w4-da9d07-kserve-7c7588b894-snqq4" event={"ID":"3d37ba9a-a648-44b2-aa04-4acf2b503a08","Type":"ContainerStarted","Data":"d9b2458873ff4883ca579cbebeb20cfff6e24cbc1808367ac031b35f91e04383"} Oct 31 00:28:57.548448 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:28:57.548421 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-da9d07/simulator-w4-da9d07-kserve-7c7588b894-snqq4" event={"ID":"3d37ba9a-a648-44b2-aa04-4acf2b503a08","Type":"ContainerStarted","Data":"75e6583c7520fa014f61f618fefb4a19255efdd97ca1fac2ed6e6e2bcf74c93e"} Oct 31 00:28:57.668752 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:28:57.668695 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-da9d07/simulator-w4-da9d07-kserve-7c7588b894-snqq4" podStartSLOduration=1.668681005 podStartE2EDuration="1.668681005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:28:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:28:57.667030216 +0000 UTC m=+830.138199966" watchObservedRunningTime="2026-10-31 00:28:57.668681005 +0000 UTC m=+830.139850754" Oct 31 00:28:57.863530 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:28:57.863453 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-da9d07/simulator-w4-da9d07-kserve-7c7588b894-snqq4" Oct 31 00:29:05.889387 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:29:05.889335 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 00:29:06.863739 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:29:06.863690 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-da9d07/simulator-w4-da9d07-kserve-7c7588b894-snqq4" Oct 31 00:29:06.877061 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:29:06.877034 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-da9d07/simulator-w4-da9d07-kserve-7c7588b894-snqq4" Oct 31 00:29:07.610291 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:29:07.610263 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-da9d07/simulator-w4-da9d07-kserve-7c7588b894-snqq4" Oct 31 00:29:15.555507 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:29:15.555470 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-79tk8"] Oct 31 00:29:19.998171 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:29:19.998122 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-412743/simulator-w5-412743-kserve-6f897dd967-5bkq5"] Oct 31 00:29:20.001720 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:29:20.001701 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-412743/simulator-w5-412743-kserve-6f897dd967-5bkq5" Oct 31 00:29:20.056929 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:29:20.056898 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-412743/simulator-w5-412743-kserve-6f897dd967-5bkq5"] Oct 31 00:29:20.076622 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:29:20.076586 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68d7fd60-bcad-45f7-84c0-11424e26f26d-dshm\") pod \"simulator-w5-412743-kserve-6f897dd967-5bkq5\" (UID: \"68d7fd60-bcad-45f7-84c0-11424e26f26d\") " pod="e2e-models-e2e-worker-w5-412743/simulator-w5-412743-kserve-6f897dd967-5bkq5" Oct 31 00:29:20.076797 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:29:20.076670 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68d7fd60-bcad-45f7-84c0-11424e26f26d-tmp-dir\") pod \"simulator-w5-412743-kserve-6f897dd967-5bkq5\" (UID: \"68d7fd60-bcad-45f7-84c0-11424e26f26d\") " pod="e2e-models-e2e-worker-w5-412743/simulator-w5-412743-kserve-6f897dd967-5bkq5" Oct 31 00:29:20.076797 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:29:20.076784 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68d7fd60-bcad-45f7-84c0-11424e26f26d-tls-certs\") pod \"simulator-w5-412743-kserve-6f897dd967-5bkq5\" (UID: \"68d7fd60-bcad-45f7-84c0-11424e26f26d\") " pod="e2e-models-e2e-worker-w5-412743/simulator-w5-412743-kserve-6f897dd967-5bkq5" Oct 31 00:29:20.076884 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:29:20.076813 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqthd\" (UniqueName: \"kubernetes.io/projected/68d7fd60-bcad-45f7-84c0-11424e26f26d-kube-api-access-vqthd\") pod \"simulator-w5-412743-kserve-6f897dd967-5bkq5\" (UID: \"68d7fd60-bcad-45f7-84c0-11424e26f26d\") " pod="e2e-models-e2e-worker-w5-412743/simulator-w5-412743-kserve-6f897dd967-5bkq5" Oct 31 00:29:20.076884 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:29:20.076850 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68d7fd60-bcad-45f7-84c0-11424e26f26d-home\") pod \"simulator-w5-412743-kserve-6f897dd967-5bkq5\" (UID: \"68d7fd60-bcad-45f7-84c0-11424e26f26d\") " pod="e2e-models-e2e-worker-w5-412743/simulator-w5-412743-kserve-6f897dd967-5bkq5" Oct 31 00:29:20.077096 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:29:20.077068 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68d7fd60-bcad-45f7-84c0-11424e26f26d-model-cache\") pod \"simulator-w5-412743-kserve-6f897dd967-5bkq5\" (UID: \"68d7fd60-bcad-45f7-84c0-11424e26f26d\") " pod="e2e-models-e2e-worker-w5-412743/simulator-w5-412743-kserve-6f897dd967-5bkq5" Oct 31 00:29:20.105711 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:29:20.105672 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-412743\"/\"kube-root-ca.crt\"" Oct 31 00:29:20.105711 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:29:20.105681 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-412743\"/\"default-dockercfg-z8bt4\"" Oct 31 00:29:20.105929 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:29:20.105746 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-412743\"/\"openshift-service-ca.crt\"" Oct 31 00:29:20.110986 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:29:20.110961 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-412743\"/\"simulator-w5-412743-kserve-self-signed-certs\"" Oct 31 00:29:20.179758 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:29:20.179688 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68d7fd60-bcad-45f7-84c0-11424e26f26d-model-cache\") pod \"simulator-w5-412743-kserve-6f897dd967-5bkq5\" (UID: \"68d7fd60-bcad-45f7-84c0-11424e26f26d\") " pod="e2e-models-e2e-worker-w5-412743/simulator-w5-412743-kserve-6f897dd967-5bkq5" Oct 31 00:29:20.180668 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:29:20.180636 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68d7fd60-bcad-45f7-84c0-11424e26f26d-dshm\") pod \"simulator-w5-412743-kserve-6f897dd967-5bkq5\" (UID: \"68d7fd60-bcad-45f7-84c0-11424e26f26d\") " pod="e2e-models-e2e-worker-w5-412743/simulator-w5-412743-kserve-6f897dd967-5bkq5" Oct 31 00:29:20.181662 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:29:20.181420 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68d7fd60-bcad-45f7-84c0-11424e26f26d-tmp-dir\") pod \"simulator-w5-412743-kserve-6f897dd967-5bkq5\" (UID: \"68d7fd60-bcad-45f7-84c0-11424e26f26d\") " pod="e2e-models-e2e-worker-w5-412743/simulator-w5-412743-kserve-6f897dd967-5bkq5" Oct 31 00:29:20.181924 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:29:20.181906 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68d7fd60-bcad-45f7-84c0-11424e26f26d-tls-certs\") pod \"simulator-w5-412743-kserve-6f897dd967-5bkq5\" (UID: \"68d7fd60-bcad-45f7-84c0-11424e26f26d\") " pod="e2e-models-e2e-worker-w5-412743/simulator-w5-412743-kserve-6f897dd967-5bkq5" Oct 31 00:29:20.182753 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:29:20.182733 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqthd\" (UniqueName: \"kubernetes.io/projected/68d7fd60-bcad-45f7-84c0-11424e26f26d-kube-api-access-vqthd\") pod \"simulator-w5-412743-kserve-6f897dd967-5bkq5\" (UID: \"68d7fd60-bcad-45f7-84c0-11424e26f26d\") " pod="e2e-models-e2e-worker-w5-412743/simulator-w5-412743-kserve-6f897dd967-5bkq5" Oct 31 00:29:20.183428 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:29:20.180524 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68d7fd60-bcad-45f7-84c0-11424e26f26d-model-cache\") pod \"simulator-w5-412743-kserve-6f897dd967-5bkq5\" (UID: \"68d7fd60-bcad-45f7-84c0-11424e26f26d\") " pod="e2e-models-e2e-worker-w5-412743/simulator-w5-412743-kserve-6f897dd967-5bkq5" Oct 31 00:29:20.183844 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:29:20.183824 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68d7fd60-bcad-45f7-84c0-11424e26f26d-tmp-dir\") pod \"simulator-w5-412743-kserve-6f897dd967-5bkq5\" (UID: \"68d7fd60-bcad-45f7-84c0-11424e26f26d\") " pod="e2e-models-e2e-worker-w5-412743/simulator-w5-412743-kserve-6f897dd967-5bkq5" Oct 31 00:29:20.184425 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:29:20.184247 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68d7fd60-bcad-45f7-84c0-11424e26f26d-home\") pod \"simulator-w5-412743-kserve-6f897dd967-5bkq5\" (UID: \"68d7fd60-bcad-45f7-84c0-11424e26f26d\") " pod="e2e-models-e2e-worker-w5-412743/simulator-w5-412743-kserve-6f897dd967-5bkq5" Oct 31 00:29:20.184964 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:29:20.183359 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68d7fd60-bcad-45f7-84c0-11424e26f26d-home\") pod \"simulator-w5-412743-kserve-6f897dd967-5bkq5\" (UID: \"68d7fd60-bcad-45f7-84c0-11424e26f26d\") " pod="e2e-models-e2e-worker-w5-412743/simulator-w5-412743-kserve-6f897dd967-5bkq5" Oct 31 00:29:20.199730 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:29:20.199696 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68d7fd60-bcad-45f7-84c0-11424e26f26d-tls-certs\") pod \"simulator-w5-412743-kserve-6f897dd967-5bkq5\" (UID: \"68d7fd60-bcad-45f7-84c0-11424e26f26d\") " pod="e2e-models-e2e-worker-w5-412743/simulator-w5-412743-kserve-6f897dd967-5bkq5" Oct 31 00:29:20.205617 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:29:20.205543 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68d7fd60-bcad-45f7-84c0-11424e26f26d-dshm\") pod \"simulator-w5-412743-kserve-6f897dd967-5bkq5\" (UID: \"68d7fd60-bcad-45f7-84c0-11424e26f26d\") " pod="e2e-models-e2e-worker-w5-412743/simulator-w5-412743-kserve-6f897dd967-5bkq5" Oct 31 00:29:20.245391 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:29:20.245337 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqthd\" (UniqueName: \"kubernetes.io/projected/68d7fd60-bcad-45f7-84c0-11424e26f26d-kube-api-access-vqthd\") pod \"simulator-w5-412743-kserve-6f897dd967-5bkq5\" (UID: \"68d7fd60-bcad-45f7-84c0-11424e26f26d\") " pod="e2e-models-e2e-worker-w5-412743/simulator-w5-412743-kserve-6f897dd967-5bkq5" Oct 31 00:29:20.314464 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:29:20.314426 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-412743/simulator-w5-412743-kserve-6f897dd967-5bkq5" Oct 31 00:29:20.689036 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:29:20.689002 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod68d7fd60_bcad_45f7_84c0_11424e26f26d.slice/crio-3360d1c4d849af0828635dd57fb1582fb43ad23ad7765db445db53a48a50ee29 WatchSource:0}: Error finding container 3360d1c4d849af0828635dd57fb1582fb43ad23ad7765db445db53a48a50ee29: Status 404 returned error can't find the container with id 3360d1c4d849af0828635dd57fb1582fb43ad23ad7765db445db53a48a50ee29 Oct 31 00:29:20.691143 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:29:20.691125 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-412743/simulator-w5-412743-kserve-6f897dd967-5bkq5"] Oct 31 00:29:21.687740 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:29:21.687704 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-412743/simulator-w5-412743-kserve-6f897dd967-5bkq5" event={"ID":"68d7fd60-bcad-45f7-84c0-11424e26f26d","Type":"ContainerStarted","Data":"11580e2d56855201eba5f576e3f0a083d778374e459e175137997cc834eb701a"} Oct 31 00:29:21.688148 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:29:21.687752 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-412743/simulator-w5-412743-kserve-6f897dd967-5bkq5" event={"ID":"68d7fd60-bcad-45f7-84c0-11424e26f26d","Type":"ContainerStarted","Data":"3360d1c4d849af0828635dd57fb1582fb43ad23ad7765db445db53a48a50ee29"} Oct 31 00:29:21.807820 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:29:21.807764 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-412743/simulator-w5-412743-kserve-6f897dd967-5bkq5" podStartSLOduration=2.8077497989999998 podStartE2EDuration="2.807749799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:29:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:29:21.807393722 +0000 UTC m=+854.278563470" watchObservedRunningTime="2026-10-31 00:29:21.807749799 +0000 UTC m=+854.278919547" Oct 31 00:29:22.314503 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:29:22.314471 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-412743/simulator-w5-412743-kserve-6f897dd967-5bkq5" Oct 31 00:29:30.315002 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:29:30.314967 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-412743/simulator-w5-412743-kserve-6f897dd967-5bkq5" Oct 31 00:29:30.327597 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:29:30.327568 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-412743/simulator-w5-412743-kserve-6f897dd967-5bkq5" Oct 31 00:29:30.746701 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:29:30.746619 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-412743/simulator-w5-412743-kserve-6f897dd967-5bkq5" Oct 31 00:29:35.890173 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:29:35.890136 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 00:29:43.038669 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:29:43.038632 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-79tk8"] Oct 31 00:29:49.030270 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:29:49.030239 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-982ae3/premium-w1-982ae3-kserve-5c8755f445-qm9bh"] Oct 31 00:29:49.033923 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:29:49.033906 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-982ae3/premium-w1-982ae3-kserve-5c8755f445-qm9bh" Oct 31 00:29:49.088588 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:29:49.088554 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-982ae3/premium-w1-982ae3-kserve-5c8755f445-qm9bh"] Oct 31 00:29:49.092007 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:29:49.091982 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ltmk\" (UniqueName: \"kubernetes.io/projected/005a2416-8e4f-4a72-8f1a-169a25e0f18c-kube-api-access-2ltmk\") pod \"premium-w1-982ae3-kserve-5c8755f445-qm9bh\" (UID: \"005a2416-8e4f-4a72-8f1a-169a25e0f18c\") " pod="e2e-models-e2e-worker-w1-982ae3/premium-w1-982ae3-kserve-5c8755f445-qm9bh" Oct 31 00:29:49.092083 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:29:49.092052 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/005a2416-8e4f-4a72-8f1a-169a25e0f18c-tls-certs\") pod \"premium-w1-982ae3-kserve-5c8755f445-qm9bh\" (UID: \"005a2416-8e4f-4a72-8f1a-169a25e0f18c\") " pod="e2e-models-e2e-worker-w1-982ae3/premium-w1-982ae3-kserve-5c8755f445-qm9bh" Oct 31 00:29:49.092134 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:29:49.092122 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/005a2416-8e4f-4a72-8f1a-169a25e0f18c-tmp-dir\") pod \"premium-w1-982ae3-kserve-5c8755f445-qm9bh\" (UID: \"005a2416-8e4f-4a72-8f1a-169a25e0f18c\") " pod="e2e-models-e2e-worker-w1-982ae3/premium-w1-982ae3-kserve-5c8755f445-qm9bh" Oct 31 00:29:49.092256 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:29:49.092233 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/005a2416-8e4f-4a72-8f1a-169a25e0f18c-home\") pod \"premium-w1-982ae3-kserve-5c8755f445-qm9bh\" (UID: \"005a2416-8e4f-4a72-8f1a-169a25e0f18c\") " pod="e2e-models-e2e-worker-w1-982ae3/premium-w1-982ae3-kserve-5c8755f445-qm9bh" Oct 31 00:29:49.092322 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:29:49.092273 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/005a2416-8e4f-4a72-8f1a-169a25e0f18c-dshm\") pod \"premium-w1-982ae3-kserve-5c8755f445-qm9bh\" (UID: \"005a2416-8e4f-4a72-8f1a-169a25e0f18c\") " pod="e2e-models-e2e-worker-w1-982ae3/premium-w1-982ae3-kserve-5c8755f445-qm9bh" Oct 31 00:29:49.092408 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:29:49.092386 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/005a2416-8e4f-4a72-8f1a-169a25e0f18c-model-cache\") pod \"premium-w1-982ae3-kserve-5c8755f445-qm9bh\" (UID: \"005a2416-8e4f-4a72-8f1a-169a25e0f18c\") " pod="e2e-models-e2e-worker-w1-982ae3/premium-w1-982ae3-kserve-5c8755f445-qm9bh" Oct 31 00:29:49.137695 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:29:49.137655 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-982ae3\"/\"openshift-service-ca.crt\"" Oct 31 00:29:49.137879 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:29:49.137859 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-982ae3\"/\"kube-root-ca.crt\"" Oct 31 00:29:49.137879 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:29:49.137862 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-982ae3\"/\"default-dockercfg-jh7nw\"" Oct 31 00:29:49.143592 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:29:49.143562 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-982ae3\"/\"premium-w1-982ae3-kserve-self-signed-certs\"" Oct 31 00:29:49.194883 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:29:49.193055 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/005a2416-8e4f-4a72-8f1a-169a25e0f18c-tls-certs\") pod \"premium-w1-982ae3-kserve-5c8755f445-qm9bh\" (UID: \"005a2416-8e4f-4a72-8f1a-169a25e0f18c\") " pod="e2e-models-e2e-worker-w1-982ae3/premium-w1-982ae3-kserve-5c8755f445-qm9bh" Oct 31 00:29:49.194883 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:29:49.193171 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/005a2416-8e4f-4a72-8f1a-169a25e0f18c-tmp-dir\") pod \"premium-w1-982ae3-kserve-5c8755f445-qm9bh\" (UID: \"005a2416-8e4f-4a72-8f1a-169a25e0f18c\") " pod="e2e-models-e2e-worker-w1-982ae3/premium-w1-982ae3-kserve-5c8755f445-qm9bh" Oct 31 00:29:49.194883 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:29:49.193212 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/005a2416-8e4f-4a72-8f1a-169a25e0f18c-home\") pod \"premium-w1-982ae3-kserve-5c8755f445-qm9bh\" (UID: \"005a2416-8e4f-4a72-8f1a-169a25e0f18c\") " pod="e2e-models-e2e-worker-w1-982ae3/premium-w1-982ae3-kserve-5c8755f445-qm9bh" Oct 31 00:29:49.194883 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:29:49.193238 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/005a2416-8e4f-4a72-8f1a-169a25e0f18c-dshm\") pod \"premium-w1-982ae3-kserve-5c8755f445-qm9bh\" (UID: \"005a2416-8e4f-4a72-8f1a-169a25e0f18c\") " pod="e2e-models-e2e-worker-w1-982ae3/premium-w1-982ae3-kserve-5c8755f445-qm9bh" Oct 31 00:29:49.194883 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:29:49.193286 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/005a2416-8e4f-4a72-8f1a-169a25e0f18c-model-cache\") pod \"premium-w1-982ae3-kserve-5c8755f445-qm9bh\" (UID: \"005a2416-8e4f-4a72-8f1a-169a25e0f18c\") " pod="e2e-models-e2e-worker-w1-982ae3/premium-w1-982ae3-kserve-5c8755f445-qm9bh" Oct 31 00:29:49.194883 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:29:49.193408 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ltmk\" (UniqueName: \"kubernetes.io/projected/005a2416-8e4f-4a72-8f1a-169a25e0f18c-kube-api-access-2ltmk\") pod \"premium-w1-982ae3-kserve-5c8755f445-qm9bh\" (UID: \"005a2416-8e4f-4a72-8f1a-169a25e0f18c\") " pod="e2e-models-e2e-worker-w1-982ae3/premium-w1-982ae3-kserve-5c8755f445-qm9bh" Oct 31 00:29:49.194883 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:29:49.193650 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/005a2416-8e4f-4a72-8f1a-169a25e0f18c-home\") pod \"premium-w1-982ae3-kserve-5c8755f445-qm9bh\" (UID: \"005a2416-8e4f-4a72-8f1a-169a25e0f18c\") " pod="e2e-models-e2e-worker-w1-982ae3/premium-w1-982ae3-kserve-5c8755f445-qm9bh" Oct 31 00:29:49.194883 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:29:49.194020 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/005a2416-8e4f-4a72-8f1a-169a25e0f18c-model-cache\") pod \"premium-w1-982ae3-kserve-5c8755f445-qm9bh\" (UID: \"005a2416-8e4f-4a72-8f1a-169a25e0f18c\") " pod="e2e-models-e2e-worker-w1-982ae3/premium-w1-982ae3-kserve-5c8755f445-qm9bh" Oct 31 00:29:49.194883 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:29:49.194358 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/005a2416-8e4f-4a72-8f1a-169a25e0f18c-tmp-dir\") pod \"premium-w1-982ae3-kserve-5c8755f445-qm9bh\" (UID: \"005a2416-8e4f-4a72-8f1a-169a25e0f18c\") " pod="e2e-models-e2e-worker-w1-982ae3/premium-w1-982ae3-kserve-5c8755f445-qm9bh" Oct 31 00:29:49.196927 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:29:49.196889 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/005a2416-8e4f-4a72-8f1a-169a25e0f18c-dshm\") pod \"premium-w1-982ae3-kserve-5c8755f445-qm9bh\" (UID: \"005a2416-8e4f-4a72-8f1a-169a25e0f18c\") " pod="e2e-models-e2e-worker-w1-982ae3/premium-w1-982ae3-kserve-5c8755f445-qm9bh" Oct 31 00:29:49.197233 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:29:49.197214 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/005a2416-8e4f-4a72-8f1a-169a25e0f18c-tls-certs\") pod \"premium-w1-982ae3-kserve-5c8755f445-qm9bh\" (UID: \"005a2416-8e4f-4a72-8f1a-169a25e0f18c\") " pod="e2e-models-e2e-worker-w1-982ae3/premium-w1-982ae3-kserve-5c8755f445-qm9bh" Oct 31 00:29:49.256451 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:29:49.256352 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ltmk\" (UniqueName: \"kubernetes.io/projected/005a2416-8e4f-4a72-8f1a-169a25e0f18c-kube-api-access-2ltmk\") pod \"premium-w1-982ae3-kserve-5c8755f445-qm9bh\" (UID: \"005a2416-8e4f-4a72-8f1a-169a25e0f18c\") " pod="e2e-models-e2e-worker-w1-982ae3/premium-w1-982ae3-kserve-5c8755f445-qm9bh" Oct 31 00:29:49.347255 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:29:49.347176 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-982ae3/premium-w1-982ae3-kserve-5c8755f445-qm9bh" Oct 31 00:29:49.741939 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:29:49.741906 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod005a2416_8e4f_4a72_8f1a_169a25e0f18c.slice/crio-c9da8f5009e90c30caa2490f14884ec1ee2911228ff8c903ae49fe0bf6349bd5 WatchSource:0}: Error finding container c9da8f5009e90c30caa2490f14884ec1ee2911228ff8c903ae49fe0bf6349bd5: Status 404 returned error can't find the container with id c9da8f5009e90c30caa2490f14884ec1ee2911228ff8c903ae49fe0bf6349bd5 Oct 31 00:29:49.743261 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:29:49.743236 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-982ae3/premium-w1-982ae3-kserve-5c8755f445-qm9bh"] Oct 31 00:29:49.857054 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:29:49.857015 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-982ae3/premium-w1-982ae3-kserve-5c8755f445-qm9bh" event={"ID":"005a2416-8e4f-4a72-8f1a-169a25e0f18c","Type":"ContainerStarted","Data":"9d8c49ba4c2fc723b0ffb62c09c7787751c22a1f30c6596085639af71f1a5246"} Oct 31 00:29:49.857267 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:29:49.857068 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-982ae3/premium-w1-982ae3-kserve-5c8755f445-qm9bh" event={"ID":"005a2416-8e4f-4a72-8f1a-169a25e0f18c","Type":"ContainerStarted","Data":"c9da8f5009e90c30caa2490f14884ec1ee2911228ff8c903ae49fe0bf6349bd5"} Oct 31 00:29:49.982659 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:29:49.982608 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-982ae3/premium-w1-982ae3-kserve-5c8755f445-qm9bh" podStartSLOduration=1.982590399 podStartE2EDuration="1.982590399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:29:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:29:49.980841692 +0000 UTC m=+882.452011443" watchObservedRunningTime="2026-10-31 00:29:49.982590399 +0000 UTC m=+882.453760147" Oct 31 00:29:50.347889 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:29:50.347851 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-982ae3/premium-w1-982ae3-kserve-5c8755f445-qm9bh" Oct 31 00:29:59.348172 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:29:59.348133 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-982ae3/premium-w1-982ae3-kserve-5c8755f445-qm9bh" Oct 31 00:29:59.361003 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:29:59.360971 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-982ae3/premium-w1-982ae3-kserve-5c8755f445-qm9bh" Oct 31 00:29:59.921399 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:29:59.921339 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-982ae3/premium-w1-982ae3-kserve-5c8755f445-qm9bh" Oct 31 00:30:00.201042 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:00.200964 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e8ba06-29890110-gcpws"] Oct 31 00:30:00.205196 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:00.205169 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e8ba06-29890110-gcpws" Oct 31 00:30:00.212035 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:00.211999 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da9d07-29890110-r6vzq"] Oct 31 00:30:00.216131 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:00.216111 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-101b9e-29890110-7zx97"] Oct 31 00:30:00.216239 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:00.216226 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da9d07-29890110-r6vzq" Oct 31 00:30:00.219663 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:00.219646 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1e5e75-29890110-zh827"] Oct 31 00:30:00.219782 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:00.219766 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-101b9e-29890110-7zx97" Oct 31 00:30:00.223303 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:00.223284 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1e5e75-29890110-zh827" Oct 31 00:30:00.252421 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:00.252391 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-412743-29890110-4245m"] Oct 31 00:30:00.256148 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:00.256131 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-412743-29890110-4245m" Oct 31 00:30:00.260336 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:00.260312 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:30:00.260336 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:00.260325 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:30:00.262708 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:00.262689 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfz9z\"" Oct 31 00:30:00.296278 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:00.296232 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e8ba06-29890110-gcpws"] Oct 31 00:30:00.305004 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:00.301838 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da9d07-29890110-r6vzq"] Oct 31 00:30:00.311248 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:00.311210 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1e5e75-29890110-zh827"] Oct 31 00:30:00.313217 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:00.313193 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-101b9e-29890110-7zx97"] Oct 31 00:30:00.320821 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:00.320786 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/156e0827-de4c-4a74-b6f0-9c2cca06ba02-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-e8ba06-29890110-gcpws\" (UID: \"156e0827-de4c-4a74-b6f0-9c2cca06ba02\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e8ba06-29890110-gcpws" Oct 31 00:30:00.320966 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:00.320839 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwr8b\" (UniqueName: \"kubernetes.io/projected/6dbebe96-28d7-4e27-bcb7-9291a84f6c71-kube-api-access-jwr8b\") pod \"maas-api-key-cleanup-e2e-worker-w5-412743-29890110-4245m\" (UID: \"6dbebe96-28d7-4e27-bcb7-9291a84f6c71\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-412743-29890110-4245m" Oct 31 00:30:00.320966 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:00.320949 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d56a9943-4360-48d4-a701-1bd9b4206908-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-da9d07-29890110-r6vzq\" (UID: \"d56a9943-4360-48d4-a701-1bd9b4206908\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da9d07-29890110-r6vzq" Oct 31 00:30:00.321090 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:00.320984 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6dbebe96-28d7-4e27-bcb7-9291a84f6c71-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-412743-29890110-4245m\" (UID: \"6dbebe96-28d7-4e27-bcb7-9291a84f6c71\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-412743-29890110-4245m" Oct 31 00:30:00.321156 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:00.321110 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4sz7\" (UniqueName: \"kubernetes.io/projected/5548598f-3919-417b-ac60-1d74ff41da94-kube-api-access-t4sz7\") pod \"maas-api-key-cleanup-e2e-worker-w3-1e5e75-29890110-zh827\" (UID: \"5548598f-3919-417b-ac60-1d74ff41da94\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1e5e75-29890110-zh827" Oct 31 00:30:00.321156 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:00.321142 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnt24\" (UniqueName: \"kubernetes.io/projected/f0d7eece-c1b5-4f51-9ed7-76d3d0564873-kube-api-access-nnt24\") pod \"maas-api-key-cleanup-e2e-worker-w0-101b9e-29890110-7zx97\" (UID: \"f0d7eece-c1b5-4f51-9ed7-76d3d0564873\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-101b9e-29890110-7zx97" Oct 31 00:30:00.321259 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:00.321177 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6j6p\" (UniqueName: \"kubernetes.io/projected/156e0827-de4c-4a74-b6f0-9c2cca06ba02-kube-api-access-p6j6p\") pod \"maas-api-key-cleanup-e2e-worker-w2-e8ba06-29890110-gcpws\" (UID: \"156e0827-de4c-4a74-b6f0-9c2cca06ba02\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e8ba06-29890110-gcpws" Oct 31 00:30:00.321259 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:00.321238 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5548598f-3919-417b-ac60-1d74ff41da94-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-1e5e75-29890110-zh827\" (UID: \"5548598f-3919-417b-ac60-1d74ff41da94\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1e5e75-29890110-zh827" Oct 31 00:30:00.321362 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:00.321268 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0d7eece-c1b5-4f51-9ed7-76d3d0564873-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-101b9e-29890110-7zx97\" (UID: \"f0d7eece-c1b5-4f51-9ed7-76d3d0564873\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-101b9e-29890110-7zx97" Oct 31 00:30:00.321362 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:00.321311 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rl6j\" (UniqueName: \"kubernetes.io/projected/d56a9943-4360-48d4-a701-1bd9b4206908-kube-api-access-4rl6j\") pod \"maas-api-key-cleanup-e2e-worker-w4-da9d07-29890110-r6vzq\" (UID: \"d56a9943-4360-48d4-a701-1bd9b4206908\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da9d07-29890110-r6vzq" Oct 31 00:30:00.336230 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:00.336200 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-412743-29890110-4245m"] Oct 31 00:30:00.422028 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:00.421988 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/156e0827-de4c-4a74-b6f0-9c2cca06ba02-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-e8ba06-29890110-gcpws\" (UID: \"156e0827-de4c-4a74-b6f0-9c2cca06ba02\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e8ba06-29890110-gcpws" Oct 31 00:30:00.422028 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:00.422033 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwr8b\" (UniqueName: \"kubernetes.io/projected/6dbebe96-28d7-4e27-bcb7-9291a84f6c71-kube-api-access-jwr8b\") pod \"maas-api-key-cleanup-e2e-worker-w5-412743-29890110-4245m\" (UID: \"6dbebe96-28d7-4e27-bcb7-9291a84f6c71\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-412743-29890110-4245m" Oct 31 00:30:00.422558 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:00.422069 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d56a9943-4360-48d4-a701-1bd9b4206908-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-da9d07-29890110-r6vzq\" (UID: \"d56a9943-4360-48d4-a701-1bd9b4206908\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da9d07-29890110-r6vzq" Oct 31 00:30:00.422558 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:00.422092 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6dbebe96-28d7-4e27-bcb7-9291a84f6c71-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-412743-29890110-4245m\" (UID: \"6dbebe96-28d7-4e27-bcb7-9291a84f6c71\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-412743-29890110-4245m" Oct 31 00:30:00.422558 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:00.422238 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4sz7\" (UniqueName: \"kubernetes.io/projected/5548598f-3919-417b-ac60-1d74ff41da94-kube-api-access-t4sz7\") pod \"maas-api-key-cleanup-e2e-worker-w3-1e5e75-29890110-zh827\" (UID: \"5548598f-3919-417b-ac60-1d74ff41da94\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1e5e75-29890110-zh827" Oct 31 00:30:00.422558 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:00.422289 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnt24\" (UniqueName: \"kubernetes.io/projected/f0d7eece-c1b5-4f51-9ed7-76d3d0564873-kube-api-access-nnt24\") pod \"maas-api-key-cleanup-e2e-worker-w0-101b9e-29890110-7zx97\" (UID: \"f0d7eece-c1b5-4f51-9ed7-76d3d0564873\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-101b9e-29890110-7zx97" Oct 31 00:30:00.422558 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:00.422326 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6j6p\" (UniqueName: \"kubernetes.io/projected/156e0827-de4c-4a74-b6f0-9c2cca06ba02-kube-api-access-p6j6p\") pod \"maas-api-key-cleanup-e2e-worker-w2-e8ba06-29890110-gcpws\" (UID: \"156e0827-de4c-4a74-b6f0-9c2cca06ba02\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e8ba06-29890110-gcpws" Oct 31 00:30:00.422800 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:00.422580 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5548598f-3919-417b-ac60-1d74ff41da94-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-1e5e75-29890110-zh827\" (UID: \"5548598f-3919-417b-ac60-1d74ff41da94\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1e5e75-29890110-zh827" Oct 31 00:30:00.422800 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:00.422630 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0d7eece-c1b5-4f51-9ed7-76d3d0564873-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-101b9e-29890110-7zx97\" (UID: \"f0d7eece-c1b5-4f51-9ed7-76d3d0564873\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-101b9e-29890110-7zx97" Oct 31 00:30:00.422800 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:00.422701 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rl6j\" (UniqueName: \"kubernetes.io/projected/d56a9943-4360-48d4-a701-1bd9b4206908-kube-api-access-4rl6j\") pod \"maas-api-key-cleanup-e2e-worker-w4-da9d07-29890110-r6vzq\" (UID: \"d56a9943-4360-48d4-a701-1bd9b4206908\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da9d07-29890110-r6vzq" Oct 31 00:30:00.422917 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:00.422879 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/156e0827-de4c-4a74-b6f0-9c2cca06ba02-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-e8ba06-29890110-gcpws\" (UID: \"156e0827-de4c-4a74-b6f0-9c2cca06ba02\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e8ba06-29890110-gcpws" Oct 31 00:30:00.422917 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:00.422903 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6dbebe96-28d7-4e27-bcb7-9291a84f6c71-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-412743-29890110-4245m\" (UID: \"6dbebe96-28d7-4e27-bcb7-9291a84f6c71\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-412743-29890110-4245m" Oct 31 00:30:00.422982 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:00.422903 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d56a9943-4360-48d4-a701-1bd9b4206908-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-da9d07-29890110-r6vzq\" (UID: \"d56a9943-4360-48d4-a701-1bd9b4206908\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da9d07-29890110-r6vzq" Oct 31 00:30:00.423425 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:00.423401 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0d7eece-c1b5-4f51-9ed7-76d3d0564873-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-101b9e-29890110-7zx97\" (UID: \"f0d7eece-c1b5-4f51-9ed7-76d3d0564873\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-101b9e-29890110-7zx97" Oct 31 00:30:00.423471 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:00.423409 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5548598f-3919-417b-ac60-1d74ff41da94-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-1e5e75-29890110-zh827\" (UID: \"5548598f-3919-417b-ac60-1d74ff41da94\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1e5e75-29890110-zh827" Oct 31 00:30:00.486594 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:00.486518 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6j6p\" (UniqueName: \"kubernetes.io/projected/156e0827-de4c-4a74-b6f0-9c2cca06ba02-kube-api-access-p6j6p\") pod \"maas-api-key-cleanup-e2e-worker-w2-e8ba06-29890110-gcpws\" (UID: \"156e0827-de4c-4a74-b6f0-9c2cca06ba02\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e8ba06-29890110-gcpws" Oct 31 00:30:00.486727 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:00.486613 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rl6j\" (UniqueName: \"kubernetes.io/projected/d56a9943-4360-48d4-a701-1bd9b4206908-kube-api-access-4rl6j\") pod \"maas-api-key-cleanup-e2e-worker-w4-da9d07-29890110-r6vzq\" (UID: \"d56a9943-4360-48d4-a701-1bd9b4206908\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da9d07-29890110-r6vzq" Oct 31 00:30:00.486727 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:00.486626 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4sz7\" (UniqueName: \"kubernetes.io/projected/5548598f-3919-417b-ac60-1d74ff41da94-kube-api-access-t4sz7\") pod \"maas-api-key-cleanup-e2e-worker-w3-1e5e75-29890110-zh827\" (UID: \"5548598f-3919-417b-ac60-1d74ff41da94\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1e5e75-29890110-zh827" Oct 31 00:30:00.486799 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:00.486735 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwr8b\" (UniqueName: \"kubernetes.io/projected/6dbebe96-28d7-4e27-bcb7-9291a84f6c71-kube-api-access-jwr8b\") pod \"maas-api-key-cleanup-e2e-worker-w5-412743-29890110-4245m\" (UID: \"6dbebe96-28d7-4e27-bcb7-9291a84f6c71\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-412743-29890110-4245m" Oct 31 00:30:00.486799 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:00.486779 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnt24\" (UniqueName: \"kubernetes.io/projected/f0d7eece-c1b5-4f51-9ed7-76d3d0564873-kube-api-access-nnt24\") pod \"maas-api-key-cleanup-e2e-worker-w0-101b9e-29890110-7zx97\" (UID: \"f0d7eece-c1b5-4f51-9ed7-76d3d0564873\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-101b9e-29890110-7zx97" Oct 31 00:30:00.521548 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:00.521521 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e8ba06-29890110-gcpws" Oct 31 00:30:00.531082 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:00.531053 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da9d07-29890110-r6vzq" Oct 31 00:30:00.539276 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:00.539253 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-101b9e-29890110-7zx97" Oct 31 00:30:00.548806 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:00.546601 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1e5e75-29890110-zh827" Oct 31 00:30:00.569822 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:00.569791 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-412743-29890110-4245m" Oct 31 00:30:00.916990 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:30:00.916954 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd56a9943_4360_48d4_a701_1bd9b4206908.slice/crio-d19d987f421d37b23279041a177336278d28fcd8392beda5651474718d425255 WatchSource:0}: Error finding container d19d987f421d37b23279041a177336278d28fcd8392beda5651474718d425255: Status 404 returned error can't find the container with id d19d987f421d37b23279041a177336278d28fcd8392beda5651474718d425255 Oct 31 00:30:00.918665 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:00.918643 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da9d07-29890110-r6vzq"] Oct 31 00:30:00.963191 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:30:00.963150 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod156e0827_de4c_4a74_b6f0_9c2cca06ba02.slice/crio-0d7b0e38255bf1243501455c74272d21abf7fb43330f0c32fea8b10b8126b398 WatchSource:0}: Error finding container 0d7b0e38255bf1243501455c74272d21abf7fb43330f0c32fea8b10b8126b398: Status 404 returned error can't find the container with id 0d7b0e38255bf1243501455c74272d21abf7fb43330f0c32fea8b10b8126b398 Oct 31 00:30:00.966298 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:00.966280 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e8ba06-29890110-gcpws"] Oct 31 00:30:01.014274 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:30:01.014238 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dbebe96_28d7_4e27_bcb7_9291a84f6c71.slice/crio-e4756721b9f5e888574e6fe7647fc139d19954f9984512f1232b4b7826fbbc99 WatchSource:0}: Error finding container e4756721b9f5e888574e6fe7647fc139d19954f9984512f1232b4b7826fbbc99: Status 404 returned error can't find the container with id e4756721b9f5e888574e6fe7647fc139d19954f9984512f1232b4b7826fbbc99 Oct 31 00:30:01.015431 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:01.015403 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-412743-29890110-4245m"] Oct 31 00:30:01.063491 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:30:01.063440 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5548598f_3919_417b_ac60_1d74ff41da94.slice/crio-ced7174181fe75b113ead98cb79bf82b50e9f3992a4a66266b693eff9d0c6c5e WatchSource:0}: Error finding container ced7174181fe75b113ead98cb79bf82b50e9f3992a4a66266b693eff9d0c6c5e: Status 404 returned error can't find the container with id ced7174181fe75b113ead98cb79bf82b50e9f3992a4a66266b693eff9d0c6c5e Oct 31 00:30:01.064489 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:30:01.064459 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0d7eece_c1b5_4f51_9ed7_76d3d0564873.slice/crio-09381f848c736572c2771540d2959ef929a3fefd3f17629f59344b60075a9074 WatchSource:0}: Error finding container 09381f848c736572c2771540d2959ef929a3fefd3f17629f59344b60075a9074: Status 404 returned error can't find the container with id 09381f848c736572c2771540d2959ef929a3fefd3f17629f59344b60075a9074 Oct 31 00:30:01.066519 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:01.066495 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-101b9e-29890110-7zx97"] Oct 31 00:30:01.068488 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:01.068466 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1e5e75-29890110-zh827"] Oct 31 00:30:01.925294 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:01.925256 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da9d07-29890110-r6vzq" event={"ID":"d56a9943-4360-48d4-a701-1bd9b4206908","Type":"ContainerStarted","Data":"d19d987f421d37b23279041a177336278d28fcd8392beda5651474718d425255"} Oct 31 00:30:01.926492 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:01.926463 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1e5e75-29890110-zh827" event={"ID":"5548598f-3919-417b-ac60-1d74ff41da94","Type":"ContainerStarted","Data":"ced7174181fe75b113ead98cb79bf82b50e9f3992a4a66266b693eff9d0c6c5e"} Oct 31 00:30:01.927603 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:01.927581 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-412743-29890110-4245m" event={"ID":"6dbebe96-28d7-4e27-bcb7-9291a84f6c71","Type":"ContainerStarted","Data":"e4756721b9f5e888574e6fe7647fc139d19954f9984512f1232b4b7826fbbc99"} Oct 31 00:30:01.928743 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:01.928719 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-101b9e-29890110-7zx97" event={"ID":"f0d7eece-c1b5-4f51-9ed7-76d3d0564873","Type":"ContainerStarted","Data":"09381f848c736572c2771540d2959ef929a3fefd3f17629f59344b60075a9074"} Oct 31 00:30:01.929769 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:01.929752 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e8ba06-29890110-gcpws" event={"ID":"156e0827-de4c-4a74-b6f0-9c2cca06ba02","Type":"ContainerStarted","Data":"0d7b0e38255bf1243501455c74272d21abf7fb43330f0c32fea8b10b8126b398"} Oct 31 00:30:05.901308 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:30:05.901267 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 00:30:07.760124 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:07.760094 5222 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h9hsf_0f8c3c03-a1fc-4376-98c6-feeeefa08068/ovn-acl-logging/0.log" Oct 31 00:30:07.761482 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:07.761456 5222 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h9hsf_0f8c3c03-a1fc-4376-98c6-feeeefa08068/ovn-acl-logging/0.log" Oct 31 00:30:07.976711 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:07.976675 5222 generic.go:356] "Generic (PLEG): container finished" podID="d56a9943-4360-48d4-a701-1bd9b4206908" containerID="05b44ac736edfbb01d5b2b635ea792ae2887c78db61850be07097289f29e6cac" exitCode=60 Oct 31 00:30:07.976901 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:07.976760 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da9d07-29890110-r6vzq" event={"ID":"d56a9943-4360-48d4-a701-1bd9b4206908","Type":"ContainerDied","Data":"05b44ac736edfbb01d5b2b635ea792ae2887c78db61850be07097289f29e6cac"} Oct 31 00:30:07.977013 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:07.976998 5222 scope.go:122] "RemoveContainer" containerID="05b44ac736edfbb01d5b2b635ea792ae2887c78db61850be07097289f29e6cac" Oct 31 00:30:07.978463 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:07.978440 5222 generic.go:356] "Generic (PLEG): container finished" podID="5548598f-3919-417b-ac60-1d74ff41da94" containerID="2c85cb15670ee31d7f83f83cef17b7a8560ac3373d3d856a2a04c6f766fa189d" exitCode=60 Oct 31 00:30:07.978573 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:07.978515 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1e5e75-29890110-zh827" event={"ID":"5548598f-3919-417b-ac60-1d74ff41da94","Type":"ContainerDied","Data":"2c85cb15670ee31d7f83f83cef17b7a8560ac3373d3d856a2a04c6f766fa189d"} Oct 31 00:30:07.978752 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:07.978736 5222 scope.go:122] "RemoveContainer" containerID="2c85cb15670ee31d7f83f83cef17b7a8560ac3373d3d856a2a04c6f766fa189d" Oct 31 00:30:07.980327 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:07.980308 5222 generic.go:356] "Generic (PLEG): container finished" podID="6dbebe96-28d7-4e27-bcb7-9291a84f6c71" containerID="381a6574b56eeb9f81bf87536c096673eb0b21341eebb294a93e53d6baf287b4" exitCode=60 Oct 31 00:30:07.980411 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:07.980382 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-412743-29890110-4245m" event={"ID":"6dbebe96-28d7-4e27-bcb7-9291a84f6c71","Type":"ContainerDied","Data":"381a6574b56eeb9f81bf87536c096673eb0b21341eebb294a93e53d6baf287b4"} Oct 31 00:30:07.980639 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:07.980623 5222 scope.go:122] "RemoveContainer" containerID="381a6574b56eeb9f81bf87536c096673eb0b21341eebb294a93e53d6baf287b4" Oct 31 00:30:07.982045 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:07.982025 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-101b9e-29890110-7zx97" event={"ID":"f0d7eece-c1b5-4f51-9ed7-76d3d0564873","Type":"ContainerStarted","Data":"c0634017a7d0d03a592d6e533be220ad4b16ca2e4f5fc8574b45c160f758af04"} Oct 31 00:30:07.983591 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:07.983573 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e8ba06-29890110-gcpws" event={"ID":"156e0827-de4c-4a74-b6f0-9c2cca06ba02","Type":"ContainerStarted","Data":"1b3ab56fbd2e1be3012043ec2fa390da16dab195b100571979b97ece3e04f5ff"} Oct 31 00:30:08.453535 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:08.453487 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e8ba06-29890110-gcpws" podStartSLOduration=2.016090001 podStartE2EDuration="8.453473058s" totalImagesPullingTime="6.437383057s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:30:08.452547063 +0000 UTC m=+900.923716813" watchObservedRunningTime="2026-10-31 00:30:08.453473058 +0000 UTC m=+900.924642867" Oct 31 00:30:08.571189 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:08.571136 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-101b9e-29890110-7zx97" podStartSLOduration=2.406990566 podStartE2EDuration="8.571119967s" totalImagesPullingTime="6.164129401s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:30:08.570460734 +0000 UTC m=+901.041630483" watchObservedRunningTime="2026-10-31 00:30:08.571119967 +0000 UTC m=+901.042289716" Oct 31 00:30:08.932449 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:08.932362 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-79tk8"] Oct 31 00:30:08.991398 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:08.991337 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-412743-29890110-4245m" event={"ID":"6dbebe96-28d7-4e27-bcb7-9291a84f6c71","Type":"ContainerStarted","Data":"55eff768534f35160edc1c1bbd8116929e3501bfa89bf22b0dd0b337b87e25ce"} Oct 31 00:30:08.992854 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:08.992822 5222 generic.go:356] "Generic (PLEG): container finished" podID="f0d7eece-c1b5-4f51-9ed7-76d3d0564873" containerID="c0634017a7d0d03a592d6e533be220ad4b16ca2e4f5fc8574b45c160f758af04" exitCode=60 Oct 31 00:30:08.992998 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:08.992908 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-101b9e-29890110-7zx97" event={"ID":"f0d7eece-c1b5-4f51-9ed7-76d3d0564873","Type":"ContainerDied","Data":"c0634017a7d0d03a592d6e533be220ad4b16ca2e4f5fc8574b45c160f758af04"} Oct 31 00:30:08.993146 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:08.993130 5222 scope.go:122] "RemoveContainer" containerID="c0634017a7d0d03a592d6e533be220ad4b16ca2e4f5fc8574b45c160f758af04" Oct 31 00:30:08.994707 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:08.994610 5222 generic.go:356] "Generic (PLEG): container finished" podID="156e0827-de4c-4a74-b6f0-9c2cca06ba02" containerID="1b3ab56fbd2e1be3012043ec2fa390da16dab195b100571979b97ece3e04f5ff" exitCode=60 Oct 31 00:30:08.994707 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:08.994695 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e8ba06-29890110-gcpws" event={"ID":"156e0827-de4c-4a74-b6f0-9c2cca06ba02","Type":"ContainerDied","Data":"1b3ab56fbd2e1be3012043ec2fa390da16dab195b100571979b97ece3e04f5ff"} Oct 31 00:30:08.994882 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:08.994864 5222 scope.go:122] "RemoveContainer" containerID="1b3ab56fbd2e1be3012043ec2fa390da16dab195b100571979b97ece3e04f5ff" Oct 31 00:30:08.996708 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:08.996672 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da9d07-29890110-r6vzq" event={"ID":"d56a9943-4360-48d4-a701-1bd9b4206908","Type":"ContainerStarted","Data":"1e919de179f400efa9306a2ccb905de003354e93667aa31e774e9a16d66f003b"} Oct 31 00:30:08.998665 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:08.998643 5222 generic.go:356] "Generic (PLEG): container finished" podID="5548598f-3919-417b-ac60-1d74ff41da94" containerID="8f1aeace78547c9ed57265d038b9c386781ab4671443ceb5852037d7a37c6b7f" exitCode=60 Oct 31 00:30:08.998732 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:08.998680 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1e5e75-29890110-zh827" event={"ID":"5548598f-3919-417b-ac60-1d74ff41da94","Type":"ContainerDied","Data":"8f1aeace78547c9ed57265d038b9c386781ab4671443ceb5852037d7a37c6b7f"} Oct 31 00:30:08.998732 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:08.998715 5222 scope.go:122] "RemoveContainer" containerID="2c85cb15670ee31d7f83f83cef17b7a8560ac3373d3d856a2a04c6f766fa189d" Oct 31 00:30:08.998966 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:08.998947 5222 scope.go:122] "RemoveContainer" containerID="8f1aeace78547c9ed57265d038b9c386781ab4671443ceb5852037d7a37c6b7f" Oct 31 00:30:08.999179 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:30:08.999157 5222 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-1e5e75-29890110-zh827_odh-ai-gateway-infra(5548598f-3919-417b-ac60-1d74ff41da94)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1e5e75-29890110-zh827" podUID="5548598f-3919-417b-ac60-1d74ff41da94" Oct 31 00:30:09.112732 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:09.112635 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-412743-29890110-4245m" podStartSLOduration=3.480687832 podStartE2EDuration="9.112616161s" totalImagesPullingTime="5.631928329s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:30:09.110629609 +0000 UTC m=+901.581799357" watchObservedRunningTime="2026-10-31 00:30:09.112616161 +0000 UTC m=+901.583785955" Oct 31 00:30:09.438881 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:09.438812 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-7lvcw"] Oct 31 00:30:09.443209 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:09.443185 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-7lvcw" Oct 31 00:30:09.472061 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:09.468937 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da9d07-29890110-r6vzq" podStartSLOduration=3.225576609 podStartE2EDuration="9.468915605s" totalImagesPullingTime="6.243338996s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:30:09.465242548 +0000 UTC m=+901.936412298" watchObservedRunningTime="2026-10-31 00:30:09.468915605 +0000 UTC m=+901.940085355" Oct 31 00:30:09.497720 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:09.497646 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-a6f4a5-openshift-default-dockercfg-r4d7m\"" Oct 31 00:30:09.500406 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:09.500334 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-7lvcw"] Oct 31 00:30:09.537426 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:09.537392 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65f6cbd7-9324-4459-8820-86b6bae5f8a2-istio-envoy\") pod \"e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-7lvcw\" (UID: \"65f6cbd7-9324-4459-8820-86b6bae5f8a2\") " pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-7lvcw" Oct 31 00:30:09.537571 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:09.537477 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65f6cbd7-9324-4459-8820-86b6bae5f8a2-istio-data\") pod \"e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-7lvcw\" (UID: \"65f6cbd7-9324-4459-8820-86b6bae5f8a2\") " pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-7lvcw" Oct 31 00:30:09.537571 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:09.537530 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntx2k\" (UniqueName: \"kubernetes.io/projected/65f6cbd7-9324-4459-8820-86b6bae5f8a2-kube-api-access-ntx2k\") pod \"e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-7lvcw\" (UID: \"65f6cbd7-9324-4459-8820-86b6bae5f8a2\") " pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-7lvcw" Oct 31 00:30:09.537571 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:09.537547 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65f6cbd7-9324-4459-8820-86b6bae5f8a2-credential-socket\") pod \"e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-7lvcw\" (UID: \"65f6cbd7-9324-4459-8820-86b6bae5f8a2\") " pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-7lvcw" Oct 31 00:30:09.537571 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:09.537566 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65f6cbd7-9324-4459-8820-86b6bae5f8a2-istio-podinfo\") pod \"e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-7lvcw\" (UID: \"65f6cbd7-9324-4459-8820-86b6bae5f8a2\") " pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-7lvcw" Oct 31 00:30:09.537767 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:09.537584 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65f6cbd7-9324-4459-8820-86b6bae5f8a2-istio-token\") pod \"e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-7lvcw\" (UID: \"65f6cbd7-9324-4459-8820-86b6bae5f8a2\") " pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-7lvcw" Oct 31 00:30:09.537767 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:09.537603 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65f6cbd7-9324-4459-8820-86b6bae5f8a2-workload-certs\") pod \"e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-7lvcw\" (UID: \"65f6cbd7-9324-4459-8820-86b6bae5f8a2\") " pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-7lvcw" Oct 31 00:30:09.537767 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:09.537622 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65f6cbd7-9324-4459-8820-86b6bae5f8a2-workload-socket\") pod \"e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-7lvcw\" (UID: \"65f6cbd7-9324-4459-8820-86b6bae5f8a2\") " pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-7lvcw" Oct 31 00:30:09.537888 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:09.537809 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65f6cbd7-9324-4459-8820-86b6bae5f8a2-istiod-ca-cert\") pod \"e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-7lvcw\" (UID: \"65f6cbd7-9324-4459-8820-86b6bae5f8a2\") " pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-7lvcw" Oct 31 00:30:09.638524 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:09.638487 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntx2k\" (UniqueName: \"kubernetes.io/projected/65f6cbd7-9324-4459-8820-86b6bae5f8a2-kube-api-access-ntx2k\") pod \"e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-7lvcw\" (UID: \"65f6cbd7-9324-4459-8820-86b6bae5f8a2\") " pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-7lvcw" Oct 31 00:30:09.638524 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:09.638523 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65f6cbd7-9324-4459-8820-86b6bae5f8a2-credential-socket\") pod \"e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-7lvcw\" (UID: \"65f6cbd7-9324-4459-8820-86b6bae5f8a2\") " pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-7lvcw" Oct 31 00:30:09.638795 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:09.638547 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65f6cbd7-9324-4459-8820-86b6bae5f8a2-istio-podinfo\") pod \"e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-7lvcw\" (UID: \"65f6cbd7-9324-4459-8820-86b6bae5f8a2\") " pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-7lvcw" Oct 31 00:30:09.638795 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:09.638563 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65f6cbd7-9324-4459-8820-86b6bae5f8a2-istio-token\") pod \"e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-7lvcw\" (UID: \"65f6cbd7-9324-4459-8820-86b6bae5f8a2\") " pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-7lvcw" Oct 31 00:30:09.638907 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:09.638821 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65f6cbd7-9324-4459-8820-86b6bae5f8a2-workload-certs\") pod \"e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-7lvcw\" (UID: \"65f6cbd7-9324-4459-8820-86b6bae5f8a2\") " pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-7lvcw" Oct 31 00:30:09.638907 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:09.638868 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65f6cbd7-9324-4459-8820-86b6bae5f8a2-workload-socket\") pod \"e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-7lvcw\" (UID: \"65f6cbd7-9324-4459-8820-86b6bae5f8a2\") " pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-7lvcw" Oct 31 00:30:09.639016 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:09.638938 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65f6cbd7-9324-4459-8820-86b6bae5f8a2-istiod-ca-cert\") pod \"e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-7lvcw\" (UID: \"65f6cbd7-9324-4459-8820-86b6bae5f8a2\") " pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-7lvcw" Oct 31 00:30:09.639016 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:09.638962 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65f6cbd7-9324-4459-8820-86b6bae5f8a2-credential-socket\") pod \"e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-7lvcw\" (UID: \"65f6cbd7-9324-4459-8820-86b6bae5f8a2\") " pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-7lvcw" Oct 31 00:30:09.639128 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:09.639013 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65f6cbd7-9324-4459-8820-86b6bae5f8a2-istio-envoy\") pod \"e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-7lvcw\" (UID: \"65f6cbd7-9324-4459-8820-86b6bae5f8a2\") " pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-7lvcw" Oct 31 00:30:09.639191 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:09.639132 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65f6cbd7-9324-4459-8820-86b6bae5f8a2-istio-data\") pod \"e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-7lvcw\" (UID: \"65f6cbd7-9324-4459-8820-86b6bae5f8a2\") " pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-7lvcw" Oct 31 00:30:09.639191 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:09.639135 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65f6cbd7-9324-4459-8820-86b6bae5f8a2-workload-certs\") pod \"e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-7lvcw\" (UID: \"65f6cbd7-9324-4459-8820-86b6bae5f8a2\") " pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-7lvcw" Oct 31 00:30:09.639298 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:09.639283 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65f6cbd7-9324-4459-8820-86b6bae5f8a2-workload-socket\") pod \"e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-7lvcw\" (UID: \"65f6cbd7-9324-4459-8820-86b6bae5f8a2\") " pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-7lvcw" Oct 31 00:30:09.639536 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:09.639514 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65f6cbd7-9324-4459-8820-86b6bae5f8a2-istio-data\") pod \"e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-7lvcw\" (UID: \"65f6cbd7-9324-4459-8820-86b6bae5f8a2\") " pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-7lvcw" Oct 31 00:30:09.639674 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:09.639645 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65f6cbd7-9324-4459-8820-86b6bae5f8a2-istiod-ca-cert\") pod \"e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-7lvcw\" (UID: \"65f6cbd7-9324-4459-8820-86b6bae5f8a2\") " pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-7lvcw" Oct 31 00:30:09.641648 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:09.641618 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65f6cbd7-9324-4459-8820-86b6bae5f8a2-istio-podinfo\") pod \"e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-7lvcw\" (UID: \"65f6cbd7-9324-4459-8820-86b6bae5f8a2\") " pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-7lvcw" Oct 31 00:30:09.641738 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:09.641658 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65f6cbd7-9324-4459-8820-86b6bae5f8a2-istio-envoy\") pod \"e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-7lvcw\" (UID: \"65f6cbd7-9324-4459-8820-86b6bae5f8a2\") " pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-7lvcw" Oct 31 00:30:09.701615 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:09.701535 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65f6cbd7-9324-4459-8820-86b6bae5f8a2-istio-token\") pod \"e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-7lvcw\" (UID: \"65f6cbd7-9324-4459-8820-86b6bae5f8a2\") " pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-7lvcw" Oct 31 00:30:09.701764 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:09.701639 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntx2k\" (UniqueName: \"kubernetes.io/projected/65f6cbd7-9324-4459-8820-86b6bae5f8a2-kube-api-access-ntx2k\") pod \"e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-7lvcw\" (UID: \"65f6cbd7-9324-4459-8820-86b6bae5f8a2\") " pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-7lvcw" Oct 31 00:30:09.773243 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:09.773211 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-7lvcw" Oct 31 00:30:10.006045 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:10.005963 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e8ba06-29890110-gcpws" event={"ID":"156e0827-de4c-4a74-b6f0-9c2cca06ba02","Type":"ContainerStarted","Data":"713135eba189a105631a14586d69e0ee661625ec9d0aa272971946c53e21cf80"} Oct 31 00:30:10.007876 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:10.007852 5222 generic.go:356] "Generic (PLEG): container finished" podID="d56a9943-4360-48d4-a701-1bd9b4206908" containerID="1e919de179f400efa9306a2ccb905de003354e93667aa31e774e9a16d66f003b" exitCode=60 Oct 31 00:30:10.008010 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:10.007925 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da9d07-29890110-r6vzq" event={"ID":"d56a9943-4360-48d4-a701-1bd9b4206908","Type":"ContainerDied","Data":"1e919de179f400efa9306a2ccb905de003354e93667aa31e774e9a16d66f003b"} Oct 31 00:30:10.008010 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:10.007973 5222 scope.go:122] "RemoveContainer" containerID="05b44ac736edfbb01d5b2b635ea792ae2887c78db61850be07097289f29e6cac" Oct 31 00:30:10.008223 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:10.008205 5222 scope.go:122] "RemoveContainer" containerID="1e919de179f400efa9306a2ccb905de003354e93667aa31e774e9a16d66f003b" Oct 31 00:30:10.008519 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:30:10.008498 5222 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-da9d07-29890110-r6vzq_odh-ai-gateway-infra(d56a9943-4360-48d4-a701-1bd9b4206908)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da9d07-29890110-r6vzq" podUID="d56a9943-4360-48d4-a701-1bd9b4206908" Oct 31 00:30:10.010028 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:10.010011 5222 scope.go:122] "RemoveContainer" containerID="8f1aeace78547c9ed57265d038b9c386781ab4671443ceb5852037d7a37c6b7f" Oct 31 00:30:10.010294 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:30:10.010259 5222 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-1e5e75-29890110-zh827_odh-ai-gateway-infra(5548598f-3919-417b-ac60-1d74ff41da94)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1e5e75-29890110-zh827" podUID="5548598f-3919-417b-ac60-1d74ff41da94" Oct 31 00:30:10.011681 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:10.011661 5222 generic.go:356] "Generic (PLEG): container finished" podID="6dbebe96-28d7-4e27-bcb7-9291a84f6c71" containerID="55eff768534f35160edc1c1bbd8116929e3501bfa89bf22b0dd0b337b87e25ce" exitCode=60 Oct 31 00:30:10.011774 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:10.011750 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-412743-29890110-4245m" event={"ID":"6dbebe96-28d7-4e27-bcb7-9291a84f6c71","Type":"ContainerDied","Data":"55eff768534f35160edc1c1bbd8116929e3501bfa89bf22b0dd0b337b87e25ce"} Oct 31 00:30:10.011945 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:10.011930 5222 scope.go:122] "RemoveContainer" containerID="55eff768534f35160edc1c1bbd8116929e3501bfa89bf22b0dd0b337b87e25ce" Oct 31 00:30:10.012195 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:30:10.012167 5222 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-412743-29890110-4245m_odh-ai-gateway-infra(6dbebe96-28d7-4e27-bcb7-9291a84f6c71)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-412743-29890110-4245m" podUID="6dbebe96-28d7-4e27-bcb7-9291a84f6c71" Oct 31 00:30:10.013921 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:10.013903 5222 generic.go:356] "Generic (PLEG): container finished" podID="f0d7eece-c1b5-4f51-9ed7-76d3d0564873" containerID="a01166a19e01bc6f2adad774411a394a45e9e6fbb21d005383a040e3c27d9723" exitCode=60 Oct 31 00:30:10.014005 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:10.013953 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-101b9e-29890110-7zx97" event={"ID":"f0d7eece-c1b5-4f51-9ed7-76d3d0564873","Type":"ContainerDied","Data":"a01166a19e01bc6f2adad774411a394a45e9e6fbb21d005383a040e3c27d9723"} Oct 31 00:30:10.014163 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:10.014149 5222 scope.go:122] "RemoveContainer" containerID="a01166a19e01bc6f2adad774411a394a45e9e6fbb21d005383a040e3c27d9723" Oct 31 00:30:10.014356 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:30:10.014342 5222 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-101b9e-29890110-7zx97_odh-ai-gateway-infra(f0d7eece-c1b5-4f51-9ed7-76d3d0564873)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-101b9e-29890110-7zx97" podUID="f0d7eece-c1b5-4f51-9ed7-76d3d0564873" Oct 31 00:30:10.020398 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:10.020363 5222 scope.go:122] "RemoveContainer" containerID="381a6574b56eeb9f81bf87536c096673eb0b21341eebb294a93e53d6baf287b4" Oct 31 00:30:10.033614 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:10.033593 5222 scope.go:122] "RemoveContainer" containerID="c0634017a7d0d03a592d6e533be220ad4b16ca2e4f5fc8574b45c160f758af04" Oct 31 00:30:10.203874 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:30:10.203833 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65f6cbd7_9324_4459_8820_86b6bae5f8a2.slice/crio-afb299c366973aeb9b962d77518413f7bc20c870ab932a24ec373581301cdcd0 WatchSource:0}: Error finding container afb299c366973aeb9b962d77518413f7bc20c870ab932a24ec373581301cdcd0: Status 404 returned error can't find the container with id afb299c366973aeb9b962d77518413f7bc20c870ab932a24ec373581301cdcd0 Oct 31 00:30:10.205844 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:10.205824 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-7lvcw"] Oct 31 00:30:10.206450 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:10.206421 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:30:10.206488 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:10.206481 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:30:10.206535 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:10.206507 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:30:11.021842 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:11.021812 5222 scope.go:122] "RemoveContainer" containerID="55eff768534f35160edc1c1bbd8116929e3501bfa89bf22b0dd0b337b87e25ce" Oct 31 00:30:11.022297 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:30:11.022105 5222 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-412743-29890110-4245m_odh-ai-gateway-infra(6dbebe96-28d7-4e27-bcb7-9291a84f6c71)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-412743-29890110-4245m" podUID="6dbebe96-28d7-4e27-bcb7-9291a84f6c71" Oct 31 00:30:11.023425 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:11.023406 5222 scope.go:122] "RemoveContainer" containerID="a01166a19e01bc6f2adad774411a394a45e9e6fbb21d005383a040e3c27d9723" Oct 31 00:30:11.023686 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:30:11.023666 5222 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-101b9e-29890110-7zx97_odh-ai-gateway-infra(f0d7eece-c1b5-4f51-9ed7-76d3d0564873)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-101b9e-29890110-7zx97" podUID="f0d7eece-c1b5-4f51-9ed7-76d3d0564873" Oct 31 00:30:11.024981 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:11.024959 5222 generic.go:356] "Generic (PLEG): container finished" podID="156e0827-de4c-4a74-b6f0-9c2cca06ba02" containerID="713135eba189a105631a14586d69e0ee661625ec9d0aa272971946c53e21cf80" exitCode=60 Oct 31 00:30:11.025066 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:11.025040 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e8ba06-29890110-gcpws" event={"ID":"156e0827-de4c-4a74-b6f0-9c2cca06ba02","Type":"ContainerDied","Data":"713135eba189a105631a14586d69e0ee661625ec9d0aa272971946c53e21cf80"} Oct 31 00:30:11.025137 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:11.025123 5222 scope.go:122] "RemoveContainer" containerID="1b3ab56fbd2e1be3012043ec2fa390da16dab195b100571979b97ece3e04f5ff" Oct 31 00:30:11.025238 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:11.025222 5222 scope.go:122] "RemoveContainer" containerID="713135eba189a105631a14586d69e0ee661625ec9d0aa272971946c53e21cf80" Oct 31 00:30:11.025505 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:30:11.025485 5222 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-e8ba06-29890110-gcpws_odh-ai-gateway-infra(156e0827-de4c-4a74-b6f0-9c2cca06ba02)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e8ba06-29890110-gcpws" podUID="156e0827-de4c-4a74-b6f0-9c2cca06ba02" Oct 31 00:30:11.026831 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:11.026814 5222 scope.go:122] "RemoveContainer" containerID="1e919de179f400efa9306a2ccb905de003354e93667aa31e774e9a16d66f003b" Oct 31 00:30:11.027046 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:30:11.027028 5222 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-da9d07-29890110-r6vzq_odh-ai-gateway-infra(d56a9943-4360-48d4-a701-1bd9b4206908)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da9d07-29890110-r6vzq" podUID="d56a9943-4360-48d4-a701-1bd9b4206908" Oct 31 00:30:11.028232 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:11.028206 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-7lvcw" event={"ID":"65f6cbd7-9324-4459-8820-86b6bae5f8a2","Type":"ContainerStarted","Data":"967352fb7fb171892c398ba632590e5522e80327263ecae9b0005f0e21178005"} Oct 31 00:30:11.028318 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:11.028257 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-7lvcw" event={"ID":"65f6cbd7-9324-4459-8820-86b6bae5f8a2","Type":"ContainerStarted","Data":"afb299c366973aeb9b962d77518413f7bc20c870ab932a24ec373581301cdcd0"} Oct 31 00:30:11.282149 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:11.282034 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-7lvcw" podStartSLOduration=2.282015805 podStartE2EDuration="2.282015805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:30:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:30:11.280473548 +0000 UTC m=+903.751643323" watchObservedRunningTime="2026-10-31 00:30:11.282015805 +0000 UTC m=+903.753185554" Oct 31 00:30:11.773966 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:11.773933 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-7lvcw" Oct 31 00:30:11.775299 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:11.775270 5222 patch_prober.go:29] interesting pod/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-7lvcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 31 00:30:11.775443 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:11.775328 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-7lvcw" podUID="65f6cbd7-9324-4459-8820-86b6bae5f8a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 31 00:30:12.035969 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:12.035892 5222 scope.go:122] "RemoveContainer" containerID="713135eba189a105631a14586d69e0ee661625ec9d0aa272971946c53e21cf80" Oct 31 00:30:12.036319 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:30:12.036155 5222 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-e8ba06-29890110-gcpws_odh-ai-gateway-infra(156e0827-de4c-4a74-b6f0-9c2cca06ba02)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e8ba06-29890110-gcpws" podUID="156e0827-de4c-4a74-b6f0-9c2cca06ba02" Oct 31 00:30:12.774319 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:12.774285 5222 patch_prober.go:29] interesting pod/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-7lvcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 31 00:30:12.774535 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:12.774340 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-7lvcw" podUID="65f6cbd7-9324-4459-8820-86b6bae5f8a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 31 00:30:13.781161 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:13.781122 5222 patch_prober.go:29] interesting pod/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-7lvcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 31 00:30:13.781670 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:13.781179 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-7lvcw" podUID="65f6cbd7-9324-4459-8820-86b6bae5f8a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 31 00:30:14.774073 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:14.773995 5222 patch_prober.go:29] interesting pod/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-7lvcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 31 00:30:14.774073 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:14.774055 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-7lvcw" podUID="65f6cbd7-9324-4459-8820-86b6bae5f8a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 31 00:30:15.774669 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:15.774634 5222 patch_prober.go:29] interesting pod/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-7lvcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 31 00:30:15.775083 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:15.774709 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-7lvcw" podUID="65f6cbd7-9324-4459-8820-86b6bae5f8a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 31 00:30:16.967321 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:16.967294 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-7lvcw" Oct 31 00:30:16.967840 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:16.967782 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-7lvcw" Oct 31 00:30:16.969783 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:16.969759 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-7lvcw" Oct 31 00:30:19.649030 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:19.648995 5222 scope.go:122] "RemoveContainer" containerID="8f1aeace78547c9ed57265d038b9c386781ab4671443ceb5852037d7a37c6b7f" Oct 31 00:30:20.108450 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:20.108400 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1e5e75-29890110-zh827" event={"ID":"5548598f-3919-417b-ac60-1d74ff41da94","Type":"ContainerStarted","Data":"6236e97cb30e473603a97e449be2a4973924331d8bf237ad99f813418e437e7f"} Oct 31 00:30:20.233077 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:20.232974 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1e5e75-29890110-zh827" podStartSLOduration=14.654497243 podStartE2EDuration="20.232959823s" totalImagesPullingTime="5.57846258s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:30:20.231851113 +0000 UTC m=+912.703020863" watchObservedRunningTime="2026-10-31 00:30:20.232959823 +0000 UTC m=+912.704129571" Oct 31 00:30:20.649675 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:20.649645 5222 scope.go:122] "RemoveContainer" containerID="1e919de179f400efa9306a2ccb905de003354e93667aa31e774e9a16d66f003b" Oct 31 00:30:21.116687 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:21.116590 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da9d07-29890110-r6vzq" event={"ID":"d56a9943-4360-48d4-a701-1bd9b4206908","Type":"ContainerStarted","Data":"b388581047012a3e31617a086f2821d488867ee388f75abf6bbe3184472297b1"} Oct 31 00:30:21.118443 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:21.118416 5222 generic.go:356] "Generic (PLEG): container finished" podID="5548598f-3919-417b-ac60-1d74ff41da94" containerID="6236e97cb30e473603a97e449be2a4973924331d8bf237ad99f813418e437e7f" exitCode=60 Oct 31 00:30:21.118592 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:21.118483 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1e5e75-29890110-zh827" event={"ID":"5548598f-3919-417b-ac60-1d74ff41da94","Type":"ContainerDied","Data":"6236e97cb30e473603a97e449be2a4973924331d8bf237ad99f813418e437e7f"} Oct 31 00:30:21.118592 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:21.118529 5222 scope.go:122] "RemoveContainer" containerID="8f1aeace78547c9ed57265d038b9c386781ab4671443ceb5852037d7a37c6b7f" Oct 31 00:30:21.118879 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:21.118858 5222 scope.go:122] "RemoveContainer" containerID="6236e97cb30e473603a97e449be2a4973924331d8bf237ad99f813418e437e7f" Oct 31 00:30:21.119177 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:30:21.119147 5222 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-1e5e75-29890110-zh827_odh-ai-gateway-infra(5548598f-3919-417b-ac60-1d74ff41da94)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1e5e75-29890110-zh827" podUID="5548598f-3919-417b-ac60-1d74ff41da94" Oct 31 00:30:21.246204 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:21.246171 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1e5e75-29890110-zh827"] Oct 31 00:30:22.130147 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:22.130112 5222 generic.go:356] "Generic (PLEG): container finished" podID="d56a9943-4360-48d4-a701-1bd9b4206908" containerID="b388581047012a3e31617a086f2821d488867ee388f75abf6bbe3184472297b1" exitCode=60 Oct 31 00:30:22.130559 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:22.130184 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da9d07-29890110-r6vzq" event={"ID":"d56a9943-4360-48d4-a701-1bd9b4206908","Type":"ContainerDied","Data":"b388581047012a3e31617a086f2821d488867ee388f75abf6bbe3184472297b1"} Oct 31 00:30:22.130559 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:22.130261 5222 scope.go:122] "RemoveContainer" containerID="1e919de179f400efa9306a2ccb905de003354e93667aa31e774e9a16d66f003b" Oct 31 00:30:22.130743 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:22.130726 5222 scope.go:122] "RemoveContainer" containerID="b388581047012a3e31617a086f2821d488867ee388f75abf6bbe3184472297b1" Oct 31 00:30:22.131099 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:30:22.131050 5222 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-da9d07-29890110-r6vzq_odh-ai-gateway-infra(d56a9943-4360-48d4-a701-1bd9b4206908)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da9d07-29890110-r6vzq" podUID="d56a9943-4360-48d4-a701-1bd9b4206908" Oct 31 00:30:22.266640 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:22.266432 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da9d07-29890110-r6vzq"] Oct 31 00:30:22.367459 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:22.367427 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1e5e75-29890110-zh827" Oct 31 00:30:22.421416 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:22.421293 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5548598f-3919-417b-ac60-1d74ff41da94-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5548598f-3919-417b-ac60-1d74ff41da94-maas-api-service-ca\") pod \"5548598f-3919-417b-ac60-1d74ff41da94\" (UID: \"5548598f-3919-417b-ac60-1d74ff41da94\") " Oct 31 00:30:22.421586 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:22.421429 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5548598f-3919-417b-ac60-1d74ff41da94-kube-api-access-t4sz7\" (UniqueName: \"kubernetes.io/projected/5548598f-3919-417b-ac60-1d74ff41da94-kube-api-access-t4sz7\") pod \"5548598f-3919-417b-ac60-1d74ff41da94\" (UID: \"5548598f-3919-417b-ac60-1d74ff41da94\") " Oct 31 00:30:22.421725 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:22.421698 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5548598f-3919-417b-ac60-1d74ff41da94-maas-api-service-ca" pod "5548598f-3919-417b-ac60-1d74ff41da94" (UID: "5548598f-3919-417b-ac60-1d74ff41da94"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:30:22.421909 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:22.421888 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5548598f-3919-417b-ac60-1d74ff41da94-maas-api-service-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:22.423939 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:22.423913 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5548598f-3919-417b-ac60-1d74ff41da94-kube-api-access-t4sz7" pod "5548598f-3919-417b-ac60-1d74ff41da94" (UID: "5548598f-3919-417b-ac60-1d74ff41da94"). InnerVolumeSpecName "kube-api-access-t4sz7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:30:22.523131 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:22.523097 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4sz7\" (UniqueName: \"kubernetes.io/projected/5548598f-3919-417b-ac60-1d74ff41da94-kube-api-access-t4sz7\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:22.649069 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:22.649026 5222 scope.go:122] "RemoveContainer" containerID="55eff768534f35160edc1c1bbd8116929e3501bfa89bf22b0dd0b337b87e25ce" Oct 31 00:30:23.137177 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:23.137140 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1e5e75-29890110-zh827" Oct 31 00:30:23.137177 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:23.137158 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1e5e75-29890110-zh827" event={"ID":"5548598f-3919-417b-ac60-1d74ff41da94","Type":"ContainerDied","Data":"ced7174181fe75b113ead98cb79bf82b50e9f3992a4a66266b693eff9d0c6c5e"} Oct 31 00:30:23.137703 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:23.137223 5222 scope.go:122] "RemoveContainer" containerID="6236e97cb30e473603a97e449be2a4973924331d8bf237ad99f813418e437e7f" Oct 31 00:30:23.139533 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:23.139500 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-412743-29890110-4245m" event={"ID":"6dbebe96-28d7-4e27-bcb7-9291a84f6c71","Type":"ContainerStarted","Data":"ab08baf3c970299d95e306d3fdafb7c08c64b30ae6408f8369047a84a292509b"} Oct 31 00:30:23.286773 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:23.286734 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da9d07-29890110-r6vzq" Oct 31 00:30:23.331692 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:23.331662 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d56a9943-4360-48d4-a701-1bd9b4206908-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d56a9943-4360-48d4-a701-1bd9b4206908-maas-api-service-ca\") pod \"d56a9943-4360-48d4-a701-1bd9b4206908\" (UID: \"d56a9943-4360-48d4-a701-1bd9b4206908\") " Oct 31 00:30:23.331874 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:23.331797 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d56a9943-4360-48d4-a701-1bd9b4206908-kube-api-access-4rl6j\" (UniqueName: \"kubernetes.io/projected/d56a9943-4360-48d4-a701-1bd9b4206908-kube-api-access-4rl6j\") pod \"d56a9943-4360-48d4-a701-1bd9b4206908\" (UID: \"d56a9943-4360-48d4-a701-1bd9b4206908\") " Oct 31 00:30:23.332104 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:23.332077 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d56a9943-4360-48d4-a701-1bd9b4206908-maas-api-service-ca" pod "d56a9943-4360-48d4-a701-1bd9b4206908" (UID: "d56a9943-4360-48d4-a701-1bd9b4206908"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:30:23.334019 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:23.333994 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d56a9943-4360-48d4-a701-1bd9b4206908-kube-api-access-4rl6j" pod "d56a9943-4360-48d4-a701-1bd9b4206908" (UID: "d56a9943-4360-48d4-a701-1bd9b4206908"). InnerVolumeSpecName "kube-api-access-4rl6j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:30:23.423082 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:23.423048 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1e5e75-29890110-zh827"] Oct 31 00:30:23.428412 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:23.428363 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1e5e75-29890110-zh827"] Oct 31 00:30:23.432842 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:23.432816 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rl6j\" (UniqueName: \"kubernetes.io/projected/d56a9943-4360-48d4-a701-1bd9b4206908-kube-api-access-4rl6j\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:23.432842 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:23.432843 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d56a9943-4360-48d4-a701-1bd9b4206908-maas-api-service-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:23.649424 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:23.649311 5222 scope.go:122] "RemoveContainer" containerID="713135eba189a105631a14586d69e0ee661625ec9d0aa272971946c53e21cf80" Oct 31 00:30:23.656358 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:23.656326 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5548598f-3919-417b-ac60-1d74ff41da94" path="/var/lib/kubelet/pods/5548598f-3919-417b-ac60-1d74ff41da94/volumes" Oct 31 00:30:24.149944 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:24.149911 5222 generic.go:356] "Generic (PLEG): container finished" podID="6dbebe96-28d7-4e27-bcb7-9291a84f6c71" containerID="ab08baf3c970299d95e306d3fdafb7c08c64b30ae6408f8369047a84a292509b" exitCode=60 Oct 31 00:30:24.150424 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:24.149985 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-412743-29890110-4245m" event={"ID":"6dbebe96-28d7-4e27-bcb7-9291a84f6c71","Type":"ContainerDied","Data":"ab08baf3c970299d95e306d3fdafb7c08c64b30ae6408f8369047a84a292509b"} Oct 31 00:30:24.150424 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:24.150043 5222 scope.go:122] "RemoveContainer" containerID="55eff768534f35160edc1c1bbd8116929e3501bfa89bf22b0dd0b337b87e25ce" Oct 31 00:30:24.150549 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:24.150436 5222 scope.go:122] "RemoveContainer" containerID="ab08baf3c970299d95e306d3fdafb7c08c64b30ae6408f8369047a84a292509b" Oct 31 00:30:24.150786 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:30:24.150757 5222 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-412743-29890110-4245m_odh-ai-gateway-infra(6dbebe96-28d7-4e27-bcb7-9291a84f6c71)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-412743-29890110-4245m" podUID="6dbebe96-28d7-4e27-bcb7-9291a84f6c71" Oct 31 00:30:24.152572 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:24.152544 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e8ba06-29890110-gcpws" event={"ID":"156e0827-de4c-4a74-b6f0-9c2cca06ba02","Type":"ContainerStarted","Data":"bab5a4d8e7aa4c44d90b647cd1d3af1c03848673bf239195d0918be5b44539b7"} Oct 31 00:30:24.153917 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:24.153896 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da9d07-29890110-r6vzq" event={"ID":"d56a9943-4360-48d4-a701-1bd9b4206908","Type":"ContainerDied","Data":"d19d987f421d37b23279041a177336278d28fcd8392beda5651474718d425255"} Oct 31 00:30:24.153917 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:24.153909 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da9d07-29890110-r6vzq" Oct 31 00:30:24.162200 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:24.162179 5222 scope.go:122] "RemoveContainer" containerID="b388581047012a3e31617a086f2821d488867ee388f75abf6bbe3184472297b1" Oct 31 00:30:24.500216 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:24.500177 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da9d07-29890110-r6vzq"] Oct 31 00:30:24.504120 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:24.504090 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da9d07-29890110-r6vzq"] Oct 31 00:30:24.553523 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:24.553486 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-412743-29890110-4245m"] Oct 31 00:30:24.649288 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:24.649252 5222 scope.go:122] "RemoveContainer" containerID="a01166a19e01bc6f2adad774411a394a45e9e6fbb21d005383a040e3c27d9723" Oct 31 00:30:25.161985 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:25.161887 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-101b9e-29890110-7zx97" event={"ID":"f0d7eece-c1b5-4f51-9ed7-76d3d0564873","Type":"ContainerStarted","Data":"05a264da0e46bfa0912e09b45ce4766ea18109679a980c71f23b8291b0680adf"} Oct 31 00:30:25.163697 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:25.163669 5222 generic.go:356] "Generic (PLEG): container finished" podID="156e0827-de4c-4a74-b6f0-9c2cca06ba02" containerID="bab5a4d8e7aa4c44d90b647cd1d3af1c03848673bf239195d0918be5b44539b7" exitCode=60 Oct 31 00:30:25.163840 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:25.163744 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e8ba06-29890110-gcpws" event={"ID":"156e0827-de4c-4a74-b6f0-9c2cca06ba02","Type":"ContainerDied","Data":"bab5a4d8e7aa4c44d90b647cd1d3af1c03848673bf239195d0918be5b44539b7"} Oct 31 00:30:25.163840 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:25.163794 5222 scope.go:122] "RemoveContainer" containerID="713135eba189a105631a14586d69e0ee661625ec9d0aa272971946c53e21cf80" Oct 31 00:30:25.164152 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:25.164095 5222 scope.go:122] "RemoveContainer" containerID="bab5a4d8e7aa4c44d90b647cd1d3af1c03848673bf239195d0918be5b44539b7" Oct 31 00:30:25.164432 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:30:25.164402 5222 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-e8ba06-29890110-gcpws_odh-ai-gateway-infra(156e0827-de4c-4a74-b6f0-9c2cca06ba02)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e8ba06-29890110-gcpws" podUID="156e0827-de4c-4a74-b6f0-9c2cca06ba02" Oct 31 00:30:25.320604 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:25.320573 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-412743-29890110-4245m" Oct 31 00:30:25.395509 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:25.395467 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e8ba06-29890110-gcpws"] Oct 31 00:30:25.455846 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:25.455803 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6dbebe96-28d7-4e27-bcb7-9291a84f6c71-kube-api-access-jwr8b\" (UniqueName: \"kubernetes.io/projected/6dbebe96-28d7-4e27-bcb7-9291a84f6c71-kube-api-access-jwr8b\") pod \"6dbebe96-28d7-4e27-bcb7-9291a84f6c71\" (UID: \"6dbebe96-28d7-4e27-bcb7-9291a84f6c71\") " Oct 31 00:30:25.456088 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:25.456064 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6dbebe96-28d7-4e27-bcb7-9291a84f6c71-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6dbebe96-28d7-4e27-bcb7-9291a84f6c71-maas-api-service-ca\") pod \"6dbebe96-28d7-4e27-bcb7-9291a84f6c71\" (UID: \"6dbebe96-28d7-4e27-bcb7-9291a84f6c71\") " Oct 31 00:30:25.456436 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:25.456407 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6dbebe96-28d7-4e27-bcb7-9291a84f6c71-maas-api-service-ca" pod "6dbebe96-28d7-4e27-bcb7-9291a84f6c71" (UID: "6dbebe96-28d7-4e27-bcb7-9291a84f6c71"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:30:25.458119 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:25.458098 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dbebe96-28d7-4e27-bcb7-9291a84f6c71-kube-api-access-jwr8b" pod "6dbebe96-28d7-4e27-bcb7-9291a84f6c71" (UID: "6dbebe96-28d7-4e27-bcb7-9291a84f6c71"). InnerVolumeSpecName "kube-api-access-jwr8b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:30:25.557082 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:25.557051 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6dbebe96-28d7-4e27-bcb7-9291a84f6c71-maas-api-service-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:25.557082 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:25.557078 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwr8b\" (UniqueName: \"kubernetes.io/projected/6dbebe96-28d7-4e27-bcb7-9291a84f6c71-kube-api-access-jwr8b\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:25.664820 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:25.664788 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d56a9943-4360-48d4-a701-1bd9b4206908" path="/var/lib/kubelet/pods/d56a9943-4360-48d4-a701-1bd9b4206908/volumes" Oct 31 00:30:26.173441 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:26.173409 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-412743-29890110-4245m" Oct 31 00:30:26.173937 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:26.173408 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-412743-29890110-4245m" event={"ID":"6dbebe96-28d7-4e27-bcb7-9291a84f6c71","Type":"ContainerDied","Data":"e4756721b9f5e888574e6fe7647fc139d19954f9984512f1232b4b7826fbbc99"} Oct 31 00:30:26.173937 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:26.173565 5222 scope.go:122] "RemoveContainer" containerID="ab08baf3c970299d95e306d3fdafb7c08c64b30ae6408f8369047a84a292509b" Oct 31 00:30:26.175799 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:26.175764 5222 generic.go:356] "Generic (PLEG): container finished" podID="f0d7eece-c1b5-4f51-9ed7-76d3d0564873" containerID="05a264da0e46bfa0912e09b45ce4766ea18109679a980c71f23b8291b0680adf" exitCode=60 Oct 31 00:30:26.175940 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:26.175906 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-101b9e-29890110-7zx97" event={"ID":"f0d7eece-c1b5-4f51-9ed7-76d3d0564873","Type":"ContainerDied","Data":"05a264da0e46bfa0912e09b45ce4766ea18109679a980c71f23b8291b0680adf"} Oct 31 00:30:26.176333 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:26.176314 5222 scope.go:122] "RemoveContainer" containerID="05a264da0e46bfa0912e09b45ce4766ea18109679a980c71f23b8291b0680adf" Oct 31 00:30:26.176657 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:30:26.176628 5222 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-101b9e-29890110-7zx97_odh-ai-gateway-infra(f0d7eece-c1b5-4f51-9ed7-76d3d0564873)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-101b9e-29890110-7zx97" podUID="f0d7eece-c1b5-4f51-9ed7-76d3d0564873" Oct 31 00:30:26.185334 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:26.185313 5222 scope.go:122] "RemoveContainer" containerID="a01166a19e01bc6f2adad774411a394a45e9e6fbb21d005383a040e3c27d9723" Oct 31 00:30:26.294361 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:26.294328 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-101b9e-29890110-7zx97"] Oct 31 00:30:26.338784 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:26.338750 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e8ba06-29890110-gcpws" Oct 31 00:30:26.341171 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:26.341144 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-412743-29890110-4245m"] Oct 31 00:30:26.345840 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:26.345815 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-412743-29890110-4245m"] Oct 31 00:30:26.470715 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:26.470619 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/156e0827-de4c-4a74-b6f0-9c2cca06ba02-kube-api-access-p6j6p\" (UniqueName: \"kubernetes.io/projected/156e0827-de4c-4a74-b6f0-9c2cca06ba02-kube-api-access-p6j6p\") pod \"156e0827-de4c-4a74-b6f0-9c2cca06ba02\" (UID: \"156e0827-de4c-4a74-b6f0-9c2cca06ba02\") " Oct 31 00:30:26.470884 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:26.470792 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/156e0827-de4c-4a74-b6f0-9c2cca06ba02-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/156e0827-de4c-4a74-b6f0-9c2cca06ba02-maas-api-service-ca\") pod \"156e0827-de4c-4a74-b6f0-9c2cca06ba02\" (UID: \"156e0827-de4c-4a74-b6f0-9c2cca06ba02\") " Oct 31 00:30:26.471186 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:26.471159 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/156e0827-de4c-4a74-b6f0-9c2cca06ba02-maas-api-service-ca" pod "156e0827-de4c-4a74-b6f0-9c2cca06ba02" (UID: "156e0827-de4c-4a74-b6f0-9c2cca06ba02"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:30:26.472951 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:26.472920 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/156e0827-de4c-4a74-b6f0-9c2cca06ba02-kube-api-access-p6j6p" pod "156e0827-de4c-4a74-b6f0-9c2cca06ba02" (UID: "156e0827-de4c-4a74-b6f0-9c2cca06ba02"). InnerVolumeSpecName "kube-api-access-p6j6p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:30:26.571900 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:26.571858 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/156e0827-de4c-4a74-b6f0-9c2cca06ba02-maas-api-service-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:26.571900 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:26.571892 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6j6p\" (UniqueName: \"kubernetes.io/projected/156e0827-de4c-4a74-b6f0-9c2cca06ba02-kube-api-access-p6j6p\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:27.187237 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:27.187213 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e8ba06-29890110-gcpws" event={"ID":"156e0827-de4c-4a74-b6f0-9c2cca06ba02","Type":"ContainerDied","Data":"0d7b0e38255bf1243501455c74272d21abf7fb43330f0c32fea8b10b8126b398"} Oct 31 00:30:27.187237 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:27.187233 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e8ba06-29890110-gcpws" Oct 31 00:30:27.187658 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:27.187270 5222 scope.go:122] "RemoveContainer" containerID="bab5a4d8e7aa4c44d90b647cd1d3af1c03848673bf239195d0918be5b44539b7" Oct 31 00:30:27.338137 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:27.338111 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-101b9e-29890110-7zx97" Oct 31 00:30:27.419502 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:27.419466 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e8ba06-29890110-gcpws"] Oct 31 00:30:27.422896 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:27.422872 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e8ba06-29890110-gcpws"] Oct 31 00:30:27.481430 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:27.481306 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0d7eece-c1b5-4f51-9ed7-76d3d0564873-kube-api-access-nnt24\" (UniqueName: \"kubernetes.io/projected/f0d7eece-c1b5-4f51-9ed7-76d3d0564873-kube-api-access-nnt24\") pod \"f0d7eece-c1b5-4f51-9ed7-76d3d0564873\" (UID: \"f0d7eece-c1b5-4f51-9ed7-76d3d0564873\") " Oct 31 00:30:27.481430 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:27.481346 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0d7eece-c1b5-4f51-9ed7-76d3d0564873-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0d7eece-c1b5-4f51-9ed7-76d3d0564873-maas-api-service-ca\") pod \"f0d7eece-c1b5-4f51-9ed7-76d3d0564873\" (UID: \"f0d7eece-c1b5-4f51-9ed7-76d3d0564873\") " Oct 31 00:30:27.481773 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:27.481745 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0d7eece-c1b5-4f51-9ed7-76d3d0564873-maas-api-service-ca" pod "f0d7eece-c1b5-4f51-9ed7-76d3d0564873" (UID: "f0d7eece-c1b5-4f51-9ed7-76d3d0564873"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:30:27.484484 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:27.484454 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0d7eece-c1b5-4f51-9ed7-76d3d0564873-kube-api-access-nnt24" pod "f0d7eece-c1b5-4f51-9ed7-76d3d0564873" (UID: "f0d7eece-c1b5-4f51-9ed7-76d3d0564873"). InnerVolumeSpecName "kube-api-access-nnt24". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:30:27.582927 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:27.582887 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnt24\" (UniqueName: \"kubernetes.io/projected/f0d7eece-c1b5-4f51-9ed7-76d3d0564873-kube-api-access-nnt24\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:27.582927 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:27.582928 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0d7eece-c1b5-4f51-9ed7-76d3d0564873-maas-api-service-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:27.655674 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:27.655639 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="156e0827-de4c-4a74-b6f0-9c2cca06ba02" path="/var/lib/kubelet/pods/156e0827-de4c-4a74-b6f0-9c2cca06ba02/volumes" Oct 31 00:30:27.656062 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:27.656048 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6dbebe96-28d7-4e27-bcb7-9291a84f6c71" path="/var/lib/kubelet/pods/6dbebe96-28d7-4e27-bcb7-9291a84f6c71/volumes" Oct 31 00:30:28.195604 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:28.195555 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-101b9e-29890110-7zx97" event={"ID":"f0d7eece-c1b5-4f51-9ed7-76d3d0564873","Type":"ContainerDied","Data":"09381f848c736572c2771540d2959ef929a3fefd3f17629f59344b60075a9074"} Oct 31 00:30:28.196148 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:28.195619 5222 scope.go:122] "RemoveContainer" containerID="05a264da0e46bfa0912e09b45ce4766ea18109679a980c71f23b8291b0680adf" Oct 31 00:30:28.196148 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:28.195572 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-101b9e-29890110-7zx97" Oct 31 00:30:28.365091 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:28.365056 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-101b9e-29890110-7zx97"] Oct 31 00:30:28.415784 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:28.415747 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-101b9e-29890110-7zx97"] Oct 31 00:30:29.655832 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:29.655799 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0d7eece-c1b5-4f51-9ed7-76d3d0564873" path="/var/lib/kubelet/pods/f0d7eece-c1b5-4f51-9ed7-76d3d0564873/volumes" Oct 31 00:30:31.390229 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:31.390196 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6b8b0b9-ddc5k"] Oct 31 00:30:31.390668 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:31.390655 5222 cpu_manager.go:414] "containerMap: removing container" podUID="f0d7eece-c1b5-4f51-9ed7-76d3d0564873" containerName="cleanup" Oct 31 00:30:31.390730 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:31.390668 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0d7eece-c1b5-4f51-9ed7-76d3d0564873" containerName="cleanup" Oct 31 00:30:31.390730 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:31.390682 5222 cpu_manager.go:414] "containerMap: removing container" podUID="6dbebe96-28d7-4e27-bcb7-9291a84f6c71" containerName="cleanup" Oct 31 00:30:31.390730 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:31.390687 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dbebe96-28d7-4e27-bcb7-9291a84f6c71" containerName="cleanup" Oct 31 00:30:31.390730 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:31.390700 5222 cpu_manager.go:414] "containerMap: removing container" podUID="156e0827-de4c-4a74-b6f0-9c2cca06ba02" containerName="cleanup" Oct 31 00:30:31.390730 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:31.390705 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="156e0827-de4c-4a74-b6f0-9c2cca06ba02" containerName="cleanup" Oct 31 00:30:31.390730 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:31.390712 5222 cpu_manager.go:414] "containerMap: removing container" podUID="d56a9943-4360-48d4-a701-1bd9b4206908" containerName="cleanup" Oct 31 00:30:31.390730 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:31.390716 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d56a9943-4360-48d4-a701-1bd9b4206908" containerName="cleanup" Oct 31 00:30:31.390730 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:31.390722 5222 cpu_manager.go:414] "containerMap: removing container" podUID="5548598f-3919-417b-ac60-1d74ff41da94" containerName="cleanup" Oct 31 00:30:31.390730 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:31.390726 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5548598f-3919-417b-ac60-1d74ff41da94" containerName="cleanup" Oct 31 00:30:31.390982 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:31.390740 5222 cpu_manager.go:414] "containerMap: removing container" podUID="d56a9943-4360-48d4-a701-1bd9b4206908" containerName="cleanup" Oct 31 00:30:31.390982 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:31.390744 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d56a9943-4360-48d4-a701-1bd9b4206908" containerName="cleanup" Oct 31 00:30:31.390982 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:31.390751 5222 cpu_manager.go:414] "containerMap: removing container" podUID="156e0827-de4c-4a74-b6f0-9c2cca06ba02" containerName="cleanup" Oct 31 00:30:31.390982 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:31.390755 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="156e0827-de4c-4a74-b6f0-9c2cca06ba02" containerName="cleanup" Oct 31 00:30:31.390982 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:31.390762 5222 cpu_manager.go:414] "containerMap: removing container" podUID="6dbebe96-28d7-4e27-bcb7-9291a84f6c71" containerName="cleanup" Oct 31 00:30:31.390982 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:31.390766 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dbebe96-28d7-4e27-bcb7-9291a84f6c71" containerName="cleanup" Oct 31 00:30:31.390982 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:31.390780 5222 cpu_manager.go:414] "containerMap: removing container" podUID="156e0827-de4c-4a74-b6f0-9c2cca06ba02" containerName="cleanup" Oct 31 00:30:31.390982 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:31.390786 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="156e0827-de4c-4a74-b6f0-9c2cca06ba02" containerName="cleanup" Oct 31 00:30:31.390982 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:31.390792 5222 cpu_manager.go:414] "containerMap: removing container" podUID="d56a9943-4360-48d4-a701-1bd9b4206908" containerName="cleanup" Oct 31 00:30:31.390982 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:31.390796 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d56a9943-4360-48d4-a701-1bd9b4206908" containerName="cleanup" Oct 31 00:30:31.390982 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:31.390805 5222 cpu_manager.go:414] "containerMap: removing container" podUID="f0d7eece-c1b5-4f51-9ed7-76d3d0564873" containerName="cleanup" Oct 31 00:30:31.390982 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:31.390809 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0d7eece-c1b5-4f51-9ed7-76d3d0564873" containerName="cleanup" Oct 31 00:30:31.390982 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:31.390818 5222 cpu_manager.go:414] "containerMap: removing container" podUID="5548598f-3919-417b-ac60-1d74ff41da94" containerName="cleanup" Oct 31 00:30:31.390982 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:31.390822 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5548598f-3919-417b-ac60-1d74ff41da94" containerName="cleanup" Oct 31 00:30:31.390982 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:31.390882 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="d56a9943-4360-48d4-a701-1bd9b4206908" containerName="cleanup" Oct 31 00:30:31.390982 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:31.390888 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0d7eece-c1b5-4f51-9ed7-76d3d0564873" containerName="cleanup" Oct 31 00:30:31.390982 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:31.390894 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="6dbebe96-28d7-4e27-bcb7-9291a84f6c71" containerName="cleanup" Oct 31 00:30:31.390982 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:31.390900 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="156e0827-de4c-4a74-b6f0-9c2cca06ba02" containerName="cleanup" Oct 31 00:30:31.390982 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:31.390905 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="6dbebe96-28d7-4e27-bcb7-9291a84f6c71" containerName="cleanup" Oct 31 00:30:31.390982 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:31.390911 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="5548598f-3919-417b-ac60-1d74ff41da94" containerName="cleanup" Oct 31 00:30:31.390982 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:31.390916 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="5548598f-3919-417b-ac60-1d74ff41da94" containerName="cleanup" Oct 31 00:30:31.390982 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:31.390920 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0d7eece-c1b5-4f51-9ed7-76d3d0564873" containerName="cleanup" Oct 31 00:30:31.390982 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:31.390926 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="156e0827-de4c-4a74-b6f0-9c2cca06ba02" containerName="cleanup" Oct 31 00:30:31.390982 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:31.390932 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="d56a9943-4360-48d4-a701-1bd9b4206908" containerName="cleanup" Oct 31 00:30:31.390982 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:31.390938 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="156e0827-de4c-4a74-b6f0-9c2cca06ba02" containerName="cleanup" Oct 31 00:30:31.395727 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:31.395705 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6b8b0b9-ddc5k" Oct 31 00:30:31.448187 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:31.448153 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6b8b0b9-ddc5k"] Oct 31 00:30:31.497281 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:31.497246 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:30:31.497281 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:31.497246 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:30:31.497281 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:31.497278 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfz9z\"" Oct 31 00:30:31.497549 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:31.497246 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:30:31.526505 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:31.526473 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c47303f2-efc8-4a4e-bffd-fb123dfc9707-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6b8b0b9-ddc5k\" (UID: \"c47303f2-efc8-4a4e-bffd-fb123dfc9707\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6b8b0b9-ddc5k" Oct 31 00:30:31.526671 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:31.526591 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rr5rx\" (UniqueName: \"kubernetes.io/projected/c47303f2-efc8-4a4e-bffd-fb123dfc9707-kube-api-access-rr5rx\") pod \"maas-api-revoke-sub-d6b8b0b9-ddc5k\" (UID: \"c47303f2-efc8-4a4e-bffd-fb123dfc9707\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6b8b0b9-ddc5k" Oct 31 00:30:31.526671 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:31.526614 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c47303f2-efc8-4a4e-bffd-fb123dfc9707-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6b8b0b9-ddc5k\" (UID: \"c47303f2-efc8-4a4e-bffd-fb123dfc9707\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6b8b0b9-ddc5k" Oct 31 00:30:31.628212 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:31.628174 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rr5rx\" (UniqueName: \"kubernetes.io/projected/c47303f2-efc8-4a4e-bffd-fb123dfc9707-kube-api-access-rr5rx\") pod \"maas-api-revoke-sub-d6b8b0b9-ddc5k\" (UID: \"c47303f2-efc8-4a4e-bffd-fb123dfc9707\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6b8b0b9-ddc5k" Oct 31 00:30:31.628212 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:31.628218 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c47303f2-efc8-4a4e-bffd-fb123dfc9707-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6b8b0b9-ddc5k\" (UID: \"c47303f2-efc8-4a4e-bffd-fb123dfc9707\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6b8b0b9-ddc5k" Oct 31 00:30:31.628457 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:31.628292 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c47303f2-efc8-4a4e-bffd-fb123dfc9707-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6b8b0b9-ddc5k\" (UID: \"c47303f2-efc8-4a4e-bffd-fb123dfc9707\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6b8b0b9-ddc5k" Oct 31 00:30:31.629017 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:31.628991 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c47303f2-efc8-4a4e-bffd-fb123dfc9707-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6b8b0b9-ddc5k\" (UID: \"c47303f2-efc8-4a4e-bffd-fb123dfc9707\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6b8b0b9-ddc5k" Oct 31 00:30:31.630746 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:31.630726 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c47303f2-efc8-4a4e-bffd-fb123dfc9707-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6b8b0b9-ddc5k\" (UID: \"c47303f2-efc8-4a4e-bffd-fb123dfc9707\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6b8b0b9-ddc5k" Oct 31 00:30:31.691105 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:31.691030 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rr5rx\" (UniqueName: \"kubernetes.io/projected/c47303f2-efc8-4a4e-bffd-fb123dfc9707-kube-api-access-rr5rx\") pod \"maas-api-revoke-sub-d6b8b0b9-ddc5k\" (UID: \"c47303f2-efc8-4a4e-bffd-fb123dfc9707\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6b8b0b9-ddc5k" Oct 31 00:30:31.710976 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:31.710145 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6b8b0b9-ddc5k" Oct 31 00:30:32.103570 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:30:32.103532 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc47303f2_efc8_4a4e_bffd_fb123dfc9707.slice/crio-3010b2b82dcccda31391ece74ccd3165b847e2f0ae37fc7ebab5eee3a6cb8eba WatchSource:0}: Error finding container 3010b2b82dcccda31391ece74ccd3165b847e2f0ae37fc7ebab5eee3a6cb8eba: Status 404 returned error can't find the container with id 3010b2b82dcccda31391ece74ccd3165b847e2f0ae37fc7ebab5eee3a6cb8eba Oct 31 00:30:32.105392 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:32.105351 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6b8b0b9-ddc5k"] Oct 31 00:30:32.220987 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:32.220954 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6b8b0b9-ddc5k" event={"ID":"c47303f2-efc8-4a4e-bffd-fb123dfc9707","Type":"ContainerStarted","Data":"3010b2b82dcccda31391ece74ccd3165b847e2f0ae37fc7ebab5eee3a6cb8eba"} Oct 31 00:30:33.228801 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:33.228708 5222 generic.go:356] "Generic (PLEG): container finished" podID="c47303f2-efc8-4a4e-bffd-fb123dfc9707" containerID="eedbfc09916da2e38df1cf93b0fca2c7ac63cdb446011c9fc7b6c9be82702f10" exitCode=0 Oct 31 00:30:33.229227 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:33.228802 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6b8b0b9-ddc5k" event={"ID":"c47303f2-efc8-4a4e-bffd-fb123dfc9707","Type":"ContainerDied","Data":"eedbfc09916da2e38df1cf93b0fca2c7ac63cdb446011c9fc7b6c9be82702f10"} Oct 31 00:30:34.380141 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:34.380114 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6b8b0b9-ddc5k" Oct 31 00:30:34.463218 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:34.463181 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c47303f2-efc8-4a4e-bffd-fb123dfc9707-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c47303f2-efc8-4a4e-bffd-fb123dfc9707-maas-api-openshift-ca\") pod \"c47303f2-efc8-4a4e-bffd-fb123dfc9707\" (UID: \"c47303f2-efc8-4a4e-bffd-fb123dfc9707\") " Oct 31 00:30:34.463434 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:34.463257 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c47303f2-efc8-4a4e-bffd-fb123dfc9707-kube-api-access-rr5rx\" (UniqueName: \"kubernetes.io/projected/c47303f2-efc8-4a4e-bffd-fb123dfc9707-kube-api-access-rr5rx\") pod \"c47303f2-efc8-4a4e-bffd-fb123dfc9707\" (UID: \"c47303f2-efc8-4a4e-bffd-fb123dfc9707\") " Oct 31 00:30:34.463434 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:34.463351 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c47303f2-efc8-4a4e-bffd-fb123dfc9707-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c47303f2-efc8-4a4e-bffd-fb123dfc9707-maas-api-xks-ca\") pod \"c47303f2-efc8-4a4e-bffd-fb123dfc9707\" (UID: \"c47303f2-efc8-4a4e-bffd-fb123dfc9707\") " Oct 31 00:30:34.463575 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:34.463548 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c47303f2-efc8-4a4e-bffd-fb123dfc9707-maas-api-openshift-ca" pod "c47303f2-efc8-4a4e-bffd-fb123dfc9707" (UID: "c47303f2-efc8-4a4e-bffd-fb123dfc9707"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:30:34.463782 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:34.463753 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c47303f2-efc8-4a4e-bffd-fb123dfc9707-maas-api-openshift-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:34.465550 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:34.465526 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c47303f2-efc8-4a4e-bffd-fb123dfc9707-kube-api-access-rr5rx" pod "c47303f2-efc8-4a4e-bffd-fb123dfc9707" (UID: "c47303f2-efc8-4a4e-bffd-fb123dfc9707"). InnerVolumeSpecName "kube-api-access-rr5rx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:30:34.465639 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:34.465563 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c47303f2-efc8-4a4e-bffd-fb123dfc9707-maas-api-xks-ca" pod "c47303f2-efc8-4a4e-bffd-fb123dfc9707" (UID: "c47303f2-efc8-4a4e-bffd-fb123dfc9707"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:30:34.564960 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:34.564911 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rr5rx\" (UniqueName: \"kubernetes.io/projected/c47303f2-efc8-4a4e-bffd-fb123dfc9707-kube-api-access-rr5rx\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:34.564960 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:34.564957 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c47303f2-efc8-4a4e-bffd-fb123dfc9707-maas-api-xks-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:35.241012 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:35.240977 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6b8b0b9-ddc5k" Oct 31 00:30:35.241188 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:35.241011 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6b8b0b9-ddc5k" event={"ID":"c47303f2-efc8-4a4e-bffd-fb123dfc9707","Type":"ContainerDied","Data":"3010b2b82dcccda31391ece74ccd3165b847e2f0ae37fc7ebab5eee3a6cb8eba"} Oct 31 00:30:35.241188 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:35.241058 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3010b2b82dcccda31391ece74ccd3165b847e2f0ae37fc7ebab5eee3a6cb8eba" Oct 31 00:30:35.890052 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:30:35.890020 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 00:30:38.610916 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:38.610872 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-101b9e/premium-w0-101b9e-kserve-dc65f694b-m4tsk"] Oct 31 00:30:38.611611 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:38.611594 5222 cpu_manager.go:414] "containerMap: removing container" podUID="5548598f-3919-417b-ac60-1d74ff41da94" containerName="cleanup" Oct 31 00:30:38.611676 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:38.611612 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5548598f-3919-417b-ac60-1d74ff41da94" containerName="cleanup" Oct 31 00:30:38.611676 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:38.611633 5222 cpu_manager.go:414] "containerMap: removing container" podUID="c47303f2-efc8-4a4e-bffd-fb123dfc9707" containerName="revoke-keys" Oct 31 00:30:38.611676 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:38.611638 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c47303f2-efc8-4a4e-bffd-fb123dfc9707" containerName="revoke-keys" Oct 31 00:30:38.611676 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:38.611647 5222 cpu_manager.go:414] "containerMap: removing container" podUID="6dbebe96-28d7-4e27-bcb7-9291a84f6c71" containerName="cleanup" Oct 31 00:30:38.611676 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:38.611653 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dbebe96-28d7-4e27-bcb7-9291a84f6c71" containerName="cleanup" Oct 31 00:30:38.611676 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:38.611671 5222 cpu_manager.go:414] "containerMap: removing container" podUID="f0d7eece-c1b5-4f51-9ed7-76d3d0564873" containerName="cleanup" Oct 31 00:30:38.611676 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:38.611675 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0d7eece-c1b5-4f51-9ed7-76d3d0564873" containerName="cleanup" Oct 31 00:30:38.611880 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:38.611750 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="d56a9943-4360-48d4-a701-1bd9b4206908" containerName="cleanup" Oct 31 00:30:38.611880 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:38.611758 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="c47303f2-efc8-4a4e-bffd-fb123dfc9707" containerName="revoke-keys" Oct 31 00:30:38.611880 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:38.611765 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0d7eece-c1b5-4f51-9ed7-76d3d0564873" containerName="cleanup" Oct 31 00:30:38.611880 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:38.611774 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="6dbebe96-28d7-4e27-bcb7-9291a84f6c71" containerName="cleanup" Oct 31 00:30:38.611880 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:38.611780 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="5548598f-3919-417b-ac60-1d74ff41da94" containerName="cleanup" Oct 31 00:30:38.615308 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:38.615289 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-101b9e/premium-w0-101b9e-kserve-dc65f694b-m4tsk" Oct 31 00:30:38.668110 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:38.668067 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-101b9e/premium-w0-101b9e-kserve-dc65f694b-m4tsk"] Oct 31 00:30:38.713897 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:38.713858 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbf98c81-6f05-41b8-ba3b-c10ecbbff5d8-home\") pod \"premium-w0-101b9e-kserve-dc65f694b-m4tsk\" (UID: \"fbf98c81-6f05-41b8-ba3b-c10ecbbff5d8\") " pod="e2e-models-e2e-worker-w0-101b9e/premium-w0-101b9e-kserve-dc65f694b-m4tsk" Oct 31 00:30:38.714046 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:38.713908 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbf98c81-6f05-41b8-ba3b-c10ecbbff5d8-model-cache\") pod \"premium-w0-101b9e-kserve-dc65f694b-m4tsk\" (UID: \"fbf98c81-6f05-41b8-ba3b-c10ecbbff5d8\") " pod="e2e-models-e2e-worker-w0-101b9e/premium-w0-101b9e-kserve-dc65f694b-m4tsk" Oct 31 00:30:38.714046 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:38.714011 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbf98c81-6f05-41b8-ba3b-c10ecbbff5d8-dshm\") pod \"premium-w0-101b9e-kserve-dc65f694b-m4tsk\" (UID: \"fbf98c81-6f05-41b8-ba3b-c10ecbbff5d8\") " pod="e2e-models-e2e-worker-w0-101b9e/premium-w0-101b9e-kserve-dc65f694b-m4tsk" Oct 31 00:30:38.714148 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:38.714049 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-459r7\" (UniqueName: \"kubernetes.io/projected/fbf98c81-6f05-41b8-ba3b-c10ecbbff5d8-kube-api-access-459r7\") pod \"premium-w0-101b9e-kserve-dc65f694b-m4tsk\" (UID: \"fbf98c81-6f05-41b8-ba3b-c10ecbbff5d8\") " pod="e2e-models-e2e-worker-w0-101b9e/premium-w0-101b9e-kserve-dc65f694b-m4tsk" Oct 31 00:30:38.714148 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:38.714074 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbf98c81-6f05-41b8-ba3b-c10ecbbff5d8-tls-certs\") pod \"premium-w0-101b9e-kserve-dc65f694b-m4tsk\" (UID: \"fbf98c81-6f05-41b8-ba3b-c10ecbbff5d8\") " pod="e2e-models-e2e-worker-w0-101b9e/premium-w0-101b9e-kserve-dc65f694b-m4tsk" Oct 31 00:30:38.714222 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:38.714166 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbf98c81-6f05-41b8-ba3b-c10ecbbff5d8-tmp-dir\") pod \"premium-w0-101b9e-kserve-dc65f694b-m4tsk\" (UID: \"fbf98c81-6f05-41b8-ba3b-c10ecbbff5d8\") " pod="e2e-models-e2e-worker-w0-101b9e/premium-w0-101b9e-kserve-dc65f694b-m4tsk" Oct 31 00:30:38.716734 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:38.716716 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-101b9e\"/\"kube-root-ca.crt\"" Oct 31 00:30:38.716823 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:38.716747 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-101b9e\"/\"openshift-service-ca.crt\"" Oct 31 00:30:38.720590 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:38.720568 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-101b9e\"/\"default-dockercfg-c98dn\"" Oct 31 00:30:38.720590 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:38.720579 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-101b9e\"/\"premium-w0-101b9e-kserve-self-signed-certs\"" Oct 31 00:30:38.815760 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:38.815724 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbf98c81-6f05-41b8-ba3b-c10ecbbff5d8-dshm\") pod \"premium-w0-101b9e-kserve-dc65f694b-m4tsk\" (UID: \"fbf98c81-6f05-41b8-ba3b-c10ecbbff5d8\") " pod="e2e-models-e2e-worker-w0-101b9e/premium-w0-101b9e-kserve-dc65f694b-m4tsk" Oct 31 00:30:38.815760 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:38.815764 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-459r7\" (UniqueName: \"kubernetes.io/projected/fbf98c81-6f05-41b8-ba3b-c10ecbbff5d8-kube-api-access-459r7\") pod \"premium-w0-101b9e-kserve-dc65f694b-m4tsk\" (UID: \"fbf98c81-6f05-41b8-ba3b-c10ecbbff5d8\") " pod="e2e-models-e2e-worker-w0-101b9e/premium-w0-101b9e-kserve-dc65f694b-m4tsk" Oct 31 00:30:38.815997 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:38.815794 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbf98c81-6f05-41b8-ba3b-c10ecbbff5d8-tls-certs\") pod \"premium-w0-101b9e-kserve-dc65f694b-m4tsk\" (UID: \"fbf98c81-6f05-41b8-ba3b-c10ecbbff5d8\") " pod="e2e-models-e2e-worker-w0-101b9e/premium-w0-101b9e-kserve-dc65f694b-m4tsk" Oct 31 00:30:38.815997 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:38.815895 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbf98c81-6f05-41b8-ba3b-c10ecbbff5d8-tmp-dir\") pod \"premium-w0-101b9e-kserve-dc65f694b-m4tsk\" (UID: \"fbf98c81-6f05-41b8-ba3b-c10ecbbff5d8\") " pod="e2e-models-e2e-worker-w0-101b9e/premium-w0-101b9e-kserve-dc65f694b-m4tsk" Oct 31 00:30:38.816258 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:38.816222 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbf98c81-6f05-41b8-ba3b-c10ecbbff5d8-home\") pod \"premium-w0-101b9e-kserve-dc65f694b-m4tsk\" (UID: \"fbf98c81-6f05-41b8-ba3b-c10ecbbff5d8\") " pod="e2e-models-e2e-worker-w0-101b9e/premium-w0-101b9e-kserve-dc65f694b-m4tsk" Oct 31 00:30:38.816401 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:38.816276 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbf98c81-6f05-41b8-ba3b-c10ecbbff5d8-tmp-dir\") pod \"premium-w0-101b9e-kserve-dc65f694b-m4tsk\" (UID: \"fbf98c81-6f05-41b8-ba3b-c10ecbbff5d8\") " pod="e2e-models-e2e-worker-w0-101b9e/premium-w0-101b9e-kserve-dc65f694b-m4tsk" Oct 31 00:30:38.816401 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:38.816312 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbf98c81-6f05-41b8-ba3b-c10ecbbff5d8-model-cache\") pod \"premium-w0-101b9e-kserve-dc65f694b-m4tsk\" (UID: \"fbf98c81-6f05-41b8-ba3b-c10ecbbff5d8\") " pod="e2e-models-e2e-worker-w0-101b9e/premium-w0-101b9e-kserve-dc65f694b-m4tsk" Oct 31 00:30:38.816576 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:38.816554 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbf98c81-6f05-41b8-ba3b-c10ecbbff5d8-home\") pod \"premium-w0-101b9e-kserve-dc65f694b-m4tsk\" (UID: \"fbf98c81-6f05-41b8-ba3b-c10ecbbff5d8\") " pod="e2e-models-e2e-worker-w0-101b9e/premium-w0-101b9e-kserve-dc65f694b-m4tsk" Oct 31 00:30:38.816686 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:38.816607 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbf98c81-6f05-41b8-ba3b-c10ecbbff5d8-model-cache\") pod \"premium-w0-101b9e-kserve-dc65f694b-m4tsk\" (UID: \"fbf98c81-6f05-41b8-ba3b-c10ecbbff5d8\") " pod="e2e-models-e2e-worker-w0-101b9e/premium-w0-101b9e-kserve-dc65f694b-m4tsk" Oct 31 00:30:38.818476 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:38.818453 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbf98c81-6f05-41b8-ba3b-c10ecbbff5d8-dshm\") pod \"premium-w0-101b9e-kserve-dc65f694b-m4tsk\" (UID: \"fbf98c81-6f05-41b8-ba3b-c10ecbbff5d8\") " pod="e2e-models-e2e-worker-w0-101b9e/premium-w0-101b9e-kserve-dc65f694b-m4tsk" Oct 31 00:30:38.818732 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:38.818713 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbf98c81-6f05-41b8-ba3b-c10ecbbff5d8-tls-certs\") pod \"premium-w0-101b9e-kserve-dc65f694b-m4tsk\" (UID: \"fbf98c81-6f05-41b8-ba3b-c10ecbbff5d8\") " pod="e2e-models-e2e-worker-w0-101b9e/premium-w0-101b9e-kserve-dc65f694b-m4tsk" Oct 31 00:30:38.879167 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:38.879089 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-459r7\" (UniqueName: \"kubernetes.io/projected/fbf98c81-6f05-41b8-ba3b-c10ecbbff5d8-kube-api-access-459r7\") pod \"premium-w0-101b9e-kserve-dc65f694b-m4tsk\" (UID: \"fbf98c81-6f05-41b8-ba3b-c10ecbbff5d8\") " pod="e2e-models-e2e-worker-w0-101b9e/premium-w0-101b9e-kserve-dc65f694b-m4tsk" Oct 31 00:30:38.928575 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:38.928534 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-101b9e/premium-w0-101b9e-kserve-dc65f694b-m4tsk" Oct 31 00:30:39.339810 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:30:39.339779 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfbf98c81_6f05_41b8_ba3b_c10ecbbff5d8.slice/crio-25f790bd521fe1376d4fbb0f9854d13eeae899987403a28c37abebeb4f57b0c5 WatchSource:0}: Error finding container 25f790bd521fe1376d4fbb0f9854d13eeae899987403a28c37abebeb4f57b0c5: Status 404 returned error can't find the container with id 25f790bd521fe1376d4fbb0f9854d13eeae899987403a28c37abebeb4f57b0c5 Oct 31 00:30:39.341495 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:39.341470 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-101b9e/premium-w0-101b9e-kserve-dc65f694b-m4tsk"] Oct 31 00:30:40.270110 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:40.270074 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-101b9e/premium-w0-101b9e-kserve-dc65f694b-m4tsk" event={"ID":"fbf98c81-6f05-41b8-ba3b-c10ecbbff5d8","Type":"ContainerStarted","Data":"3e93fc0c53f37807e7cb279c8763e1f1bf5e2414e5424875b92b886d6127dcbe"} Oct 31 00:30:40.270110 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:40.270118 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-101b9e/premium-w0-101b9e-kserve-dc65f694b-m4tsk" event={"ID":"fbf98c81-6f05-41b8-ba3b-c10ecbbff5d8","Type":"ContainerStarted","Data":"25f790bd521fe1376d4fbb0f9854d13eeae899987403a28c37abebeb4f57b0c5"} Oct 31 00:30:40.389943 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:40.389873 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-101b9e/premium-w0-101b9e-kserve-dc65f694b-m4tsk" podStartSLOduration=2.389857509 podStartE2EDuration="2.389857509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:30:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:30:40.389480604 +0000 UTC m=+932.860650353" watchObservedRunningTime="2026-10-31 00:30:40.389857509 +0000 UTC m=+932.861027260" Oct 31 00:30:40.929224 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:40.929195 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-101b9e/premium-w0-101b9e-kserve-dc65f694b-m4tsk" Oct 31 00:30:41.076395 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:41.076344 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-79tk8"] Oct 31 00:30:47.266662 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:47.266625 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-faf6816a-8t8tg"] Oct 31 00:30:47.273857 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:47.273831 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-faf6816a-8t8tg" Oct 31 00:30:47.332847 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:47.332815 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-faf6816a-8t8tg"] Oct 31 00:30:47.381958 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:47.381909 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:30:47.381958 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:47.381951 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfz9z\"" Oct 31 00:30:47.382175 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:47.381979 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:30:47.386124 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:47.386098 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:30:47.428333 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:47.428288 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31d1a760-c354-43e8-b130-eb62fcb9f2f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-faf6816a-8t8tg\" (UID: \"31d1a760-c354-43e8-b130-eb62fcb9f2f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-faf6816a-8t8tg" Oct 31 00:30:47.428544 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:47.428474 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-755tl\" (UniqueName: \"kubernetes.io/projected/31d1a760-c354-43e8-b130-eb62fcb9f2f1-kube-api-access-755tl\") pod \"maas-api-revoke-sub-faf6816a-8t8tg\" (UID: \"31d1a760-c354-43e8-b130-eb62fcb9f2f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-faf6816a-8t8tg" Oct 31 00:30:47.428544 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:47.428533 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31d1a760-c354-43e8-b130-eb62fcb9f2f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-faf6816a-8t8tg\" (UID: \"31d1a760-c354-43e8-b130-eb62fcb9f2f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-faf6816a-8t8tg" Oct 31 00:30:47.529428 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:47.529303 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31d1a760-c354-43e8-b130-eb62fcb9f2f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-faf6816a-8t8tg\" (UID: \"31d1a760-c354-43e8-b130-eb62fcb9f2f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-faf6816a-8t8tg" Oct 31 00:30:47.529606 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:47.529433 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31d1a760-c354-43e8-b130-eb62fcb9f2f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-faf6816a-8t8tg\" (UID: \"31d1a760-c354-43e8-b130-eb62fcb9f2f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-faf6816a-8t8tg" Oct 31 00:30:47.529606 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:47.529539 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-755tl\" (UniqueName: \"kubernetes.io/projected/31d1a760-c354-43e8-b130-eb62fcb9f2f1-kube-api-access-755tl\") pod \"maas-api-revoke-sub-faf6816a-8t8tg\" (UID: \"31d1a760-c354-43e8-b130-eb62fcb9f2f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-faf6816a-8t8tg" Oct 31 00:30:47.530044 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:47.530022 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31d1a760-c354-43e8-b130-eb62fcb9f2f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-faf6816a-8t8tg\" (UID: \"31d1a760-c354-43e8-b130-eb62fcb9f2f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-faf6816a-8t8tg" Oct 31 00:30:47.531965 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:47.531944 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31d1a760-c354-43e8-b130-eb62fcb9f2f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-faf6816a-8t8tg\" (UID: \"31d1a760-c354-43e8-b130-eb62fcb9f2f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-faf6816a-8t8tg" Oct 31 00:30:47.592203 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:47.592166 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-755tl\" (UniqueName: \"kubernetes.io/projected/31d1a760-c354-43e8-b130-eb62fcb9f2f1-kube-api-access-755tl\") pod \"maas-api-revoke-sub-faf6816a-8t8tg\" (UID: \"31d1a760-c354-43e8-b130-eb62fcb9f2f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-faf6816a-8t8tg" Oct 31 00:30:47.892445 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:47.892313 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-faf6816a-8t8tg" Oct 31 00:30:48.283177 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:30:48.283110 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31d1a760_c354_43e8_b130_eb62fcb9f2f1.slice/crio-3d56d1ee34a64da6d9a3cafe6ad1f9cfcd14076480b7e581c510fa20d188b447 WatchSource:0}: Error finding container 3d56d1ee34a64da6d9a3cafe6ad1f9cfcd14076480b7e581c510fa20d188b447: Status 404 returned error can't find the container with id 3d56d1ee34a64da6d9a3cafe6ad1f9cfcd14076480b7e581c510fa20d188b447 Oct 31 00:30:48.286829 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:48.286799 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-faf6816a-8t8tg"] Oct 31 00:30:48.317047 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:48.317017 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-faf6816a-8t8tg" event={"ID":"31d1a760-c354-43e8-b130-eb62fcb9f2f1","Type":"ContainerStarted","Data":"3d56d1ee34a64da6d9a3cafe6ad1f9cfcd14076480b7e581c510fa20d188b447"} Oct 31 00:30:48.929743 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:48.929651 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-101b9e/premium-w0-101b9e-kserve-dc65f694b-m4tsk" Oct 31 00:30:48.942516 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:48.942489 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-101b9e/premium-w0-101b9e-kserve-dc65f694b-m4tsk" Oct 31 00:30:49.324459 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:49.324417 5222 generic.go:356] "Generic (PLEG): container finished" podID="31d1a760-c354-43e8-b130-eb62fcb9f2f1" containerID="035db8436c66d133a6be99d922a2cbc3ced94d4ee9e87fa0c821f7d1586de576" exitCode=0 Oct 31 00:30:49.324863 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:49.324502 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-faf6816a-8t8tg" event={"ID":"31d1a760-c354-43e8-b130-eb62fcb9f2f1","Type":"ContainerDied","Data":"035db8436c66d133a6be99d922a2cbc3ced94d4ee9e87fa0c821f7d1586de576"} Oct 31 00:30:49.336301 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:49.336271 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-101b9e/premium-w0-101b9e-kserve-dc65f694b-m4tsk" Oct 31 00:30:50.474779 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:50.474752 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-faf6816a-8t8tg" Oct 31 00:30:50.561869 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:50.561829 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31d1a760-c354-43e8-b130-eb62fcb9f2f1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31d1a760-c354-43e8-b130-eb62fcb9f2f1-maas-api-xks-ca\") pod \"31d1a760-c354-43e8-b130-eb62fcb9f2f1\" (UID: \"31d1a760-c354-43e8-b130-eb62fcb9f2f1\") " Oct 31 00:30:50.562073 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:50.561973 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31d1a760-c354-43e8-b130-eb62fcb9f2f1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31d1a760-c354-43e8-b130-eb62fcb9f2f1-maas-api-openshift-ca\") pod \"31d1a760-c354-43e8-b130-eb62fcb9f2f1\" (UID: \"31d1a760-c354-43e8-b130-eb62fcb9f2f1\") " Oct 31 00:30:50.562073 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:50.562031 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31d1a760-c354-43e8-b130-eb62fcb9f2f1-kube-api-access-755tl\" (UniqueName: \"kubernetes.io/projected/31d1a760-c354-43e8-b130-eb62fcb9f2f1-kube-api-access-755tl\") pod \"31d1a760-c354-43e8-b130-eb62fcb9f2f1\" (UID: \"31d1a760-c354-43e8-b130-eb62fcb9f2f1\") " Oct 31 00:30:50.562357 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:50.562333 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31d1a760-c354-43e8-b130-eb62fcb9f2f1-maas-api-openshift-ca" pod "31d1a760-c354-43e8-b130-eb62fcb9f2f1" (UID: "31d1a760-c354-43e8-b130-eb62fcb9f2f1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:30:50.564191 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:50.564164 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31d1a760-c354-43e8-b130-eb62fcb9f2f1-maas-api-xks-ca" pod "31d1a760-c354-43e8-b130-eb62fcb9f2f1" (UID: "31d1a760-c354-43e8-b130-eb62fcb9f2f1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:30:50.564286 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:50.564214 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31d1a760-c354-43e8-b130-eb62fcb9f2f1-kube-api-access-755tl" pod "31d1a760-c354-43e8-b130-eb62fcb9f2f1" (UID: "31d1a760-c354-43e8-b130-eb62fcb9f2f1"). InnerVolumeSpecName "kube-api-access-755tl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:30:50.663345 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:50.663261 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31d1a760-c354-43e8-b130-eb62fcb9f2f1-maas-api-xks-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:50.663345 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:50.663291 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31d1a760-c354-43e8-b130-eb62fcb9f2f1-maas-api-openshift-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:50.663345 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:50.663303 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-755tl\" (UniqueName: \"kubernetes.io/projected/31d1a760-c354-43e8-b130-eb62fcb9f2f1-kube-api-access-755tl\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:51.338230 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:51.338198 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-faf6816a-8t8tg" Oct 31 00:30:51.338407 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:51.338234 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-faf6816a-8t8tg" event={"ID":"31d1a760-c354-43e8-b130-eb62fcb9f2f1","Type":"ContainerDied","Data":"3d56d1ee34a64da6d9a3cafe6ad1f9cfcd14076480b7e581c510fa20d188b447"} Oct 31 00:30:51.338407 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:30:51.338300 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3d56d1ee34a64da6d9a3cafe6ad1f9cfcd14076480b7e581c510fa20d188b447" Oct 31 00:31:05.890315 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:31:05.890278 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 00:31:07.762627 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:07.762592 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c305a8e6-2v7dx"] Oct 31 00:31:07.763314 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:07.763294 5222 cpu_manager.go:414] "containerMap: removing container" podUID="31d1a760-c354-43e8-b130-eb62fcb9f2f1" containerName="revoke-keys" Oct 31 00:31:07.763355 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:07.763319 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31d1a760-c354-43e8-b130-eb62fcb9f2f1" containerName="revoke-keys" Oct 31 00:31:07.763523 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:07.763511 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="31d1a760-c354-43e8-b130-eb62fcb9f2f1" containerName="revoke-keys" Oct 31 00:31:07.768673 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:07.768650 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c305a8e6-2v7dx" Oct 31 00:31:07.822705 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:07.822670 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c305a8e6-2v7dx"] Oct 31 00:31:07.854157 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:07.854126 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b134273-ee2e-4348-8580-f5eba829c608-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c305a8e6-2v7dx\" (UID: \"2b134273-ee2e-4348-8580-f5eba829c608\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c305a8e6-2v7dx" Oct 31 00:31:07.854345 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:07.854176 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b134273-ee2e-4348-8580-f5eba829c608-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c305a8e6-2v7dx\" (UID: \"2b134273-ee2e-4348-8580-f5eba829c608\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c305a8e6-2v7dx" Oct 31 00:31:07.854345 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:07.854317 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qls4w\" (UniqueName: \"kubernetes.io/projected/2b134273-ee2e-4348-8580-f5eba829c608-kube-api-access-qls4w\") pod \"maas-api-revoke-sub-c305a8e6-2v7dx\" (UID: \"2b134273-ee2e-4348-8580-f5eba829c608\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c305a8e6-2v7dx" Oct 31 00:31:07.870860 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:07.870825 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:31:07.870860 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:07.870850 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:31:07.870860 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:07.870844 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:31:07.873576 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:07.873557 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfz9z\"" Oct 31 00:31:07.955752 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:07.955715 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qls4w\" (UniqueName: \"kubernetes.io/projected/2b134273-ee2e-4348-8580-f5eba829c608-kube-api-access-qls4w\") pod \"maas-api-revoke-sub-c305a8e6-2v7dx\" (UID: \"2b134273-ee2e-4348-8580-f5eba829c608\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c305a8e6-2v7dx" Oct 31 00:31:07.955943 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:07.955840 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b134273-ee2e-4348-8580-f5eba829c608-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c305a8e6-2v7dx\" (UID: \"2b134273-ee2e-4348-8580-f5eba829c608\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c305a8e6-2v7dx" Oct 31 00:31:07.955943 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:07.955882 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b134273-ee2e-4348-8580-f5eba829c608-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c305a8e6-2v7dx\" (UID: \"2b134273-ee2e-4348-8580-f5eba829c608\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c305a8e6-2v7dx" Oct 31 00:31:07.956673 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:07.956653 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b134273-ee2e-4348-8580-f5eba829c608-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c305a8e6-2v7dx\" (UID: \"2b134273-ee2e-4348-8580-f5eba829c608\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c305a8e6-2v7dx" Oct 31 00:31:07.958503 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:07.958477 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b134273-ee2e-4348-8580-f5eba829c608-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c305a8e6-2v7dx\" (UID: \"2b134273-ee2e-4348-8580-f5eba829c608\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c305a8e6-2v7dx" Oct 31 00:31:08.019071 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:08.019042 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qls4w\" (UniqueName: \"kubernetes.io/projected/2b134273-ee2e-4348-8580-f5eba829c608-kube-api-access-qls4w\") pod \"maas-api-revoke-sub-c305a8e6-2v7dx\" (UID: \"2b134273-ee2e-4348-8580-f5eba829c608\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c305a8e6-2v7dx" Oct 31 00:31:08.088306 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:08.088272 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c305a8e6-2v7dx" Oct 31 00:31:08.460181 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:31:08.460149 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b134273_ee2e_4348_8580_f5eba829c608.slice/crio-bf23450b8cfaf16d4ae75c71954ea47477d63336d0cdbf9ee62711eb96f91288 WatchSource:0}: Error finding container bf23450b8cfaf16d4ae75c71954ea47477d63336d0cdbf9ee62711eb96f91288: Status 404 returned error can't find the container with id bf23450b8cfaf16d4ae75c71954ea47477d63336d0cdbf9ee62711eb96f91288 Oct 31 00:31:08.463125 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:08.463101 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c305a8e6-2v7dx"] Oct 31 00:31:09.441251 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:09.441214 5222 generic.go:356] "Generic (PLEG): container finished" podID="2b134273-ee2e-4348-8580-f5eba829c608" containerID="8a41330184fa93477ea658d81dc8a739f5af95f3b537b2575b917eff601b9e13" exitCode=0 Oct 31 00:31:09.441685 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:09.441302 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c305a8e6-2v7dx" event={"ID":"2b134273-ee2e-4348-8580-f5eba829c608","Type":"ContainerDied","Data":"8a41330184fa93477ea658d81dc8a739f5af95f3b537b2575b917eff601b9e13"} Oct 31 00:31:09.441685 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:09.441354 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c305a8e6-2v7dx" event={"ID":"2b134273-ee2e-4348-8580-f5eba829c608","Type":"ContainerStarted","Data":"bf23450b8cfaf16d4ae75c71954ea47477d63336d0cdbf9ee62711eb96f91288"} Oct 31 00:31:10.598248 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:10.598214 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c305a8e6-2v7dx" Oct 31 00:31:10.686257 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:10.686217 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b134273-ee2e-4348-8580-f5eba829c608-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b134273-ee2e-4348-8580-f5eba829c608-maas-api-xks-ca\") pod \"2b134273-ee2e-4348-8580-f5eba829c608\" (UID: \"2b134273-ee2e-4348-8580-f5eba829c608\") " Oct 31 00:31:10.686471 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:10.686329 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b134273-ee2e-4348-8580-f5eba829c608-kube-api-access-qls4w\" (UniqueName: \"kubernetes.io/projected/2b134273-ee2e-4348-8580-f5eba829c608-kube-api-access-qls4w\") pod \"2b134273-ee2e-4348-8580-f5eba829c608\" (UID: \"2b134273-ee2e-4348-8580-f5eba829c608\") " Oct 31 00:31:10.686471 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:10.686396 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b134273-ee2e-4348-8580-f5eba829c608-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b134273-ee2e-4348-8580-f5eba829c608-maas-api-openshift-ca\") pod \"2b134273-ee2e-4348-8580-f5eba829c608\" (UID: \"2b134273-ee2e-4348-8580-f5eba829c608\") " Oct 31 00:31:10.686794 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:10.686770 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b134273-ee2e-4348-8580-f5eba829c608-maas-api-openshift-ca" pod "2b134273-ee2e-4348-8580-f5eba829c608" (UID: "2b134273-ee2e-4348-8580-f5eba829c608"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:31:10.688554 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:10.688521 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b134273-ee2e-4348-8580-f5eba829c608-maas-api-xks-ca" pod "2b134273-ee2e-4348-8580-f5eba829c608" (UID: "2b134273-ee2e-4348-8580-f5eba829c608"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:31:10.688668 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:10.688644 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b134273-ee2e-4348-8580-f5eba829c608-kube-api-access-qls4w" pod "2b134273-ee2e-4348-8580-f5eba829c608" (UID: "2b134273-ee2e-4348-8580-f5eba829c608"). InnerVolumeSpecName "kube-api-access-qls4w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:31:10.788277 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:10.788240 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qls4w\" (UniqueName: \"kubernetes.io/projected/2b134273-ee2e-4348-8580-f5eba829c608-kube-api-access-qls4w\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:10.788277 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:10.788273 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b134273-ee2e-4348-8580-f5eba829c608-maas-api-openshift-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:10.788277 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:10.788285 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b134273-ee2e-4348-8580-f5eba829c608-maas-api-xks-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:11.455363 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:11.455325 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c305a8e6-2v7dx" event={"ID":"2b134273-ee2e-4348-8580-f5eba829c608","Type":"ContainerDied","Data":"bf23450b8cfaf16d4ae75c71954ea47477d63336d0cdbf9ee62711eb96f91288"} Oct 31 00:31:11.455549 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:11.455395 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c305a8e6-2v7dx" Oct 31 00:31:11.455549 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:11.455417 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf23450b8cfaf16d4ae75c71954ea47477d63336d0cdbf9ee62711eb96f91288" Oct 31 00:31:17.020975 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:17.020938 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-79tk8"] Oct 31 00:31:18.580087 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:18.580050 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-729c92a6-2vqlw"] Oct 31 00:31:18.580633 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:18.580574 5222 cpu_manager.go:414] "containerMap: removing container" podUID="2b134273-ee2e-4348-8580-f5eba829c608" containerName="revoke-keys" Oct 31 00:31:18.580633 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:18.580587 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b134273-ee2e-4348-8580-f5eba829c608" containerName="revoke-keys" Oct 31 00:31:18.580705 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:18.580654 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b134273-ee2e-4348-8580-f5eba829c608" containerName="revoke-keys" Oct 31 00:31:18.583856 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:18.583838 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-729c92a6-2vqlw" Oct 31 00:31:18.638004 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:18.637968 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-729c92a6-2vqlw"] Oct 31 00:31:18.684711 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:18.684676 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/040ddba8-7817-4ff7-b301-229cc557f8d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-729c92a6-2vqlw\" (UID: \"040ddba8-7817-4ff7-b301-229cc557f8d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-729c92a6-2vqlw" Oct 31 00:31:18.684711 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:18.684715 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmh2s\" (UniqueName: \"kubernetes.io/projected/040ddba8-7817-4ff7-b301-229cc557f8d5-kube-api-access-hmh2s\") pod \"maas-api-revoke-sub-729c92a6-2vqlw\" (UID: \"040ddba8-7817-4ff7-b301-229cc557f8d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-729c92a6-2vqlw" Oct 31 00:31:18.684931 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:18.684890 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/040ddba8-7817-4ff7-b301-229cc557f8d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-729c92a6-2vqlw\" (UID: \"040ddba8-7817-4ff7-b301-229cc557f8d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-729c92a6-2vqlw" Oct 31 00:31:18.685890 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:18.685862 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:31:18.685890 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:18.685885 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:31:18.686023 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:18.685918 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfz9z\"" Oct 31 00:31:18.688580 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:18.688559 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:31:18.785714 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:18.785675 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/040ddba8-7817-4ff7-b301-229cc557f8d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-729c92a6-2vqlw\" (UID: \"040ddba8-7817-4ff7-b301-229cc557f8d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-729c92a6-2vqlw" Oct 31 00:31:18.785714 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:18.785716 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmh2s\" (UniqueName: \"kubernetes.io/projected/040ddba8-7817-4ff7-b301-229cc557f8d5-kube-api-access-hmh2s\") pod \"maas-api-revoke-sub-729c92a6-2vqlw\" (UID: \"040ddba8-7817-4ff7-b301-229cc557f8d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-729c92a6-2vqlw" Oct 31 00:31:18.785956 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:18.785783 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/040ddba8-7817-4ff7-b301-229cc557f8d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-729c92a6-2vqlw\" (UID: \"040ddba8-7817-4ff7-b301-229cc557f8d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-729c92a6-2vqlw" Oct 31 00:31:18.786360 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:18.786334 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/040ddba8-7817-4ff7-b301-229cc557f8d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-729c92a6-2vqlw\" (UID: \"040ddba8-7817-4ff7-b301-229cc557f8d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-729c92a6-2vqlw" Oct 31 00:31:18.788359 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:18.788335 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/040ddba8-7817-4ff7-b301-229cc557f8d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-729c92a6-2vqlw\" (UID: \"040ddba8-7817-4ff7-b301-229cc557f8d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-729c92a6-2vqlw" Oct 31 00:31:18.848072 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:18.848004 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmh2s\" (UniqueName: \"kubernetes.io/projected/040ddba8-7817-4ff7-b301-229cc557f8d5-kube-api-access-hmh2s\") pod \"maas-api-revoke-sub-729c92a6-2vqlw\" (UID: \"040ddba8-7817-4ff7-b301-229cc557f8d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-729c92a6-2vqlw" Oct 31 00:31:18.905153 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:18.905116 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-729c92a6-2vqlw" Oct 31 00:31:19.316103 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:31:19.316075 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod040ddba8_7817_4ff7_b301_229cc557f8d5.slice/crio-139a4a2fbe47af378dce754a5d1eef639aa33a6e58d9a008a93d79e341129006 WatchSource:0}: Error finding container 139a4a2fbe47af378dce754a5d1eef639aa33a6e58d9a008a93d79e341129006: Status 404 returned error can't find the container with id 139a4a2fbe47af378dce754a5d1eef639aa33a6e58d9a008a93d79e341129006 Oct 31 00:31:19.317853 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:19.317819 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-729c92a6-2vqlw"] Oct 31 00:31:19.500984 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:19.500945 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-729c92a6-2vqlw" event={"ID":"040ddba8-7817-4ff7-b301-229cc557f8d5","Type":"ContainerStarted","Data":"139a4a2fbe47af378dce754a5d1eef639aa33a6e58d9a008a93d79e341129006"} Oct 31 00:31:20.507692 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:20.507603 5222 generic.go:356] "Generic (PLEG): container finished" podID="040ddba8-7817-4ff7-b301-229cc557f8d5" containerID="012e592bd4d58e7011f26601b2c2e7bebe0df450fbe7d4f9c5fa75c3f01de8b6" exitCode=0 Oct 31 00:31:20.507692 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:20.507678 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-729c92a6-2vqlw" event={"ID":"040ddba8-7817-4ff7-b301-229cc557f8d5","Type":"ContainerDied","Data":"012e592bd4d58e7011f26601b2c2e7bebe0df450fbe7d4f9c5fa75c3f01de8b6"} Oct 31 00:31:21.682347 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:21.682313 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-729c92a6-2vqlw" Oct 31 00:31:21.717315 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:21.717282 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/040ddba8-7817-4ff7-b301-229cc557f8d5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/040ddba8-7817-4ff7-b301-229cc557f8d5-maas-api-xks-ca\") pod \"040ddba8-7817-4ff7-b301-229cc557f8d5\" (UID: \"040ddba8-7817-4ff7-b301-229cc557f8d5\") " Oct 31 00:31:21.717315 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:21.717322 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/040ddba8-7817-4ff7-b301-229cc557f8d5-kube-api-access-hmh2s\" (UniqueName: \"kubernetes.io/projected/040ddba8-7817-4ff7-b301-229cc557f8d5-kube-api-access-hmh2s\") pod \"040ddba8-7817-4ff7-b301-229cc557f8d5\" (UID: \"040ddba8-7817-4ff7-b301-229cc557f8d5\") " Oct 31 00:31:21.717537 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:21.717470 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/040ddba8-7817-4ff7-b301-229cc557f8d5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/040ddba8-7817-4ff7-b301-229cc557f8d5-maas-api-openshift-ca\") pod \"040ddba8-7817-4ff7-b301-229cc557f8d5\" (UID: \"040ddba8-7817-4ff7-b301-229cc557f8d5\") " Oct 31 00:31:21.717860 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:21.717833 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/040ddba8-7817-4ff7-b301-229cc557f8d5-maas-api-openshift-ca" pod "040ddba8-7817-4ff7-b301-229cc557f8d5" (UID: "040ddba8-7817-4ff7-b301-229cc557f8d5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:31:21.719681 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:21.719657 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/040ddba8-7817-4ff7-b301-229cc557f8d5-maas-api-xks-ca" pod "040ddba8-7817-4ff7-b301-229cc557f8d5" (UID: "040ddba8-7817-4ff7-b301-229cc557f8d5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:31:21.719815 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:21.719795 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/040ddba8-7817-4ff7-b301-229cc557f8d5-kube-api-access-hmh2s" pod "040ddba8-7817-4ff7-b301-229cc557f8d5" (UID: "040ddba8-7817-4ff7-b301-229cc557f8d5"). InnerVolumeSpecName "kube-api-access-hmh2s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:31:21.819405 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:21.819305 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/040ddba8-7817-4ff7-b301-229cc557f8d5-maas-api-xks-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:21.819405 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:21.819334 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmh2s\" (UniqueName: \"kubernetes.io/projected/040ddba8-7817-4ff7-b301-229cc557f8d5-kube-api-access-hmh2s\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:21.819405 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:21.819346 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/040ddba8-7817-4ff7-b301-229cc557f8d5-maas-api-openshift-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:22.536884 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:22.536852 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-729c92a6-2vqlw" event={"ID":"040ddba8-7817-4ff7-b301-229cc557f8d5","Type":"ContainerDied","Data":"139a4a2fbe47af378dce754a5d1eef639aa33a6e58d9a008a93d79e341129006"} Oct 31 00:31:22.537063 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:22.536901 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="139a4a2fbe47af378dce754a5d1eef639aa33a6e58d9a008a93d79e341129006" Oct 31 00:31:22.537063 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:22.536863 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-729c92a6-2vqlw" Oct 31 00:31:23.913397 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:23.913249 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25c7e9be-89hxv"] Oct 31 00:31:23.915868 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:23.915838 5222 cpu_manager.go:414] "containerMap: removing container" podUID="040ddba8-7817-4ff7-b301-229cc557f8d5" containerName="revoke-keys" Oct 31 00:31:23.915868 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:23.915867 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="040ddba8-7817-4ff7-b301-229cc557f8d5" containerName="revoke-keys" Oct 31 00:31:23.916105 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:23.916015 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="040ddba8-7817-4ff7-b301-229cc557f8d5" containerName="revoke-keys" Oct 31 00:31:23.919674 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:23.919652 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25c7e9be-89hxv" Oct 31 00:31:23.941409 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:23.941359 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06d94ff5-8b6d-44c5-a617-1fc79d585fb8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-25c7e9be-89hxv\" (UID: \"06d94ff5-8b6d-44c5-a617-1fc79d585fb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25c7e9be-89hxv" Oct 31 00:31:23.941595 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:23.941528 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2c2w7\" (UniqueName: \"kubernetes.io/projected/06d94ff5-8b6d-44c5-a617-1fc79d585fb8-kube-api-access-2c2w7\") pod \"maas-api-revoke-sub-25c7e9be-89hxv\" (UID: \"06d94ff5-8b6d-44c5-a617-1fc79d585fb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25c7e9be-89hxv" Oct 31 00:31:23.941595 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:23.941581 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06d94ff5-8b6d-44c5-a617-1fc79d585fb8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-25c7e9be-89hxv\" (UID: \"06d94ff5-8b6d-44c5-a617-1fc79d585fb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25c7e9be-89hxv" Oct 31 00:31:23.961261 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:23.961224 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25c7e9be-89hxv"] Oct 31 00:31:23.975550 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:23.975517 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:31:24.008364 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:24.008327 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:31:24.008364 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:24.008355 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfz9z\"" Oct 31 00:31:24.026897 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:24.026860 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:31:24.042506 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:24.042477 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06d94ff5-8b6d-44c5-a617-1fc79d585fb8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-25c7e9be-89hxv\" (UID: \"06d94ff5-8b6d-44c5-a617-1fc79d585fb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25c7e9be-89hxv" Oct 31 00:31:24.042623 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:24.042536 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c2w7\" (UniqueName: \"kubernetes.io/projected/06d94ff5-8b6d-44c5-a617-1fc79d585fb8-kube-api-access-2c2w7\") pod \"maas-api-revoke-sub-25c7e9be-89hxv\" (UID: \"06d94ff5-8b6d-44c5-a617-1fc79d585fb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25c7e9be-89hxv" Oct 31 00:31:24.042623 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:24.042561 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06d94ff5-8b6d-44c5-a617-1fc79d585fb8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-25c7e9be-89hxv\" (UID: \"06d94ff5-8b6d-44c5-a617-1fc79d585fb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25c7e9be-89hxv" Oct 31 00:31:24.043240 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:24.043217 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06d94ff5-8b6d-44c5-a617-1fc79d585fb8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-25c7e9be-89hxv\" (UID: \"06d94ff5-8b6d-44c5-a617-1fc79d585fb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25c7e9be-89hxv" Oct 31 00:31:24.045012 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:24.044984 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06d94ff5-8b6d-44c5-a617-1fc79d585fb8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-25c7e9be-89hxv\" (UID: \"06d94ff5-8b6d-44c5-a617-1fc79d585fb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25c7e9be-89hxv" Oct 31 00:31:24.105397 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:24.105336 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2c2w7\" (UniqueName: \"kubernetes.io/projected/06d94ff5-8b6d-44c5-a617-1fc79d585fb8-kube-api-access-2c2w7\") pod \"maas-api-revoke-sub-25c7e9be-89hxv\" (UID: \"06d94ff5-8b6d-44c5-a617-1fc79d585fb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25c7e9be-89hxv" Oct 31 00:31:24.234856 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:24.234767 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25c7e9be-89hxv" Oct 31 00:31:24.608589 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:31:24.608545 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06d94ff5_8b6d_44c5_a617_1fc79d585fb8.slice/crio-4de434ba8a9857f007e3500e69948e2ffa9c00eac0f00870b566634a914b54bc WatchSource:0}: Error finding container 4de434ba8a9857f007e3500e69948e2ffa9c00eac0f00870b566634a914b54bc: Status 404 returned error can't find the container with id 4de434ba8a9857f007e3500e69948e2ffa9c00eac0f00870b566634a914b54bc Oct 31 00:31:24.611764 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:24.611737 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25c7e9be-89hxv"] Oct 31 00:31:25.555993 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:25.555964 5222 generic.go:356] "Generic (PLEG): container finished" podID="06d94ff5-8b6d-44c5-a617-1fc79d585fb8" containerID="46591766be9065bdf2812ee0a20812d995ca7db03e2257b566edd4223d18c8c8" exitCode=0 Oct 31 00:31:25.556343 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:25.556042 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25c7e9be-89hxv" event={"ID":"06d94ff5-8b6d-44c5-a617-1fc79d585fb8","Type":"ContainerDied","Data":"46591766be9065bdf2812ee0a20812d995ca7db03e2257b566edd4223d18c8c8"} Oct 31 00:31:25.556343 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:25.556097 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25c7e9be-89hxv" event={"ID":"06d94ff5-8b6d-44c5-a617-1fc79d585fb8","Type":"ContainerStarted","Data":"4de434ba8a9857f007e3500e69948e2ffa9c00eac0f00870b566634a914b54bc"} Oct 31 00:31:26.711657 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:26.711633 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25c7e9be-89hxv" Oct 31 00:31:26.772493 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:26.772463 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06d94ff5-8b6d-44c5-a617-1fc79d585fb8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06d94ff5-8b6d-44c5-a617-1fc79d585fb8-maas-api-openshift-ca\") pod \"06d94ff5-8b6d-44c5-a617-1fc79d585fb8\" (UID: \"06d94ff5-8b6d-44c5-a617-1fc79d585fb8\") " Oct 31 00:31:26.772493 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:26.772499 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06d94ff5-8b6d-44c5-a617-1fc79d585fb8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06d94ff5-8b6d-44c5-a617-1fc79d585fb8-maas-api-xks-ca\") pod \"06d94ff5-8b6d-44c5-a617-1fc79d585fb8\" (UID: \"06d94ff5-8b6d-44c5-a617-1fc79d585fb8\") " Oct 31 00:31:26.772721 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:26.772530 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06d94ff5-8b6d-44c5-a617-1fc79d585fb8-kube-api-access-2c2w7\" (UniqueName: \"kubernetes.io/projected/06d94ff5-8b6d-44c5-a617-1fc79d585fb8-kube-api-access-2c2w7\") pod \"06d94ff5-8b6d-44c5-a617-1fc79d585fb8\" (UID: \"06d94ff5-8b6d-44c5-a617-1fc79d585fb8\") " Oct 31 00:31:26.772892 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:26.772870 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06d94ff5-8b6d-44c5-a617-1fc79d585fb8-maas-api-openshift-ca" pod "06d94ff5-8b6d-44c5-a617-1fc79d585fb8" (UID: "06d94ff5-8b6d-44c5-a617-1fc79d585fb8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:31:26.773013 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:26.772999 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06d94ff5-8b6d-44c5-a617-1fc79d585fb8-maas-api-openshift-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:26.774835 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:26.774801 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06d94ff5-8b6d-44c5-a617-1fc79d585fb8-maas-api-xks-ca" pod "06d94ff5-8b6d-44c5-a617-1fc79d585fb8" (UID: "06d94ff5-8b6d-44c5-a617-1fc79d585fb8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:31:26.774949 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:26.774867 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06d94ff5-8b6d-44c5-a617-1fc79d585fb8-kube-api-access-2c2w7" pod "06d94ff5-8b6d-44c5-a617-1fc79d585fb8" (UID: "06d94ff5-8b6d-44c5-a617-1fc79d585fb8"). InnerVolumeSpecName "kube-api-access-2c2w7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:31:26.873855 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:26.873777 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06d94ff5-8b6d-44c5-a617-1fc79d585fb8-maas-api-xks-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:26.873855 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:26.873805 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2c2w7\" (UniqueName: \"kubernetes.io/projected/06d94ff5-8b6d-44c5-a617-1fc79d585fb8-kube-api-access-2c2w7\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:27.568947 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:27.568916 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25c7e9be-89hxv" Oct 31 00:31:27.569131 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:27.568993 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25c7e9be-89hxv" event={"ID":"06d94ff5-8b6d-44c5-a617-1fc79d585fb8","Type":"ContainerDied","Data":"4de434ba8a9857f007e3500e69948e2ffa9c00eac0f00870b566634a914b54bc"} Oct 31 00:31:27.569131 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:27.569062 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4de434ba8a9857f007e3500e69948e2ffa9c00eac0f00870b566634a914b54bc" Oct 31 00:31:34.423469 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:34.423433 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ztfwh"] Oct 31 00:31:34.423912 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:34.423655 5222 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ztfwh" podUID="9980a823-4548-4604-ae1b-a9da1aa6d30a" containerName="istio-proxy" containerID="cri-o://7074fcaa39f01299f615d78cc7e76185ea3fa29ae56c620a2ed4b41a3f1e6d76" gracePeriod=30 Oct 31 00:31:35.890671 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:31:35.890635 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 00:31:39.668547 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:39.668511 5222 generic.go:356] "Generic (PLEG): container finished" podID="9980a823-4548-4604-ae1b-a9da1aa6d30a" containerID="7074fcaa39f01299f615d78cc7e76185ea3fa29ae56c620a2ed4b41a3f1e6d76" exitCode=0 Oct 31 00:31:39.669017 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:39.668825 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ztfwh" event={"ID":"9980a823-4548-4604-ae1b-a9da1aa6d30a","Type":"ContainerDied","Data":"7074fcaa39f01299f615d78cc7e76185ea3fa29ae56c620a2ed4b41a3f1e6d76"} Oct 31 00:31:39.697050 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:39.696971 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ztfwh" Oct 31 00:31:39.816690 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:39.816654 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9980a823-4548-4604-ae1b-a9da1aa6d30a-kube-api-access-vbmq9\" (UniqueName: \"kubernetes.io/projected/9980a823-4548-4604-ae1b-a9da1aa6d30a-kube-api-access-vbmq9\") pod \"9980a823-4548-4604-ae1b-a9da1aa6d30a\" (UID: \"9980a823-4548-4604-ae1b-a9da1aa6d30a\") " Oct 31 00:31:39.816872 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:39.816715 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9980a823-4548-4604-ae1b-a9da1aa6d30a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9980a823-4548-4604-ae1b-a9da1aa6d30a-istio-data\") pod \"9980a823-4548-4604-ae1b-a9da1aa6d30a\" (UID: \"9980a823-4548-4604-ae1b-a9da1aa6d30a\") " Oct 31 00:31:39.816872 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:39.816817 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9980a823-4548-4604-ae1b-a9da1aa6d30a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9980a823-4548-4604-ae1b-a9da1aa6d30a-credential-socket\") pod \"9980a823-4548-4604-ae1b-a9da1aa6d30a\" (UID: \"9980a823-4548-4604-ae1b-a9da1aa6d30a\") " Oct 31 00:31:39.816872 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:39.816851 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9980a823-4548-4604-ae1b-a9da1aa6d30a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9980a823-4548-4604-ae1b-a9da1aa6d30a-istio-envoy\") pod \"9980a823-4548-4604-ae1b-a9da1aa6d30a\" (UID: \"9980a823-4548-4604-ae1b-a9da1aa6d30a\") " Oct 31 00:31:39.816991 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:39.816907 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9980a823-4548-4604-ae1b-a9da1aa6d30a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9980a823-4548-4604-ae1b-a9da1aa6d30a-istio-podinfo\") pod \"9980a823-4548-4604-ae1b-a9da1aa6d30a\" (UID: \"9980a823-4548-4604-ae1b-a9da1aa6d30a\") " Oct 31 00:31:39.816991 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:39.816936 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9980a823-4548-4604-ae1b-a9da1aa6d30a-istio-token\" (UniqueName: \"kubernetes.io/projected/9980a823-4548-4604-ae1b-a9da1aa6d30a-istio-token\") pod \"9980a823-4548-4604-ae1b-a9da1aa6d30a\" (UID: \"9980a823-4548-4604-ae1b-a9da1aa6d30a\") " Oct 31 00:31:39.817090 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:39.816992 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9980a823-4548-4604-ae1b-a9da1aa6d30a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9980a823-4548-4604-ae1b-a9da1aa6d30a-istiod-ca-cert\") pod \"9980a823-4548-4604-ae1b-a9da1aa6d30a\" (UID: \"9980a823-4548-4604-ae1b-a9da1aa6d30a\") " Oct 31 00:31:39.817090 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:39.817032 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9980a823-4548-4604-ae1b-a9da1aa6d30a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9980a823-4548-4604-ae1b-a9da1aa6d30a-workload-socket\") pod \"9980a823-4548-4604-ae1b-a9da1aa6d30a\" (UID: \"9980a823-4548-4604-ae1b-a9da1aa6d30a\") " Oct 31 00:31:39.817090 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:39.817063 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9980a823-4548-4604-ae1b-a9da1aa6d30a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9980a823-4548-4604-ae1b-a9da1aa6d30a-workload-certs\") pod \"9980a823-4548-4604-ae1b-a9da1aa6d30a\" (UID: \"9980a823-4548-4604-ae1b-a9da1aa6d30a\") " Oct 31 00:31:39.817242 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:39.817088 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9980a823-4548-4604-ae1b-a9da1aa6d30a-istio-data" pod "9980a823-4548-4604-ae1b-a9da1aa6d30a" (UID: "9980a823-4548-4604-ae1b-a9da1aa6d30a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:31:39.817242 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:39.817124 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9980a823-4548-4604-ae1b-a9da1aa6d30a-credential-socket" pod "9980a823-4548-4604-ae1b-a9da1aa6d30a" (UID: "9980a823-4548-4604-ae1b-a9da1aa6d30a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:31:39.817343 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:39.817298 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9980a823-4548-4604-ae1b-a9da1aa6d30a-workload-socket" pod "9980a823-4548-4604-ae1b-a9da1aa6d30a" (UID: "9980a823-4548-4604-ae1b-a9da1aa6d30a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:31:39.817461 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:39.817427 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9980a823-4548-4604-ae1b-a9da1aa6d30a-istiod-ca-cert" pod "9980a823-4548-4604-ae1b-a9da1aa6d30a" (UID: "9980a823-4548-4604-ae1b-a9da1aa6d30a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:31:39.817590 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:39.817531 5222 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9980a823-4548-4604-ae1b-a9da1aa6d30a-istiod-ca-cert\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:39.817590 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:39.817552 5222 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9980a823-4548-4604-ae1b-a9da1aa6d30a-workload-socket\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:39.817590 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:39.817565 5222 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9980a823-4548-4604-ae1b-a9da1aa6d30a-istio-data\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:39.817590 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:39.817574 5222 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9980a823-4548-4604-ae1b-a9da1aa6d30a-credential-socket\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:39.817913 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:39.817722 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9980a823-4548-4604-ae1b-a9da1aa6d30a-workload-certs" pod "9980a823-4548-4604-ae1b-a9da1aa6d30a" (UID: "9980a823-4548-4604-ae1b-a9da1aa6d30a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:31:39.819320 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:39.819295 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9980a823-4548-4604-ae1b-a9da1aa6d30a-kube-api-access-vbmq9" pod "9980a823-4548-4604-ae1b-a9da1aa6d30a" (UID: "9980a823-4548-4604-ae1b-a9da1aa6d30a"). InnerVolumeSpecName "kube-api-access-vbmq9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:31:39.819468 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:39.819417 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9980a823-4548-4604-ae1b-a9da1aa6d30a-istio-podinfo" pod "9980a823-4548-4604-ae1b-a9da1aa6d30a" (UID: "9980a823-4548-4604-ae1b-a9da1aa6d30a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:31:39.819744 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:39.819720 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9980a823-4548-4604-ae1b-a9da1aa6d30a-istio-token" pod "9980a823-4548-4604-ae1b-a9da1aa6d30a" (UID: "9980a823-4548-4604-ae1b-a9da1aa6d30a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:31:39.819744 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:39.819728 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9980a823-4548-4604-ae1b-a9da1aa6d30a-istio-envoy" pod "9980a823-4548-4604-ae1b-a9da1aa6d30a" (UID: "9980a823-4548-4604-ae1b-a9da1aa6d30a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:31:39.919063 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:39.919029 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbmq9\" (UniqueName: \"kubernetes.io/projected/9980a823-4548-4604-ae1b-a9da1aa6d30a-kube-api-access-vbmq9\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:39.919063 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:39.919063 5222 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9980a823-4548-4604-ae1b-a9da1aa6d30a-istio-envoy\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:39.919271 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:39.919075 5222 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9980a823-4548-4604-ae1b-a9da1aa6d30a-istio-podinfo\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:39.919271 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:39.919087 5222 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9980a823-4548-4604-ae1b-a9da1aa6d30a-istio-token\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:39.919271 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:39.919101 5222 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9980a823-4548-4604-ae1b-a9da1aa6d30a-workload-certs\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:40.675654 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:40.675621 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ztfwh" Oct 31 00:31:40.676151 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:40.675664 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ztfwh" event={"ID":"9980a823-4548-4604-ae1b-a9da1aa6d30a","Type":"ContainerDied","Data":"889bec921eb97972bc3421a636b9c37101590251cd860808ac37b0bcedaaadd7"} Oct 31 00:31:40.676151 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:40.675729 5222 scope.go:122] "RemoveContainer" containerID="7074fcaa39f01299f615d78cc7e76185ea3fa29ae56c620a2ed4b41a3f1e6d76" Oct 31 00:31:40.896358 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:40.896323 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ztfwh"] Oct 31 00:31:40.950027 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:40.949928 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ztfwh"] Oct 31 00:31:41.007503 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:41.007463 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f04bac1c-nlzq5"] Oct 31 00:31:41.008071 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:41.008046 5222 cpu_manager.go:414] "containerMap: removing container" podUID="9980a823-4548-4604-ae1b-a9da1aa6d30a" containerName="istio-proxy" Oct 31 00:31:41.008071 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:41.008065 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9980a823-4548-4604-ae1b-a9da1aa6d30a" containerName="istio-proxy" Oct 31 00:31:41.008178 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:41.008081 5222 cpu_manager.go:414] "containerMap: removing container" podUID="06d94ff5-8b6d-44c5-a617-1fc79d585fb8" containerName="revoke-keys" Oct 31 00:31:41.008178 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:41.008086 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06d94ff5-8b6d-44c5-a617-1fc79d585fb8" containerName="revoke-keys" Oct 31 00:31:41.008178 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:41.008159 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="9980a823-4548-4604-ae1b-a9da1aa6d30a" containerName="istio-proxy" Oct 31 00:31:41.008178 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:41.008167 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="06d94ff5-8b6d-44c5-a617-1fc79d585fb8" containerName="revoke-keys" Oct 31 00:31:41.012751 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:41.012723 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f04bac1c-nlzq5" Oct 31 00:31:41.065680 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:41.065644 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f04bac1c-nlzq5"] Oct 31 00:31:41.066705 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:41.066679 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfz9z\"" Oct 31 00:31:41.066831 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:41.066773 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:31:41.066831 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:41.066803 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:31:41.067477 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:41.067446 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:31:41.130898 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:41.130858 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5e26610-abd9-476d-b1d5-151a53d88520-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f04bac1c-nlzq5\" (UID: \"a5e26610-abd9-476d-b1d5-151a53d88520\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f04bac1c-nlzq5" Oct 31 00:31:41.131104 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:41.131026 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5e26610-abd9-476d-b1d5-151a53d88520-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f04bac1c-nlzq5\" (UID: \"a5e26610-abd9-476d-b1d5-151a53d88520\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f04bac1c-nlzq5" Oct 31 00:31:41.131104 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:41.131079 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpzz9\" (UniqueName: \"kubernetes.io/projected/a5e26610-abd9-476d-b1d5-151a53d88520-kube-api-access-kpzz9\") pod \"maas-api-revoke-sub-f04bac1c-nlzq5\" (UID: \"a5e26610-abd9-476d-b1d5-151a53d88520\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f04bac1c-nlzq5" Oct 31 00:31:41.231812 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:41.231726 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5e26610-abd9-476d-b1d5-151a53d88520-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f04bac1c-nlzq5\" (UID: \"a5e26610-abd9-476d-b1d5-151a53d88520\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f04bac1c-nlzq5" Oct 31 00:31:41.231812 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:41.231778 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpzz9\" (UniqueName: \"kubernetes.io/projected/a5e26610-abd9-476d-b1d5-151a53d88520-kube-api-access-kpzz9\") pod \"maas-api-revoke-sub-f04bac1c-nlzq5\" (UID: \"a5e26610-abd9-476d-b1d5-151a53d88520\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f04bac1c-nlzq5" Oct 31 00:31:41.232004 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:41.231823 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5e26610-abd9-476d-b1d5-151a53d88520-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f04bac1c-nlzq5\" (UID: \"a5e26610-abd9-476d-b1d5-151a53d88520\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f04bac1c-nlzq5" Oct 31 00:31:41.232485 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:41.232460 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5e26610-abd9-476d-b1d5-151a53d88520-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f04bac1c-nlzq5\" (UID: \"a5e26610-abd9-476d-b1d5-151a53d88520\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f04bac1c-nlzq5" Oct 31 00:31:41.234260 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:41.234239 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5e26610-abd9-476d-b1d5-151a53d88520-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f04bac1c-nlzq5\" (UID: \"a5e26610-abd9-476d-b1d5-151a53d88520\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f04bac1c-nlzq5" Oct 31 00:31:41.296046 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:41.296008 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpzz9\" (UniqueName: \"kubernetes.io/projected/a5e26610-abd9-476d-b1d5-151a53d88520-kube-api-access-kpzz9\") pod \"maas-api-revoke-sub-f04bac1c-nlzq5\" (UID: \"a5e26610-abd9-476d-b1d5-151a53d88520\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f04bac1c-nlzq5" Oct 31 00:31:41.326328 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:41.326299 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f04bac1c-nlzq5" Oct 31 00:31:41.655879 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:41.655846 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9980a823-4548-4604-ae1b-a9da1aa6d30a" path="/var/lib/kubelet/pods/9980a823-4548-4604-ae1b-a9da1aa6d30a/volumes" Oct 31 00:31:41.701036 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:31:41.701005 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5e26610_abd9_476d_b1d5_151a53d88520.slice/crio-60fa58eb0482ca8d5b17a9e812049f6bfcfc6d02eda6f3f350cabb90f87ea8da WatchSource:0}: Error finding container 60fa58eb0482ca8d5b17a9e812049f6bfcfc6d02eda6f3f350cabb90f87ea8da: Status 404 returned error can't find the container with id 60fa58eb0482ca8d5b17a9e812049f6bfcfc6d02eda6f3f350cabb90f87ea8da Oct 31 00:31:41.703322 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:41.703296 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f04bac1c-nlzq5"] Oct 31 00:31:42.114872 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:42.114837 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-beee7c3c-568ms"] Oct 31 00:31:42.118709 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:42.118687 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8010cb27-mhmdq"] Oct 31 00:31:42.118847 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:42.118803 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-beee7c3c-568ms" Oct 31 00:31:42.122269 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:42.122253 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8010cb27-mhmdq" Oct 31 00:31:42.211794 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:42.211685 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8010cb27-mhmdq"] Oct 31 00:31:42.214058 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:42.214030 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-beee7c3c-568ms"] Oct 31 00:31:42.243886 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:42.243850 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jl2zx\" (UniqueName: \"kubernetes.io/projected/9dd3e16f-40ab-458e-801c-fffa6b8b000b-kube-api-access-jl2zx\") pod \"maas-api-revoke-sub-beee7c3c-568ms\" (UID: \"9dd3e16f-40ab-458e-801c-fffa6b8b000b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-beee7c3c-568ms" Oct 31 00:31:42.244035 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:42.243890 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dd3e16f-40ab-458e-801c-fffa6b8b000b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-beee7c3c-568ms\" (UID: \"9dd3e16f-40ab-458e-801c-fffa6b8b000b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-beee7c3c-568ms" Oct 31 00:31:42.244086 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:42.244067 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b533be37-fc63-4eb5-9e7e-5f2b6d6cd815-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8010cb27-mhmdq\" (UID: \"b533be37-fc63-4eb5-9e7e-5f2b6d6cd815\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8010cb27-mhmdq" Oct 31 00:31:42.244126 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:42.244111 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b533be37-fc63-4eb5-9e7e-5f2b6d6cd815-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8010cb27-mhmdq\" (UID: \"b533be37-fc63-4eb5-9e7e-5f2b6d6cd815\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8010cb27-mhmdq" Oct 31 00:31:42.244231 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:42.244212 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6684\" (UniqueName: \"kubernetes.io/projected/b533be37-fc63-4eb5-9e7e-5f2b6d6cd815-kube-api-access-n6684\") pod \"maas-api-revoke-sub-8010cb27-mhmdq\" (UID: \"b533be37-fc63-4eb5-9e7e-5f2b6d6cd815\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8010cb27-mhmdq" Oct 31 00:31:42.244317 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:42.244301 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dd3e16f-40ab-458e-801c-fffa6b8b000b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-beee7c3c-568ms\" (UID: \"9dd3e16f-40ab-458e-801c-fffa6b8b000b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-beee7c3c-568ms" Oct 31 00:31:42.345356 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:42.345313 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b533be37-fc63-4eb5-9e7e-5f2b6d6cd815-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8010cb27-mhmdq\" (UID: \"b533be37-fc63-4eb5-9e7e-5f2b6d6cd815\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8010cb27-mhmdq" Oct 31 00:31:42.345591 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:42.345403 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b533be37-fc63-4eb5-9e7e-5f2b6d6cd815-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8010cb27-mhmdq\" (UID: \"b533be37-fc63-4eb5-9e7e-5f2b6d6cd815\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8010cb27-mhmdq" Oct 31 00:31:42.345591 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:42.345457 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6684\" (UniqueName: \"kubernetes.io/projected/b533be37-fc63-4eb5-9e7e-5f2b6d6cd815-kube-api-access-n6684\") pod \"maas-api-revoke-sub-8010cb27-mhmdq\" (UID: \"b533be37-fc63-4eb5-9e7e-5f2b6d6cd815\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8010cb27-mhmdq" Oct 31 00:31:42.345591 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:42.345561 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dd3e16f-40ab-458e-801c-fffa6b8b000b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-beee7c3c-568ms\" (UID: \"9dd3e16f-40ab-458e-801c-fffa6b8b000b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-beee7c3c-568ms" Oct 31 00:31:42.345766 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:42.345619 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jl2zx\" (UniqueName: \"kubernetes.io/projected/9dd3e16f-40ab-458e-801c-fffa6b8b000b-kube-api-access-jl2zx\") pod \"maas-api-revoke-sub-beee7c3c-568ms\" (UID: \"9dd3e16f-40ab-458e-801c-fffa6b8b000b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-beee7c3c-568ms" Oct 31 00:31:42.345766 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:42.345649 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dd3e16f-40ab-458e-801c-fffa6b8b000b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-beee7c3c-568ms\" (UID: \"9dd3e16f-40ab-458e-801c-fffa6b8b000b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-beee7c3c-568ms" Oct 31 00:31:42.346285 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:42.346261 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b533be37-fc63-4eb5-9e7e-5f2b6d6cd815-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8010cb27-mhmdq\" (UID: \"b533be37-fc63-4eb5-9e7e-5f2b6d6cd815\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8010cb27-mhmdq" Oct 31 00:31:42.346448 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:42.346430 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dd3e16f-40ab-458e-801c-fffa6b8b000b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-beee7c3c-568ms\" (UID: \"9dd3e16f-40ab-458e-801c-fffa6b8b000b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-beee7c3c-568ms" Oct 31 00:31:42.348224 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:42.348196 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dd3e16f-40ab-458e-801c-fffa6b8b000b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-beee7c3c-568ms\" (UID: \"9dd3e16f-40ab-458e-801c-fffa6b8b000b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-beee7c3c-568ms" Oct 31 00:31:42.348333 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:42.348270 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b533be37-fc63-4eb5-9e7e-5f2b6d6cd815-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8010cb27-mhmdq\" (UID: \"b533be37-fc63-4eb5-9e7e-5f2b6d6cd815\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8010cb27-mhmdq" Oct 31 00:31:42.408633 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:42.408604 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jl2zx\" (UniqueName: \"kubernetes.io/projected/9dd3e16f-40ab-458e-801c-fffa6b8b000b-kube-api-access-jl2zx\") pod \"maas-api-revoke-sub-beee7c3c-568ms\" (UID: \"9dd3e16f-40ab-458e-801c-fffa6b8b000b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-beee7c3c-568ms" Oct 31 00:31:42.408788 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:42.408765 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6684\" (UniqueName: \"kubernetes.io/projected/b533be37-fc63-4eb5-9e7e-5f2b6d6cd815-kube-api-access-n6684\") pod \"maas-api-revoke-sub-8010cb27-mhmdq\" (UID: \"b533be37-fc63-4eb5-9e7e-5f2b6d6cd815\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8010cb27-mhmdq" Oct 31 00:31:42.432568 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:42.432531 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-beee7c3c-568ms" Oct 31 00:31:42.439890 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:42.439864 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8010cb27-mhmdq" Oct 31 00:31:42.690508 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:42.690474 5222 generic.go:356] "Generic (PLEG): container finished" podID="a5e26610-abd9-476d-b1d5-151a53d88520" containerID="f991ce90313763e7cca6fa78c7b201564cc071fe2c2b234ac452ceab8171600f" exitCode=0 Oct 31 00:31:42.690680 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:42.690557 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f04bac1c-nlzq5" event={"ID":"a5e26610-abd9-476d-b1d5-151a53d88520","Type":"ContainerDied","Data":"f991ce90313763e7cca6fa78c7b201564cc071fe2c2b234ac452ceab8171600f"} Oct 31 00:31:42.690680 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:42.690606 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f04bac1c-nlzq5" event={"ID":"a5e26610-abd9-476d-b1d5-151a53d88520","Type":"ContainerStarted","Data":"60fa58eb0482ca8d5b17a9e812049f6bfcfc6d02eda6f3f350cabb90f87ea8da"} Oct 31 00:31:42.809048 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:31:42.809016 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dd3e16f_40ab_458e_801c_fffa6b8b000b.slice/crio-989bdd560f43b00500b7c9078399f6fa73ba3b5cb50fc2465b657e1d005ed2fa WatchSource:0}: Error finding container 989bdd560f43b00500b7c9078399f6fa73ba3b5cb50fc2465b657e1d005ed2fa: Status 404 returned error can't find the container with id 989bdd560f43b00500b7c9078399f6fa73ba3b5cb50fc2465b657e1d005ed2fa Oct 31 00:31:42.811777 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:42.811722 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-beee7c3c-568ms"] Oct 31 00:31:42.857130 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:31:42.857096 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb533be37_fc63_4eb5_9e7e_5f2b6d6cd815.slice/crio-f23b2671814686afcf1b6882de607ff81d1b09a52ae44404481385e7813ba6f7 WatchSource:0}: Error finding container f23b2671814686afcf1b6882de607ff81d1b09a52ae44404481385e7813ba6f7: Status 404 returned error can't find the container with id f23b2671814686afcf1b6882de607ff81d1b09a52ae44404481385e7813ba6f7 Oct 31 00:31:42.862806 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:42.862777 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8010cb27-mhmdq"] Oct 31 00:31:43.698855 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:43.698816 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-beee7c3c-568ms" event={"ID":"9dd3e16f-40ab-458e-801c-fffa6b8b000b","Type":"ContainerStarted","Data":"7d7ab68a6eb6e3db473a1401cffd059cec48363481f339d3fb00f01315ad0034"} Oct 31 00:31:43.699004 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:43.698877 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-beee7c3c-568ms" event={"ID":"9dd3e16f-40ab-458e-801c-fffa6b8b000b","Type":"ContainerStarted","Data":"989bdd560f43b00500b7c9078399f6fa73ba3b5cb50fc2465b657e1d005ed2fa"} Oct 31 00:31:43.700540 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:43.700515 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8010cb27-mhmdq" event={"ID":"b533be37-fc63-4eb5-9e7e-5f2b6d6cd815","Type":"ContainerStarted","Data":"4e0667aa29e4ff76f6fe216b47514652c469fd2b5398d90e3c0f7e0f300ee5bb"} Oct 31 00:31:43.700629 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:43.700557 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8010cb27-mhmdq" event={"ID":"b533be37-fc63-4eb5-9e7e-5f2b6d6cd815","Type":"ContainerStarted","Data":"f23b2671814686afcf1b6882de607ff81d1b09a52ae44404481385e7813ba6f7"} Oct 31 00:31:43.820793 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:43.820739 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-beee7c3c-568ms" podStartSLOduration=1.820723761 podStartE2EDuration="1.820723761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:31:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:31:43.81868023 +0000 UTC m=+996.289849980" watchObservedRunningTime="2026-10-31 00:31:43.820723761 +0000 UTC m=+996.291893509" Oct 31 00:31:43.922197 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:43.922172 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f04bac1c-nlzq5" Oct 31 00:31:43.939039 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:43.938492 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8010cb27-mhmdq" podStartSLOduration=1.938473783 podStartE2EDuration="1.938473783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:31:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:31:43.936645124 +0000 UTC m=+996.407814874" watchObservedRunningTime="2026-10-31 00:31:43.938473783 +0000 UTC m=+996.409643531" Oct 31 00:31:44.066126 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:44.066094 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5e26610-abd9-476d-b1d5-151a53d88520-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5e26610-abd9-476d-b1d5-151a53d88520-maas-api-xks-ca\") pod \"a5e26610-abd9-476d-b1d5-151a53d88520\" (UID: \"a5e26610-abd9-476d-b1d5-151a53d88520\") " Oct 31 00:31:44.066290 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:44.066217 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5e26610-abd9-476d-b1d5-151a53d88520-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5e26610-abd9-476d-b1d5-151a53d88520-maas-api-openshift-ca\") pod \"a5e26610-abd9-476d-b1d5-151a53d88520\" (UID: \"a5e26610-abd9-476d-b1d5-151a53d88520\") " Oct 31 00:31:44.066334 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:44.066292 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5e26610-abd9-476d-b1d5-151a53d88520-kube-api-access-kpzz9\" (UniqueName: \"kubernetes.io/projected/a5e26610-abd9-476d-b1d5-151a53d88520-kube-api-access-kpzz9\") pod \"a5e26610-abd9-476d-b1d5-151a53d88520\" (UID: \"a5e26610-abd9-476d-b1d5-151a53d88520\") " Oct 31 00:31:44.066658 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:44.066626 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5e26610-abd9-476d-b1d5-151a53d88520-maas-api-openshift-ca" pod "a5e26610-abd9-476d-b1d5-151a53d88520" (UID: "a5e26610-abd9-476d-b1d5-151a53d88520"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:31:44.066832 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:44.066811 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5e26610-abd9-476d-b1d5-151a53d88520-maas-api-openshift-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:44.068460 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:44.068433 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5e26610-abd9-476d-b1d5-151a53d88520-maas-api-xks-ca" pod "a5e26610-abd9-476d-b1d5-151a53d88520" (UID: "a5e26610-abd9-476d-b1d5-151a53d88520"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:31:44.068961 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:44.068937 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5e26610-abd9-476d-b1d5-151a53d88520-kube-api-access-kpzz9" pod "a5e26610-abd9-476d-b1d5-151a53d88520" (UID: "a5e26610-abd9-476d-b1d5-151a53d88520"). InnerVolumeSpecName "kube-api-access-kpzz9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:31:44.168447 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:44.168409 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpzz9\" (UniqueName: \"kubernetes.io/projected/a5e26610-abd9-476d-b1d5-151a53d88520-kube-api-access-kpzz9\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:44.168447 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:44.168445 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5e26610-abd9-476d-b1d5-151a53d88520-maas-api-xks-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:44.707121 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:44.707093 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f04bac1c-nlzq5" Oct 31 00:31:44.707333 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:44.707090 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f04bac1c-nlzq5" event={"ID":"a5e26610-abd9-476d-b1d5-151a53d88520","Type":"ContainerDied","Data":"60fa58eb0482ca8d5b17a9e812049f6bfcfc6d02eda6f3f350cabb90f87ea8da"} Oct 31 00:31:44.707333 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:44.707232 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="60fa58eb0482ca8d5b17a9e812049f6bfcfc6d02eda6f3f350cabb90f87ea8da" Oct 31 00:31:44.708692 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:44.708669 5222 generic.go:356] "Generic (PLEG): container finished" podID="b533be37-fc63-4eb5-9e7e-5f2b6d6cd815" containerID="4e0667aa29e4ff76f6fe216b47514652c469fd2b5398d90e3c0f7e0f300ee5bb" exitCode=0 Oct 31 00:31:44.708798 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:44.708759 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8010cb27-mhmdq" event={"ID":"b533be37-fc63-4eb5-9e7e-5f2b6d6cd815","Type":"ContainerDied","Data":"4e0667aa29e4ff76f6fe216b47514652c469fd2b5398d90e3c0f7e0f300ee5bb"} Oct 31 00:31:44.710448 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:44.710419 5222 generic.go:356] "Generic (PLEG): container finished" podID="9dd3e16f-40ab-458e-801c-fffa6b8b000b" containerID="7d7ab68a6eb6e3db473a1401cffd059cec48363481f339d3fb00f01315ad0034" exitCode=0 Oct 31 00:31:44.710534 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:44.710490 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-beee7c3c-568ms" event={"ID":"9dd3e16f-40ab-458e-801c-fffa6b8b000b","Type":"ContainerDied","Data":"7d7ab68a6eb6e3db473a1401cffd059cec48363481f339d3fb00f01315ad0034"} Oct 31 00:31:45.929049 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:45.929022 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-beee7c3c-568ms" Oct 31 00:31:45.931281 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:45.931261 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8010cb27-mhmdq" Oct 31 00:31:46.093753 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:46.093717 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b533be37-fc63-4eb5-9e7e-5f2b6d6cd815-kube-api-access-n6684\" (UniqueName: \"kubernetes.io/projected/b533be37-fc63-4eb5-9e7e-5f2b6d6cd815-kube-api-access-n6684\") pod \"b533be37-fc63-4eb5-9e7e-5f2b6d6cd815\" (UID: \"b533be37-fc63-4eb5-9e7e-5f2b6d6cd815\") " Oct 31 00:31:46.093910 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:46.093766 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b533be37-fc63-4eb5-9e7e-5f2b6d6cd815-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b533be37-fc63-4eb5-9e7e-5f2b6d6cd815-maas-api-openshift-ca\") pod \"b533be37-fc63-4eb5-9e7e-5f2b6d6cd815\" (UID: \"b533be37-fc63-4eb5-9e7e-5f2b6d6cd815\") " Oct 31 00:31:46.093910 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:46.093817 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b533be37-fc63-4eb5-9e7e-5f2b6d6cd815-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b533be37-fc63-4eb5-9e7e-5f2b6d6cd815-maas-api-xks-ca\") pod \"b533be37-fc63-4eb5-9e7e-5f2b6d6cd815\" (UID: \"b533be37-fc63-4eb5-9e7e-5f2b6d6cd815\") " Oct 31 00:31:46.093910 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:46.093880 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9dd3e16f-40ab-458e-801c-fffa6b8b000b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dd3e16f-40ab-458e-801c-fffa6b8b000b-maas-api-xks-ca\") pod \"9dd3e16f-40ab-458e-801c-fffa6b8b000b\" (UID: \"9dd3e16f-40ab-458e-801c-fffa6b8b000b\") " Oct 31 00:31:46.093910 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:46.093906 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9dd3e16f-40ab-458e-801c-fffa6b8b000b-kube-api-access-jl2zx\" (UniqueName: \"kubernetes.io/projected/9dd3e16f-40ab-458e-801c-fffa6b8b000b-kube-api-access-jl2zx\") pod \"9dd3e16f-40ab-458e-801c-fffa6b8b000b\" (UID: \"9dd3e16f-40ab-458e-801c-fffa6b8b000b\") " Oct 31 00:31:46.094129 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:46.093944 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9dd3e16f-40ab-458e-801c-fffa6b8b000b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dd3e16f-40ab-458e-801c-fffa6b8b000b-maas-api-openshift-ca\") pod \"9dd3e16f-40ab-458e-801c-fffa6b8b000b\" (UID: \"9dd3e16f-40ab-458e-801c-fffa6b8b000b\") " Oct 31 00:31:46.094496 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:46.094353 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b533be37-fc63-4eb5-9e7e-5f2b6d6cd815-maas-api-openshift-ca" pod "b533be37-fc63-4eb5-9e7e-5f2b6d6cd815" (UID: "b533be37-fc63-4eb5-9e7e-5f2b6d6cd815"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:31:46.094818 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:46.094535 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9dd3e16f-40ab-458e-801c-fffa6b8b000b-maas-api-openshift-ca" pod "9dd3e16f-40ab-458e-801c-fffa6b8b000b" (UID: "9dd3e16f-40ab-458e-801c-fffa6b8b000b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:31:46.096407 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:46.096284 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9dd3e16f-40ab-458e-801c-fffa6b8b000b-maas-api-xks-ca" pod "9dd3e16f-40ab-458e-801c-fffa6b8b000b" (UID: "9dd3e16f-40ab-458e-801c-fffa6b8b000b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:31:46.096517 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:46.096458 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b533be37-fc63-4eb5-9e7e-5f2b6d6cd815-kube-api-access-n6684" pod "b533be37-fc63-4eb5-9e7e-5f2b6d6cd815" (UID: "b533be37-fc63-4eb5-9e7e-5f2b6d6cd815"). InnerVolumeSpecName "kube-api-access-n6684". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:31:46.096756 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:46.096728 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b533be37-fc63-4eb5-9e7e-5f2b6d6cd815-maas-api-xks-ca" pod "b533be37-fc63-4eb5-9e7e-5f2b6d6cd815" (UID: "b533be37-fc63-4eb5-9e7e-5f2b6d6cd815"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:31:46.096842 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:46.096819 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dd3e16f-40ab-458e-801c-fffa6b8b000b-kube-api-access-jl2zx" pod "9dd3e16f-40ab-458e-801c-fffa6b8b000b" (UID: "9dd3e16f-40ab-458e-801c-fffa6b8b000b"). InnerVolumeSpecName "kube-api-access-jl2zx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:31:46.195204 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:46.195168 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6684\" (UniqueName: \"kubernetes.io/projected/b533be37-fc63-4eb5-9e7e-5f2b6d6cd815-kube-api-access-n6684\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:46.195204 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:46.195199 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b533be37-fc63-4eb5-9e7e-5f2b6d6cd815-maas-api-openshift-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:46.195204 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:46.195210 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b533be37-fc63-4eb5-9e7e-5f2b6d6cd815-maas-api-xks-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:46.195529 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:46.195219 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dd3e16f-40ab-458e-801c-fffa6b8b000b-maas-api-xks-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:46.195529 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:46.195228 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jl2zx\" (UniqueName: \"kubernetes.io/projected/9dd3e16f-40ab-458e-801c-fffa6b8b000b-kube-api-access-jl2zx\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:46.195529 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:46.195238 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dd3e16f-40ab-458e-801c-fffa6b8b000b-maas-api-openshift-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:46.727601 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:46.727564 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8010cb27-mhmdq" event={"ID":"b533be37-fc63-4eb5-9e7e-5f2b6d6cd815","Type":"ContainerDied","Data":"f23b2671814686afcf1b6882de607ff81d1b09a52ae44404481385e7813ba6f7"} Oct 31 00:31:46.727797 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:46.727628 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f23b2671814686afcf1b6882de607ff81d1b09a52ae44404481385e7813ba6f7" Oct 31 00:31:46.727891 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:46.727811 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8010cb27-mhmdq" Oct 31 00:31:46.730430 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:46.730354 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-beee7c3c-568ms" event={"ID":"9dd3e16f-40ab-458e-801c-fffa6b8b000b","Type":"ContainerDied","Data":"989bdd560f43b00500b7c9078399f6fa73ba3b5cb50fc2465b657e1d005ed2fa"} Oct 31 00:31:46.730430 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:46.730410 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-beee7c3c-568ms" Oct 31 00:31:46.730713 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:46.730681 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="989bdd560f43b00500b7c9078399f6fa73ba3b5cb50fc2465b657e1d005ed2fa" Oct 31 00:31:58.597402 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:58.597354 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-701a0bb8-l4ld8"] Oct 31 00:31:58.597896 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:58.597882 5222 cpu_manager.go:414] "containerMap: removing container" podUID="a5e26610-abd9-476d-b1d5-151a53d88520" containerName="revoke-keys" Oct 31 00:31:58.597944 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:58.597897 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5e26610-abd9-476d-b1d5-151a53d88520" containerName="revoke-keys" Oct 31 00:31:58.597944 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:58.597917 5222 cpu_manager.go:414] "containerMap: removing container" podUID="9dd3e16f-40ab-458e-801c-fffa6b8b000b" containerName="revoke-keys" Oct 31 00:31:58.597944 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:58.597922 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9dd3e16f-40ab-458e-801c-fffa6b8b000b" containerName="revoke-keys" Oct 31 00:31:58.598036 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:58.597945 5222 cpu_manager.go:414] "containerMap: removing container" podUID="b533be37-fc63-4eb5-9e7e-5f2b6d6cd815" containerName="revoke-keys" Oct 31 00:31:58.598036 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:58.597950 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b533be37-fc63-4eb5-9e7e-5f2b6d6cd815" containerName="revoke-keys" Oct 31 00:31:58.598036 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:58.598028 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5e26610-abd9-476d-b1d5-151a53d88520" containerName="revoke-keys" Oct 31 00:31:58.598125 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:58.598038 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="9dd3e16f-40ab-458e-801c-fffa6b8b000b" containerName="revoke-keys" Oct 31 00:31:58.598125 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:58.598046 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="b533be37-fc63-4eb5-9e7e-5f2b6d6cd815" containerName="revoke-keys" Oct 31 00:31:58.601470 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:58.601449 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-701a0bb8-l4ld8" Oct 31 00:31:58.630313 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:58.630274 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5f79f3d-1edc-4406-94bc-0785f51d9cb8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-701a0bb8-l4ld8\" (UID: \"d5f79f3d-1edc-4406-94bc-0785f51d9cb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-701a0bb8-l4ld8" Oct 31 00:31:58.630523 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:58.630422 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vbqd\" (UniqueName: \"kubernetes.io/projected/d5f79f3d-1edc-4406-94bc-0785f51d9cb8-kube-api-access-6vbqd\") pod \"maas-api-revoke-sub-701a0bb8-l4ld8\" (UID: \"d5f79f3d-1edc-4406-94bc-0785f51d9cb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-701a0bb8-l4ld8" Oct 31 00:31:58.630523 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:58.630503 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5f79f3d-1edc-4406-94bc-0785f51d9cb8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-701a0bb8-l4ld8\" (UID: \"d5f79f3d-1edc-4406-94bc-0785f51d9cb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-701a0bb8-l4ld8" Oct 31 00:31:58.655674 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:58.655642 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-701a0bb8-l4ld8"] Oct 31 00:31:58.704394 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:58.704317 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:31:58.704583 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:58.704421 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfz9z\"" Oct 31 00:31:58.704583 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:58.704465 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:31:58.704583 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:58.704518 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:31:58.732302 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:58.732263 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5f79f3d-1edc-4406-94bc-0785f51d9cb8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-701a0bb8-l4ld8\" (UID: \"d5f79f3d-1edc-4406-94bc-0785f51d9cb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-701a0bb8-l4ld8" Oct 31 00:31:58.732517 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:58.732360 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5f79f3d-1edc-4406-94bc-0785f51d9cb8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-701a0bb8-l4ld8\" (UID: \"d5f79f3d-1edc-4406-94bc-0785f51d9cb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-701a0bb8-l4ld8" Oct 31 00:31:58.732517 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:58.732460 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vbqd\" (UniqueName: \"kubernetes.io/projected/d5f79f3d-1edc-4406-94bc-0785f51d9cb8-kube-api-access-6vbqd\") pod \"maas-api-revoke-sub-701a0bb8-l4ld8\" (UID: \"d5f79f3d-1edc-4406-94bc-0785f51d9cb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-701a0bb8-l4ld8" Oct 31 00:31:58.732997 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:58.732971 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5f79f3d-1edc-4406-94bc-0785f51d9cb8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-701a0bb8-l4ld8\" (UID: \"d5f79f3d-1edc-4406-94bc-0785f51d9cb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-701a0bb8-l4ld8" Oct 31 00:31:58.734937 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:58.734913 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5f79f3d-1edc-4406-94bc-0785f51d9cb8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-701a0bb8-l4ld8\" (UID: \"d5f79f3d-1edc-4406-94bc-0785f51d9cb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-701a0bb8-l4ld8" Oct 31 00:31:58.795245 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:58.795212 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vbqd\" (UniqueName: \"kubernetes.io/projected/d5f79f3d-1edc-4406-94bc-0785f51d9cb8-kube-api-access-6vbqd\") pod \"maas-api-revoke-sub-701a0bb8-l4ld8\" (UID: \"d5f79f3d-1edc-4406-94bc-0785f51d9cb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-701a0bb8-l4ld8" Oct 31 00:31:58.915077 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:58.914999 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-701a0bb8-l4ld8" Oct 31 00:31:58.918753 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:58.918563 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37870f8f-9mrhh"] Oct 31 00:31:58.922673 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:58.922651 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37870f8f-9mrhh" Oct 31 00:31:58.934964 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:58.934926 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b8ba874-e544-43e0-a229-3fb9c78f57e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37870f8f-9mrhh\" (UID: \"2b8ba874-e544-43e0-a229-3fb9c78f57e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37870f8f-9mrhh" Oct 31 00:31:58.935164 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:58.935144 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fd22z\" (UniqueName: \"kubernetes.io/projected/2b8ba874-e544-43e0-a229-3fb9c78f57e8-kube-api-access-fd22z\") pod \"maas-api-revoke-sub-37870f8f-9mrhh\" (UID: \"2b8ba874-e544-43e0-a229-3fb9c78f57e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37870f8f-9mrhh" Oct 31 00:31:58.935252 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:58.935187 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b8ba874-e544-43e0-a229-3fb9c78f57e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37870f8f-9mrhh\" (UID: \"2b8ba874-e544-43e0-a229-3fb9c78f57e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37870f8f-9mrhh" Oct 31 00:31:58.978709 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:58.978611 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37870f8f-9mrhh"] Oct 31 00:31:59.036442 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:59.036404 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fd22z\" (UniqueName: \"kubernetes.io/projected/2b8ba874-e544-43e0-a229-3fb9c78f57e8-kube-api-access-fd22z\") pod \"maas-api-revoke-sub-37870f8f-9mrhh\" (UID: \"2b8ba874-e544-43e0-a229-3fb9c78f57e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37870f8f-9mrhh" Oct 31 00:31:59.036609 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:59.036450 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b8ba874-e544-43e0-a229-3fb9c78f57e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37870f8f-9mrhh\" (UID: \"2b8ba874-e544-43e0-a229-3fb9c78f57e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37870f8f-9mrhh" Oct 31 00:31:59.036609 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:59.036513 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b8ba874-e544-43e0-a229-3fb9c78f57e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37870f8f-9mrhh\" (UID: \"2b8ba874-e544-43e0-a229-3fb9c78f57e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37870f8f-9mrhh" Oct 31 00:31:59.037287 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:59.037223 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b8ba874-e544-43e0-a229-3fb9c78f57e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37870f8f-9mrhh\" (UID: \"2b8ba874-e544-43e0-a229-3fb9c78f57e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37870f8f-9mrhh" Oct 31 00:31:59.039008 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:59.038984 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b8ba874-e544-43e0-a229-3fb9c78f57e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37870f8f-9mrhh\" (UID: \"2b8ba874-e544-43e0-a229-3fb9c78f57e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37870f8f-9mrhh" Oct 31 00:31:59.098930 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:59.098899 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fd22z\" (UniqueName: \"kubernetes.io/projected/2b8ba874-e544-43e0-a229-3fb9c78f57e8-kube-api-access-fd22z\") pod \"maas-api-revoke-sub-37870f8f-9mrhh\" (UID: \"2b8ba874-e544-43e0-a229-3fb9c78f57e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37870f8f-9mrhh" Oct 31 00:31:59.256593 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:59.256512 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37870f8f-9mrhh" Oct 31 00:31:59.294913 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:31:59.294883 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5f79f3d_1edc_4406_94bc_0785f51d9cb8.slice/crio-fa5330f72abceb43c2b89916621069335a47b94dce00614e4b5bd4a0af99ea44 WatchSource:0}: Error finding container fa5330f72abceb43c2b89916621069335a47b94dce00614e4b5bd4a0af99ea44: Status 404 returned error can't find the container with id fa5330f72abceb43c2b89916621069335a47b94dce00614e4b5bd4a0af99ea44 Oct 31 00:31:59.296882 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:59.296572 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-701a0bb8-l4ld8"] Oct 31 00:31:59.574436 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:59.574401 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b5d5936-5dgtj"] Oct 31 00:31:59.579456 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:59.579431 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b5d5936-5dgtj" Oct 31 00:31:59.633659 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:59.633611 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b5d5936-5dgtj"] Oct 31 00:31:59.637049 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:31:59.637020 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b8ba874_e544_43e0_a229_3fb9c78f57e8.slice/crio-3ace98d367dd34c85c0e9dd93fc6929ec7aa4a7d53a630eaead3a2934200ce2f WatchSource:0}: Error finding container 3ace98d367dd34c85c0e9dd93fc6929ec7aa4a7d53a630eaead3a2934200ce2f: Status 404 returned error can't find the container with id 3ace98d367dd34c85c0e9dd93fc6929ec7aa4a7d53a630eaead3a2934200ce2f Oct 31 00:31:59.637169 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:59.637062 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37870f8f-9mrhh"] Oct 31 00:31:59.643187 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:59.643152 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37a9c8d6-0dc8-485a-82e5-ef26bb3785a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b5d5936-5dgtj\" (UID: \"37a9c8d6-0dc8-485a-82e5-ef26bb3785a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b5d5936-5dgtj" Oct 31 00:31:59.643304 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:59.643246 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pglj\" (UniqueName: \"kubernetes.io/projected/37a9c8d6-0dc8-485a-82e5-ef26bb3785a0-kube-api-access-7pglj\") pod \"maas-api-revoke-sub-3b5d5936-5dgtj\" (UID: \"37a9c8d6-0dc8-485a-82e5-ef26bb3785a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b5d5936-5dgtj" Oct 31 00:31:59.643466 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:59.643308 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37a9c8d6-0dc8-485a-82e5-ef26bb3785a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b5d5936-5dgtj\" (UID: \"37a9c8d6-0dc8-485a-82e5-ef26bb3785a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b5d5936-5dgtj" Oct 31 00:31:59.744477 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:59.744445 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37a9c8d6-0dc8-485a-82e5-ef26bb3785a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b5d5936-5dgtj\" (UID: \"37a9c8d6-0dc8-485a-82e5-ef26bb3785a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b5d5936-5dgtj" Oct 31 00:31:59.744643 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:59.744510 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pglj\" (UniqueName: \"kubernetes.io/projected/37a9c8d6-0dc8-485a-82e5-ef26bb3785a0-kube-api-access-7pglj\") pod \"maas-api-revoke-sub-3b5d5936-5dgtj\" (UID: \"37a9c8d6-0dc8-485a-82e5-ef26bb3785a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b5d5936-5dgtj" Oct 31 00:31:59.744781 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:59.744762 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37a9c8d6-0dc8-485a-82e5-ef26bb3785a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b5d5936-5dgtj\" (UID: \"37a9c8d6-0dc8-485a-82e5-ef26bb3785a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b5d5936-5dgtj" Oct 31 00:31:59.745190 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:59.745162 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37a9c8d6-0dc8-485a-82e5-ef26bb3785a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b5d5936-5dgtj\" (UID: \"37a9c8d6-0dc8-485a-82e5-ef26bb3785a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b5d5936-5dgtj" Oct 31 00:31:59.747317 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:59.747276 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37a9c8d6-0dc8-485a-82e5-ef26bb3785a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b5d5936-5dgtj\" (UID: \"37a9c8d6-0dc8-485a-82e5-ef26bb3785a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b5d5936-5dgtj" Oct 31 00:31:59.808016 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:59.807931 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pglj\" (UniqueName: \"kubernetes.io/projected/37a9c8d6-0dc8-485a-82e5-ef26bb3785a0-kube-api-access-7pglj\") pod \"maas-api-revoke-sub-3b5d5936-5dgtj\" (UID: \"37a9c8d6-0dc8-485a-82e5-ef26bb3785a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b5d5936-5dgtj" Oct 31 00:31:59.809552 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:59.809519 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37870f8f-9mrhh" event={"ID":"2b8ba874-e544-43e0-a229-3fb9c78f57e8","Type":"ContainerStarted","Data":"3ace98d367dd34c85c0e9dd93fc6929ec7aa4a7d53a630eaead3a2934200ce2f"} Oct 31 00:31:59.811240 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:59.811215 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-701a0bb8-l4ld8" event={"ID":"d5f79f3d-1edc-4406-94bc-0785f51d9cb8","Type":"ContainerStarted","Data":"8f58f08d71ab0fdf926cf0be72b607fd43ebeb004875348db78548d204ea059b"} Oct 31 00:31:59.811395 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:59.811257 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-701a0bb8-l4ld8" event={"ID":"d5f79f3d-1edc-4406-94bc-0785f51d9cb8","Type":"ContainerStarted","Data":"fa5330f72abceb43c2b89916621069335a47b94dce00614e4b5bd4a0af99ea44"} Oct 31 00:31:59.893735 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:59.893704 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b5d5936-5dgtj" Oct 31 00:31:59.929611 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:31:59.929561 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-701a0bb8-l4ld8" podStartSLOduration=1.9295438 podStartE2EDuration="1.9295438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:31:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:31:59.928962719 +0000 UTC m=+1012.400132472" watchObservedRunningTime="2026-10-31 00:31:59.9295438 +0000 UTC m=+1012.400713548" Oct 31 00:32:00.273824 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:32:00.273790 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37a9c8d6_0dc8_485a_82e5_ef26bb3785a0.slice/crio-5e8116d1c7f17e50edd41fc943420d75832ea60ab0a0285c88434e10d2972b84 WatchSource:0}: Error finding container 5e8116d1c7f17e50edd41fc943420d75832ea60ab0a0285c88434e10d2972b84: Status 404 returned error can't find the container with id 5e8116d1c7f17e50edd41fc943420d75832ea60ab0a0285c88434e10d2972b84 Oct 31 00:32:00.277268 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:00.277243 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b5d5936-5dgtj"] Oct 31 00:32:00.818542 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:00.818459 5222 generic.go:356] "Generic (PLEG): container finished" podID="d5f79f3d-1edc-4406-94bc-0785f51d9cb8" containerID="8f58f08d71ab0fdf926cf0be72b607fd43ebeb004875348db78548d204ea059b" exitCode=0 Oct 31 00:32:00.818972 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:00.818546 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-701a0bb8-l4ld8" event={"ID":"d5f79f3d-1edc-4406-94bc-0785f51d9cb8","Type":"ContainerDied","Data":"8f58f08d71ab0fdf926cf0be72b607fd43ebeb004875348db78548d204ea059b"} Oct 31 00:32:00.820116 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:00.820092 5222 generic.go:356] "Generic (PLEG): container finished" podID="2b8ba874-e544-43e0-a229-3fb9c78f57e8" containerID="9513938b764987ac85d4d4635b2936364bf983688b1d5b4080a9ab5d0fa913b1" exitCode=0 Oct 31 00:32:00.820223 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:00.820177 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37870f8f-9mrhh" event={"ID":"2b8ba874-e544-43e0-a229-3fb9c78f57e8","Type":"ContainerDied","Data":"9513938b764987ac85d4d4635b2936364bf983688b1d5b4080a9ab5d0fa913b1"} Oct 31 00:32:00.821666 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:00.821644 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b5d5936-5dgtj" event={"ID":"37a9c8d6-0dc8-485a-82e5-ef26bb3785a0","Type":"ContainerStarted","Data":"2ed38402b1edb1bbc029705953f1a0b94cf4e7b7030fa33f904daa015148df8e"} Oct 31 00:32:00.821748 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:00.821688 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b5d5936-5dgtj" event={"ID":"37a9c8d6-0dc8-485a-82e5-ef26bb3785a0","Type":"ContainerStarted","Data":"5e8116d1c7f17e50edd41fc943420d75832ea60ab0a0285c88434e10d2972b84"} Oct 31 00:32:01.174548 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:01.174492 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b5d5936-5dgtj" podStartSLOduration=2.174477645 podStartE2EDuration="2.174477645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:31:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:32:01.173038529 +0000 UTC m=+1013.644208295" watchObservedRunningTime="2026-10-31 00:32:01.174477645 +0000 UTC m=+1013.645647393" Oct 31 00:32:01.828871 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:01.828835 5222 generic.go:356] "Generic (PLEG): container finished" podID="37a9c8d6-0dc8-485a-82e5-ef26bb3785a0" containerID="2ed38402b1edb1bbc029705953f1a0b94cf4e7b7030fa33f904daa015148df8e" exitCode=0 Oct 31 00:32:01.829291 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:01.828922 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b5d5936-5dgtj" event={"ID":"37a9c8d6-0dc8-485a-82e5-ef26bb3785a0","Type":"ContainerDied","Data":"2ed38402b1edb1bbc029705953f1a0b94cf4e7b7030fa33f904daa015148df8e"} Oct 31 00:32:01.996619 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:01.996590 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37870f8f-9mrhh" Oct 31 00:32:02.025181 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:02.025152 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-701a0bb8-l4ld8" Oct 31 00:32:02.070492 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:02.070454 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b8ba874-e544-43e0-a229-3fb9c78f57e8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b8ba874-e544-43e0-a229-3fb9c78f57e8-maas-api-xks-ca\") pod \"2b8ba874-e544-43e0-a229-3fb9c78f57e8\" (UID: \"2b8ba874-e544-43e0-a229-3fb9c78f57e8\") " Oct 31 00:32:02.070642 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:02.070518 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5f79f3d-1edc-4406-94bc-0785f51d9cb8-kube-api-access-6vbqd\" (UniqueName: \"kubernetes.io/projected/d5f79f3d-1edc-4406-94bc-0785f51d9cb8-kube-api-access-6vbqd\") pod \"d5f79f3d-1edc-4406-94bc-0785f51d9cb8\" (UID: \"d5f79f3d-1edc-4406-94bc-0785f51d9cb8\") " Oct 31 00:32:02.070642 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:02.070598 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5f79f3d-1edc-4406-94bc-0785f51d9cb8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5f79f3d-1edc-4406-94bc-0785f51d9cb8-maas-api-xks-ca\") pod \"d5f79f3d-1edc-4406-94bc-0785f51d9cb8\" (UID: \"d5f79f3d-1edc-4406-94bc-0785f51d9cb8\") " Oct 31 00:32:02.070781 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:02.070690 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b8ba874-e544-43e0-a229-3fb9c78f57e8-kube-api-access-fd22z\" (UniqueName: \"kubernetes.io/projected/2b8ba874-e544-43e0-a229-3fb9c78f57e8-kube-api-access-fd22z\") pod \"2b8ba874-e544-43e0-a229-3fb9c78f57e8\" (UID: \"2b8ba874-e544-43e0-a229-3fb9c78f57e8\") " Oct 31 00:32:02.070781 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:02.070750 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5f79f3d-1edc-4406-94bc-0785f51d9cb8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5f79f3d-1edc-4406-94bc-0785f51d9cb8-maas-api-openshift-ca\") pod \"d5f79f3d-1edc-4406-94bc-0785f51d9cb8\" (UID: \"d5f79f3d-1edc-4406-94bc-0785f51d9cb8\") " Oct 31 00:32:02.070899 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:02.070876 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b8ba874-e544-43e0-a229-3fb9c78f57e8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b8ba874-e544-43e0-a229-3fb9c78f57e8-maas-api-openshift-ca\") pod \"2b8ba874-e544-43e0-a229-3fb9c78f57e8\" (UID: \"2b8ba874-e544-43e0-a229-3fb9c78f57e8\") " Oct 31 00:32:02.071172 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:02.071142 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5f79f3d-1edc-4406-94bc-0785f51d9cb8-maas-api-openshift-ca" pod "d5f79f3d-1edc-4406-94bc-0785f51d9cb8" (UID: "d5f79f3d-1edc-4406-94bc-0785f51d9cb8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:32:02.071511 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:02.071486 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5f79f3d-1edc-4406-94bc-0785f51d9cb8-maas-api-openshift-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:02.071595 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:02.071487 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b8ba874-e544-43e0-a229-3fb9c78f57e8-maas-api-openshift-ca" pod "2b8ba874-e544-43e0-a229-3fb9c78f57e8" (UID: "2b8ba874-e544-43e0-a229-3fb9c78f57e8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:32:02.072916 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:02.072889 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b8ba874-e544-43e0-a229-3fb9c78f57e8-maas-api-xks-ca" pod "2b8ba874-e544-43e0-a229-3fb9c78f57e8" (UID: "2b8ba874-e544-43e0-a229-3fb9c78f57e8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:32:02.073004 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:02.072939 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5f79f3d-1edc-4406-94bc-0785f51d9cb8-kube-api-access-6vbqd" pod "d5f79f3d-1edc-4406-94bc-0785f51d9cb8" (UID: "d5f79f3d-1edc-4406-94bc-0785f51d9cb8"). InnerVolumeSpecName "kube-api-access-6vbqd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:32:02.073390 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:02.073352 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5f79f3d-1edc-4406-94bc-0785f51d9cb8-maas-api-xks-ca" pod "d5f79f3d-1edc-4406-94bc-0785f51d9cb8" (UID: "d5f79f3d-1edc-4406-94bc-0785f51d9cb8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:32:02.073390 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:02.073357 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b8ba874-e544-43e0-a229-3fb9c78f57e8-kube-api-access-fd22z" pod "2b8ba874-e544-43e0-a229-3fb9c78f57e8" (UID: "2b8ba874-e544-43e0-a229-3fb9c78f57e8"). InnerVolumeSpecName "kube-api-access-fd22z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:32:02.172429 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:02.172395 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fd22z\" (UniqueName: \"kubernetes.io/projected/2b8ba874-e544-43e0-a229-3fb9c78f57e8-kube-api-access-fd22z\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:02.172429 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:02.172422 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b8ba874-e544-43e0-a229-3fb9c78f57e8-maas-api-openshift-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:02.172429 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:02.172440 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b8ba874-e544-43e0-a229-3fb9c78f57e8-maas-api-xks-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:02.172429 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:02.172450 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vbqd\" (UniqueName: \"kubernetes.io/projected/d5f79f3d-1edc-4406-94bc-0785f51d9cb8-kube-api-access-6vbqd\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:02.172704 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:02.172460 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5f79f3d-1edc-4406-94bc-0785f51d9cb8-maas-api-xks-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:02.836054 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:02.836030 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-701a0bb8-l4ld8" Oct 31 00:32:02.836054 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:02.836038 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-701a0bb8-l4ld8" event={"ID":"d5f79f3d-1edc-4406-94bc-0785f51d9cb8","Type":"ContainerDied","Data":"fa5330f72abceb43c2b89916621069335a47b94dce00614e4b5bd4a0af99ea44"} Oct 31 00:32:02.836633 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:02.836090 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fa5330f72abceb43c2b89916621069335a47b94dce00614e4b5bd4a0af99ea44" Oct 31 00:32:02.837569 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:02.837551 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37870f8f-9mrhh" Oct 31 00:32:02.837663 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:02.837610 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37870f8f-9mrhh" event={"ID":"2b8ba874-e544-43e0-a229-3fb9c78f57e8","Type":"ContainerDied","Data":"3ace98d367dd34c85c0e9dd93fc6929ec7aa4a7d53a630eaead3a2934200ce2f"} Oct 31 00:32:02.837713 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:02.837661 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3ace98d367dd34c85c0e9dd93fc6929ec7aa4a7d53a630eaead3a2934200ce2f" Oct 31 00:32:02.970521 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:02.970494 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b5d5936-5dgtj" Oct 31 00:32:03.082773 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:03.082738 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37a9c8d6-0dc8-485a-82e5-ef26bb3785a0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37a9c8d6-0dc8-485a-82e5-ef26bb3785a0-maas-api-openshift-ca\") pod \"37a9c8d6-0dc8-485a-82e5-ef26bb3785a0\" (UID: \"37a9c8d6-0dc8-485a-82e5-ef26bb3785a0\") " Oct 31 00:32:03.082954 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:03.082783 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37a9c8d6-0dc8-485a-82e5-ef26bb3785a0-kube-api-access-7pglj\" (UniqueName: \"kubernetes.io/projected/37a9c8d6-0dc8-485a-82e5-ef26bb3785a0-kube-api-access-7pglj\") pod \"37a9c8d6-0dc8-485a-82e5-ef26bb3785a0\" (UID: \"37a9c8d6-0dc8-485a-82e5-ef26bb3785a0\") " Oct 31 00:32:03.082954 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:03.082807 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37a9c8d6-0dc8-485a-82e5-ef26bb3785a0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37a9c8d6-0dc8-485a-82e5-ef26bb3785a0-maas-api-xks-ca\") pod \"37a9c8d6-0dc8-485a-82e5-ef26bb3785a0\" (UID: \"37a9c8d6-0dc8-485a-82e5-ef26bb3785a0\") " Oct 31 00:32:03.083137 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:03.083113 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37a9c8d6-0dc8-485a-82e5-ef26bb3785a0-maas-api-openshift-ca" pod "37a9c8d6-0dc8-485a-82e5-ef26bb3785a0" (UID: "37a9c8d6-0dc8-485a-82e5-ef26bb3785a0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:32:03.085617 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:03.085583 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37a9c8d6-0dc8-485a-82e5-ef26bb3785a0-kube-api-access-7pglj" pod "37a9c8d6-0dc8-485a-82e5-ef26bb3785a0" (UID: "37a9c8d6-0dc8-485a-82e5-ef26bb3785a0"). InnerVolumeSpecName "kube-api-access-7pglj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:32:03.085617 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:03.085590 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37a9c8d6-0dc8-485a-82e5-ef26bb3785a0-maas-api-xks-ca" pod "37a9c8d6-0dc8-485a-82e5-ef26bb3785a0" (UID: "37a9c8d6-0dc8-485a-82e5-ef26bb3785a0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:32:03.184071 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:03.184031 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37a9c8d6-0dc8-485a-82e5-ef26bb3785a0-maas-api-openshift-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:03.184071 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:03.184065 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pglj\" (UniqueName: \"kubernetes.io/projected/37a9c8d6-0dc8-485a-82e5-ef26bb3785a0-kube-api-access-7pglj\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:03.184071 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:03.184075 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37a9c8d6-0dc8-485a-82e5-ef26bb3785a0-maas-api-xks-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:03.563207 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:03.563166 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4b13d6e-5mvsk"] Oct 31 00:32:03.563924 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:03.563907 5222 cpu_manager.go:414] "containerMap: removing container" podUID="d5f79f3d-1edc-4406-94bc-0785f51d9cb8" containerName="revoke-keys" Oct 31 00:32:03.563976 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:03.563928 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5f79f3d-1edc-4406-94bc-0785f51d9cb8" containerName="revoke-keys" Oct 31 00:32:03.564011 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:03.563982 5222 cpu_manager.go:414] "containerMap: removing container" podUID="2b8ba874-e544-43e0-a229-3fb9c78f57e8" containerName="revoke-keys" Oct 31 00:32:03.564011 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:03.563992 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b8ba874-e544-43e0-a229-3fb9c78f57e8" containerName="revoke-keys" Oct 31 00:32:03.564078 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:03.564017 5222 cpu_manager.go:414] "containerMap: removing container" podUID="37a9c8d6-0dc8-485a-82e5-ef26bb3785a0" containerName="revoke-keys" Oct 31 00:32:03.564078 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:03.564026 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37a9c8d6-0dc8-485a-82e5-ef26bb3785a0" containerName="revoke-keys" Oct 31 00:32:03.564155 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:03.564144 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5f79f3d-1edc-4406-94bc-0785f51d9cb8" containerName="revoke-keys" Oct 31 00:32:03.564188 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:03.564164 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b8ba874-e544-43e0-a229-3fb9c78f57e8" containerName="revoke-keys" Oct 31 00:32:03.564188 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:03.564177 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="37a9c8d6-0dc8-485a-82e5-ef26bb3785a0" containerName="revoke-keys" Oct 31 00:32:03.568131 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:03.568106 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4b13d6e-5mvsk" Oct 31 00:32:03.587250 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:03.587214 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d25ee3cb-5017-4649-8f50-c9527f4c9dd9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f4b13d6e-5mvsk\" (UID: \"d25ee3cb-5017-4649-8f50-c9527f4c9dd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4b13d6e-5mvsk" Oct 31 00:32:03.587440 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:03.587307 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfmx9\" (UniqueName: \"kubernetes.io/projected/d25ee3cb-5017-4649-8f50-c9527f4c9dd9-kube-api-access-vfmx9\") pod \"maas-api-revoke-sub-f4b13d6e-5mvsk\" (UID: \"d25ee3cb-5017-4649-8f50-c9527f4c9dd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4b13d6e-5mvsk" Oct 31 00:32:03.587601 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:03.587572 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d25ee3cb-5017-4649-8f50-c9527f4c9dd9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f4b13d6e-5mvsk\" (UID: \"d25ee3cb-5017-4649-8f50-c9527f4c9dd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4b13d6e-5mvsk" Oct 31 00:32:03.623175 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:03.622527 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4b13d6e-5mvsk"] Oct 31 00:32:03.688516 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:03.688474 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d25ee3cb-5017-4649-8f50-c9527f4c9dd9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f4b13d6e-5mvsk\" (UID: \"d25ee3cb-5017-4649-8f50-c9527f4c9dd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4b13d6e-5mvsk" Oct 31 00:32:03.688709 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:03.688557 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d25ee3cb-5017-4649-8f50-c9527f4c9dd9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f4b13d6e-5mvsk\" (UID: \"d25ee3cb-5017-4649-8f50-c9527f4c9dd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4b13d6e-5mvsk" Oct 31 00:32:03.688709 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:03.688650 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfmx9\" (UniqueName: \"kubernetes.io/projected/d25ee3cb-5017-4649-8f50-c9527f4c9dd9-kube-api-access-vfmx9\") pod \"maas-api-revoke-sub-f4b13d6e-5mvsk\" (UID: \"d25ee3cb-5017-4649-8f50-c9527f4c9dd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4b13d6e-5mvsk" Oct 31 00:32:03.689191 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:03.689170 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d25ee3cb-5017-4649-8f50-c9527f4c9dd9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f4b13d6e-5mvsk\" (UID: \"d25ee3cb-5017-4649-8f50-c9527f4c9dd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4b13d6e-5mvsk" Oct 31 00:32:03.691000 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:03.690978 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d25ee3cb-5017-4649-8f50-c9527f4c9dd9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f4b13d6e-5mvsk\" (UID: \"d25ee3cb-5017-4649-8f50-c9527f4c9dd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4b13d6e-5mvsk" Oct 31 00:32:03.752167 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:03.752126 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfmx9\" (UniqueName: \"kubernetes.io/projected/d25ee3cb-5017-4649-8f50-c9527f4c9dd9-kube-api-access-vfmx9\") pod \"maas-api-revoke-sub-f4b13d6e-5mvsk\" (UID: \"d25ee3cb-5017-4649-8f50-c9527f4c9dd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4b13d6e-5mvsk" Oct 31 00:32:03.844800 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:03.844722 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b5d5936-5dgtj" Oct 31 00:32:03.844800 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:03.844761 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b5d5936-5dgtj" event={"ID":"37a9c8d6-0dc8-485a-82e5-ef26bb3785a0","Type":"ContainerDied","Data":"5e8116d1c7f17e50edd41fc943420d75832ea60ab0a0285c88434e10d2972b84"} Oct 31 00:32:03.845228 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:03.844811 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5e8116d1c7f17e50edd41fc943420d75832ea60ab0a0285c88434e10d2972b84" Oct 31 00:32:03.882038 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:03.882002 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4b13d6e-5mvsk" Oct 31 00:32:04.261643 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:32:04.261611 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd25ee3cb_5017_4649_8f50_c9527f4c9dd9.slice/crio-f2dd2b2a176d3d0b7aadc71c99fa8d005d301ec39a7a77275f6fdac299282aa3 WatchSource:0}: Error finding container f2dd2b2a176d3d0b7aadc71c99fa8d005d301ec39a7a77275f6fdac299282aa3: Status 404 returned error can't find the container with id f2dd2b2a176d3d0b7aadc71c99fa8d005d301ec39a7a77275f6fdac299282aa3 Oct 31 00:32:04.265051 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:04.265023 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4b13d6e-5mvsk"] Oct 31 00:32:04.852903 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:04.852803 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4b13d6e-5mvsk" event={"ID":"d25ee3cb-5017-4649-8f50-c9527f4c9dd9","Type":"ContainerStarted","Data":"4ffc3baf8b66849b9896e0bf196049946e39462e71295d5ea20095a85f3fa6d2"} Oct 31 00:32:04.852903 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:04.852860 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4b13d6e-5mvsk" event={"ID":"d25ee3cb-5017-4649-8f50-c9527f4c9dd9","Type":"ContainerStarted","Data":"f2dd2b2a176d3d0b7aadc71c99fa8d005d301ec39a7a77275f6fdac299282aa3"} Oct 31 00:32:04.973518 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:04.973456 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4b13d6e-5mvsk" podStartSLOduration=1.973437909 podStartE2EDuration="1.973437909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:32:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:32:04.971207918 +0000 UTC m=+1017.442377668" watchObservedRunningTime="2026-10-31 00:32:04.973437909 +0000 UTC m=+1017.444607657" Oct 31 00:32:05.860525 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:05.860485 5222 generic.go:356] "Generic (PLEG): container finished" podID="d25ee3cb-5017-4649-8f50-c9527f4c9dd9" containerID="4ffc3baf8b66849b9896e0bf196049946e39462e71295d5ea20095a85f3fa6d2" exitCode=0 Oct 31 00:32:05.860968 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:05.860568 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4b13d6e-5mvsk" event={"ID":"d25ee3cb-5017-4649-8f50-c9527f4c9dd9","Type":"ContainerDied","Data":"4ffc3baf8b66849b9896e0bf196049946e39462e71295d5ea20095a85f3fa6d2"} Oct 31 00:32:05.889169 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:32:05.889128 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 00:32:07.009494 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:07.009469 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4b13d6e-5mvsk" Oct 31 00:32:07.127584 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:07.127507 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d25ee3cb-5017-4649-8f50-c9527f4c9dd9-kube-api-access-vfmx9\" (UniqueName: \"kubernetes.io/projected/d25ee3cb-5017-4649-8f50-c9527f4c9dd9-kube-api-access-vfmx9\") pod \"d25ee3cb-5017-4649-8f50-c9527f4c9dd9\" (UID: \"d25ee3cb-5017-4649-8f50-c9527f4c9dd9\") " Oct 31 00:32:07.127584 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:07.127559 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d25ee3cb-5017-4649-8f50-c9527f4c9dd9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d25ee3cb-5017-4649-8f50-c9527f4c9dd9-maas-api-openshift-ca\") pod \"d25ee3cb-5017-4649-8f50-c9527f4c9dd9\" (UID: \"d25ee3cb-5017-4649-8f50-c9527f4c9dd9\") " Oct 31 00:32:07.127791 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:07.127747 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d25ee3cb-5017-4649-8f50-c9527f4c9dd9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d25ee3cb-5017-4649-8f50-c9527f4c9dd9-maas-api-xks-ca\") pod \"d25ee3cb-5017-4649-8f50-c9527f4c9dd9\" (UID: \"d25ee3cb-5017-4649-8f50-c9527f4c9dd9\") " Oct 31 00:32:07.127920 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:07.127900 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d25ee3cb-5017-4649-8f50-c9527f4c9dd9-maas-api-openshift-ca" pod "d25ee3cb-5017-4649-8f50-c9527f4c9dd9" (UID: "d25ee3cb-5017-4649-8f50-c9527f4c9dd9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:32:07.130980 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:07.128394 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d25ee3cb-5017-4649-8f50-c9527f4c9dd9-maas-api-openshift-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:07.130980 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:07.130241 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d25ee3cb-5017-4649-8f50-c9527f4c9dd9-maas-api-xks-ca" pod "d25ee3cb-5017-4649-8f50-c9527f4c9dd9" (UID: "d25ee3cb-5017-4649-8f50-c9527f4c9dd9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:32:07.130980 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:07.130271 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d25ee3cb-5017-4649-8f50-c9527f4c9dd9-kube-api-access-vfmx9" pod "d25ee3cb-5017-4649-8f50-c9527f4c9dd9" (UID: "d25ee3cb-5017-4649-8f50-c9527f4c9dd9"). InnerVolumeSpecName "kube-api-access-vfmx9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:32:07.229978 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:07.229938 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfmx9\" (UniqueName: \"kubernetes.io/projected/d25ee3cb-5017-4649-8f50-c9527f4c9dd9-kube-api-access-vfmx9\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:07.229978 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:07.229974 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d25ee3cb-5017-4649-8f50-c9527f4c9dd9-maas-api-xks-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:07.873593 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:07.873563 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4b13d6e-5mvsk" Oct 31 00:32:07.873764 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:07.873603 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4b13d6e-5mvsk" event={"ID":"d25ee3cb-5017-4649-8f50-c9527f4c9dd9","Type":"ContainerDied","Data":"f2dd2b2a176d3d0b7aadc71c99fa8d005d301ec39a7a77275f6fdac299282aa3"} Oct 31 00:32:07.873764 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:07.873680 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f2dd2b2a176d3d0b7aadc71c99fa8d005d301ec39a7a77275f6fdac299282aa3" Oct 31 00:32:09.942538 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:09.942503 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-79tk8"] Oct 31 00:32:10.463556 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:10.463514 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52695c5b-btcq4"] Oct 31 00:32:10.464323 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:10.464301 5222 cpu_manager.go:414] "containerMap: removing container" podUID="d25ee3cb-5017-4649-8f50-c9527f4c9dd9" containerName="revoke-keys" Oct 31 00:32:10.464385 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:10.464329 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d25ee3cb-5017-4649-8f50-c9527f4c9dd9" containerName="revoke-keys" Oct 31 00:32:10.464506 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:10.464491 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="d25ee3cb-5017-4649-8f50-c9527f4c9dd9" containerName="revoke-keys" Oct 31 00:32:10.469876 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:10.469848 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52695c5b-btcq4" Oct 31 00:32:10.580464 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:10.573644 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:32:10.580464 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:10.573725 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:32:10.580464 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:10.573813 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:32:10.580464 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:10.574492 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ab68e7d-5046-4ed2-9ec7-f8270d3f478b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52695c5b-btcq4\" (UID: \"8ab68e7d-5046-4ed2-9ec7-f8270d3f478b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52695c5b-btcq4" Oct 31 00:32:10.580464 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:10.574594 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ab68e7d-5046-4ed2-9ec7-f8270d3f478b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52695c5b-btcq4\" (UID: \"8ab68e7d-5046-4ed2-9ec7-f8270d3f478b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52695c5b-btcq4" Oct 31 00:32:10.580464 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:10.574631 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgjg8\" (UniqueName: \"kubernetes.io/projected/8ab68e7d-5046-4ed2-9ec7-f8270d3f478b-kube-api-access-rgjg8\") pod \"maas-api-revoke-sub-52695c5b-btcq4\" (UID: \"8ab68e7d-5046-4ed2-9ec7-f8270d3f478b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52695c5b-btcq4" Oct 31 00:32:10.580464 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:10.578544 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52695c5b-btcq4"] Oct 31 00:32:10.615751 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:10.615681 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfz9z\"" Oct 31 00:32:10.676396 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:10.675794 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ab68e7d-5046-4ed2-9ec7-f8270d3f478b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52695c5b-btcq4\" (UID: \"8ab68e7d-5046-4ed2-9ec7-f8270d3f478b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52695c5b-btcq4" Oct 31 00:32:10.676396 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:10.675975 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ab68e7d-5046-4ed2-9ec7-f8270d3f478b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52695c5b-btcq4\" (UID: \"8ab68e7d-5046-4ed2-9ec7-f8270d3f478b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52695c5b-btcq4" Oct 31 00:32:10.677576 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:10.677267 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ab68e7d-5046-4ed2-9ec7-f8270d3f478b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52695c5b-btcq4\" (UID: \"8ab68e7d-5046-4ed2-9ec7-f8270d3f478b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52695c5b-btcq4" Oct 31 00:32:10.677576 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:10.677324 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgjg8\" (UniqueName: \"kubernetes.io/projected/8ab68e7d-5046-4ed2-9ec7-f8270d3f478b-kube-api-access-rgjg8\") pod \"maas-api-revoke-sub-52695c5b-btcq4\" (UID: \"8ab68e7d-5046-4ed2-9ec7-f8270d3f478b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52695c5b-btcq4" Oct 31 00:32:10.686961 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:10.686899 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ab68e7d-5046-4ed2-9ec7-f8270d3f478b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52695c5b-btcq4\" (UID: \"8ab68e7d-5046-4ed2-9ec7-f8270d3f478b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52695c5b-btcq4" Oct 31 00:32:10.743447 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:10.739400 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgjg8\" (UniqueName: \"kubernetes.io/projected/8ab68e7d-5046-4ed2-9ec7-f8270d3f478b-kube-api-access-rgjg8\") pod \"maas-api-revoke-sub-52695c5b-btcq4\" (UID: \"8ab68e7d-5046-4ed2-9ec7-f8270d3f478b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52695c5b-btcq4" Oct 31 00:32:10.785394 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:10.784898 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52695c5b-btcq4" Oct 31 00:32:11.251515 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:32:11.251470 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ab68e7d_5046_4ed2_9ec7_f8270d3f478b.slice/crio-bcd76795cccf07f8a2b3460f37e70dcb9665c4f498bbf9c280f758d430198518 WatchSource:0}: Error finding container bcd76795cccf07f8a2b3460f37e70dcb9665c4f498bbf9c280f758d430198518: Status 404 returned error can't find the container with id bcd76795cccf07f8a2b3460f37e70dcb9665c4f498bbf9c280f758d430198518 Oct 31 00:32:11.253313 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:11.253289 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52695c5b-btcq4"] Oct 31 00:32:11.919238 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:11.919146 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52695c5b-btcq4" event={"ID":"8ab68e7d-5046-4ed2-9ec7-f8270d3f478b","Type":"ContainerStarted","Data":"ae1fb3a0805b693656a2cab1f3dd851283af6bfaee832b69438f156f74414c12"} Oct 31 00:32:11.919238 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:11.919211 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52695c5b-btcq4" event={"ID":"8ab68e7d-5046-4ed2-9ec7-f8270d3f478b","Type":"ContainerStarted","Data":"bcd76795cccf07f8a2b3460f37e70dcb9665c4f498bbf9c280f758d430198518"} Oct 31 00:32:12.067956 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:12.067900 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52695c5b-btcq4" podStartSLOduration=2.067886794 podStartE2EDuration="2.067886794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:32:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:32:12.066811876 +0000 UTC m=+1024.537981628" watchObservedRunningTime="2026-10-31 00:32:12.067886794 +0000 UTC m=+1024.539056542" Oct 31 00:32:12.927261 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:12.927227 5222 generic.go:356] "Generic (PLEG): container finished" podID="8ab68e7d-5046-4ed2-9ec7-f8270d3f478b" containerID="ae1fb3a0805b693656a2cab1f3dd851283af6bfaee832b69438f156f74414c12" exitCode=0 Oct 31 00:32:12.927753 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:12.927296 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52695c5b-btcq4" event={"ID":"8ab68e7d-5046-4ed2-9ec7-f8270d3f478b","Type":"ContainerDied","Data":"ae1fb3a0805b693656a2cab1f3dd851283af6bfaee832b69438f156f74414c12"} Oct 31 00:32:14.076985 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:14.076957 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52695c5b-btcq4" Oct 31 00:32:14.221299 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:14.221198 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ab68e7d-5046-4ed2-9ec7-f8270d3f478b-kube-api-access-rgjg8\" (UniqueName: \"kubernetes.io/projected/8ab68e7d-5046-4ed2-9ec7-f8270d3f478b-kube-api-access-rgjg8\") pod \"8ab68e7d-5046-4ed2-9ec7-f8270d3f478b\" (UID: \"8ab68e7d-5046-4ed2-9ec7-f8270d3f478b\") " Oct 31 00:32:14.221490 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:14.221301 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ab68e7d-5046-4ed2-9ec7-f8270d3f478b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ab68e7d-5046-4ed2-9ec7-f8270d3f478b-maas-api-xks-ca\") pod \"8ab68e7d-5046-4ed2-9ec7-f8270d3f478b\" (UID: \"8ab68e7d-5046-4ed2-9ec7-f8270d3f478b\") " Oct 31 00:32:14.221490 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:14.221355 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ab68e7d-5046-4ed2-9ec7-f8270d3f478b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ab68e7d-5046-4ed2-9ec7-f8270d3f478b-maas-api-openshift-ca\") pod \"8ab68e7d-5046-4ed2-9ec7-f8270d3f478b\" (UID: \"8ab68e7d-5046-4ed2-9ec7-f8270d3f478b\") " Oct 31 00:32:14.221788 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:14.221754 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ab68e7d-5046-4ed2-9ec7-f8270d3f478b-maas-api-openshift-ca" pod "8ab68e7d-5046-4ed2-9ec7-f8270d3f478b" (UID: "8ab68e7d-5046-4ed2-9ec7-f8270d3f478b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:32:14.221948 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:14.221916 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ab68e7d-5046-4ed2-9ec7-f8270d3f478b-maas-api-openshift-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:14.223688 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:14.223658 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ab68e7d-5046-4ed2-9ec7-f8270d3f478b-kube-api-access-rgjg8" pod "8ab68e7d-5046-4ed2-9ec7-f8270d3f478b" (UID: "8ab68e7d-5046-4ed2-9ec7-f8270d3f478b"). InnerVolumeSpecName "kube-api-access-rgjg8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:32:14.223771 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:14.223680 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ab68e7d-5046-4ed2-9ec7-f8270d3f478b-maas-api-xks-ca" pod "8ab68e7d-5046-4ed2-9ec7-f8270d3f478b" (UID: "8ab68e7d-5046-4ed2-9ec7-f8270d3f478b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:32:14.322935 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:14.322893 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgjg8\" (UniqueName: \"kubernetes.io/projected/8ab68e7d-5046-4ed2-9ec7-f8270d3f478b-kube-api-access-rgjg8\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:14.322935 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:14.322929 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ab68e7d-5046-4ed2-9ec7-f8270d3f478b-maas-api-xks-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:14.940569 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:14.940532 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52695c5b-btcq4" event={"ID":"8ab68e7d-5046-4ed2-9ec7-f8270d3f478b","Type":"ContainerDied","Data":"bcd76795cccf07f8a2b3460f37e70dcb9665c4f498bbf9c280f758d430198518"} Oct 31 00:32:14.940569 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:14.940565 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52695c5b-btcq4" Oct 31 00:32:14.940788 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:14.940590 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bcd76795cccf07f8a2b3460f37e70dcb9665c4f498bbf9c280f758d430198518" Oct 31 00:32:15.713290 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:15.713252 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1019bddd-hdvj8"] Oct 31 00:32:15.713974 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:15.713956 5222 cpu_manager.go:414] "containerMap: removing container" podUID="8ab68e7d-5046-4ed2-9ec7-f8270d3f478b" containerName="revoke-keys" Oct 31 00:32:15.714018 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:15.713977 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ab68e7d-5046-4ed2-9ec7-f8270d3f478b" containerName="revoke-keys" Oct 31 00:32:15.714157 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:15.714147 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ab68e7d-5046-4ed2-9ec7-f8270d3f478b" containerName="revoke-keys" Oct 31 00:32:15.718203 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:15.718185 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1019bddd-hdvj8" Oct 31 00:32:15.769202 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:15.769154 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1019bddd-hdvj8"] Oct 31 00:32:15.772248 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:15.772219 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:32:15.772397 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:15.772270 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:32:15.817353 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:15.817321 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:32:15.817542 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:15.817397 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfz9z\"" Oct 31 00:32:15.837733 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:15.837698 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dfec126-5c72-4be9-ad1f-c605c63bea90-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1019bddd-hdvj8\" (UID: \"6dfec126-5c72-4be9-ad1f-c605c63bea90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1019bddd-hdvj8" Oct 31 00:32:15.837894 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:15.837767 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dfec126-5c72-4be9-ad1f-c605c63bea90-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1019bddd-hdvj8\" (UID: \"6dfec126-5c72-4be9-ad1f-c605c63bea90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1019bddd-hdvj8" Oct 31 00:32:15.837959 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:15.837908 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzslp\" (UniqueName: \"kubernetes.io/projected/6dfec126-5c72-4be9-ad1f-c605c63bea90-kube-api-access-hzslp\") pod \"maas-api-revoke-sub-1019bddd-hdvj8\" (UID: \"6dfec126-5c72-4be9-ad1f-c605c63bea90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1019bddd-hdvj8" Oct 31 00:32:15.939431 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:15.939357 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dfec126-5c72-4be9-ad1f-c605c63bea90-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1019bddd-hdvj8\" (UID: \"6dfec126-5c72-4be9-ad1f-c605c63bea90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1019bddd-hdvj8" Oct 31 00:32:15.939641 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:15.939493 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dfec126-5c72-4be9-ad1f-c605c63bea90-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1019bddd-hdvj8\" (UID: \"6dfec126-5c72-4be9-ad1f-c605c63bea90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1019bddd-hdvj8" Oct 31 00:32:15.939641 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:15.939568 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzslp\" (UniqueName: \"kubernetes.io/projected/6dfec126-5c72-4be9-ad1f-c605c63bea90-kube-api-access-hzslp\") pod \"maas-api-revoke-sub-1019bddd-hdvj8\" (UID: \"6dfec126-5c72-4be9-ad1f-c605c63bea90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1019bddd-hdvj8" Oct 31 00:32:15.940270 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:15.940235 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dfec126-5c72-4be9-ad1f-c605c63bea90-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1019bddd-hdvj8\" (UID: \"6dfec126-5c72-4be9-ad1f-c605c63bea90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1019bddd-hdvj8" Oct 31 00:32:15.942003 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:15.941975 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dfec126-5c72-4be9-ad1f-c605c63bea90-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1019bddd-hdvj8\" (UID: \"6dfec126-5c72-4be9-ad1f-c605c63bea90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1019bddd-hdvj8" Oct 31 00:32:16.002155 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:16.002078 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzslp\" (UniqueName: \"kubernetes.io/projected/6dfec126-5c72-4be9-ad1f-c605c63bea90-kube-api-access-hzslp\") pod \"maas-api-revoke-sub-1019bddd-hdvj8\" (UID: \"6dfec126-5c72-4be9-ad1f-c605c63bea90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1019bddd-hdvj8" Oct 31 00:32:16.033239 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:16.033202 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1019bddd-hdvj8" Oct 31 00:32:16.081077 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:16.081043 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da56edec-wglq9"] Oct 31 00:32:16.086855 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:16.086825 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da56edec-wglq9" Oct 31 00:32:16.137940 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:16.137907 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da56edec-wglq9"] Oct 31 00:32:16.242245 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:16.242211 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98010426-9a98-407e-ad55-7eeb3313b555-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da56edec-wglq9\" (UID: \"98010426-9a98-407e-ad55-7eeb3313b555\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da56edec-wglq9" Oct 31 00:32:16.242422 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:16.242364 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvzw5\" (UniqueName: \"kubernetes.io/projected/98010426-9a98-407e-ad55-7eeb3313b555-kube-api-access-nvzw5\") pod \"maas-api-revoke-sub-da56edec-wglq9\" (UID: \"98010426-9a98-407e-ad55-7eeb3313b555\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da56edec-wglq9" Oct 31 00:32:16.242484 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:16.242469 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98010426-9a98-407e-ad55-7eeb3313b555-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da56edec-wglq9\" (UID: \"98010426-9a98-407e-ad55-7eeb3313b555\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da56edec-wglq9" Oct 31 00:32:16.343865 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:16.343822 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvzw5\" (UniqueName: \"kubernetes.io/projected/98010426-9a98-407e-ad55-7eeb3313b555-kube-api-access-nvzw5\") pod \"maas-api-revoke-sub-da56edec-wglq9\" (UID: \"98010426-9a98-407e-ad55-7eeb3313b555\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da56edec-wglq9" Oct 31 00:32:16.343865 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:16.343869 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98010426-9a98-407e-ad55-7eeb3313b555-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da56edec-wglq9\" (UID: \"98010426-9a98-407e-ad55-7eeb3313b555\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da56edec-wglq9" Oct 31 00:32:16.344147 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:16.343942 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98010426-9a98-407e-ad55-7eeb3313b555-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da56edec-wglq9\" (UID: \"98010426-9a98-407e-ad55-7eeb3313b555\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da56edec-wglq9" Oct 31 00:32:16.344755 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:16.344724 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98010426-9a98-407e-ad55-7eeb3313b555-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da56edec-wglq9\" (UID: \"98010426-9a98-407e-ad55-7eeb3313b555\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da56edec-wglq9" Oct 31 00:32:16.346971 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:16.346949 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98010426-9a98-407e-ad55-7eeb3313b555-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da56edec-wglq9\" (UID: \"98010426-9a98-407e-ad55-7eeb3313b555\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da56edec-wglq9" Oct 31 00:32:16.407048 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:16.407019 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvzw5\" (UniqueName: \"kubernetes.io/projected/98010426-9a98-407e-ad55-7eeb3313b555-kube-api-access-nvzw5\") pod \"maas-api-revoke-sub-da56edec-wglq9\" (UID: \"98010426-9a98-407e-ad55-7eeb3313b555\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da56edec-wglq9" Oct 31 00:32:16.412561 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:32:16.412528 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dfec126_5c72_4be9_ad1f_c605c63bea90.slice/crio-6370bf76eaf98e3b7d5897d60265b0c999dbefb4cc1efc225424e09f2bd86a17 WatchSource:0}: Error finding container 6370bf76eaf98e3b7d5897d60265b0c999dbefb4cc1efc225424e09f2bd86a17: Status 404 returned error can't find the container with id 6370bf76eaf98e3b7d5897d60265b0c999dbefb4cc1efc225424e09f2bd86a17 Oct 31 00:32:16.416211 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:16.416186 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1019bddd-hdvj8"] Oct 31 00:32:16.707627 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:16.707591 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da56edec-wglq9" Oct 31 00:32:16.954405 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:16.954301 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1019bddd-hdvj8" event={"ID":"6dfec126-5c72-4be9-ad1f-c605c63bea90","Type":"ContainerStarted","Data":"5baba50f3c9b6095727232636feab713930ac0d1e31e8adde43d0baac0399c7b"} Oct 31 00:32:16.954405 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:16.954382 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1019bddd-hdvj8" event={"ID":"6dfec126-5c72-4be9-ad1f-c605c63bea90","Type":"ContainerStarted","Data":"6370bf76eaf98e3b7d5897d60265b0c999dbefb4cc1efc225424e09f2bd86a17"} Oct 31 00:32:17.076317 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:17.076259 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1019bddd-hdvj8" podStartSLOduration=2.076244083 podStartE2EDuration="2.076244083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:32:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:32:17.072928952 +0000 UTC m=+1029.544098701" watchObservedRunningTime="2026-10-31 00:32:17.076244083 +0000 UTC m=+1029.547413831" Oct 31 00:32:17.088087 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:32:17.088052 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98010426_9a98_407e_ad55_7eeb3313b555.slice/crio-92ff205049c793da4bb3eeb35124a081c12a06efc0e2aacc68d62af6f1563291 WatchSource:0}: Error finding container 92ff205049c793da4bb3eeb35124a081c12a06efc0e2aacc68d62af6f1563291: Status 404 returned error can't find the container with id 92ff205049c793da4bb3eeb35124a081c12a06efc0e2aacc68d62af6f1563291 Oct 31 00:32:17.092144 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:17.092097 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da56edec-wglq9"] Oct 31 00:32:17.962043 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:17.962013 5222 generic.go:356] "Generic (PLEG): container finished" podID="98010426-9a98-407e-ad55-7eeb3313b555" containerID="0b5ce50f468b6eda9f9ea7b6ddb322e82a65ab5897aa9dc265225db5758d8a22" exitCode=0 Oct 31 00:32:17.962489 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:17.962106 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da56edec-wglq9" event={"ID":"98010426-9a98-407e-ad55-7eeb3313b555","Type":"ContainerDied","Data":"0b5ce50f468b6eda9f9ea7b6ddb322e82a65ab5897aa9dc265225db5758d8a22"} Oct 31 00:32:17.962489 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:17.962165 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da56edec-wglq9" event={"ID":"98010426-9a98-407e-ad55-7eeb3313b555","Type":"ContainerStarted","Data":"92ff205049c793da4bb3eeb35124a081c12a06efc0e2aacc68d62af6f1563291"} Oct 31 00:32:17.963641 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:17.963617 5222 generic.go:356] "Generic (PLEG): container finished" podID="6dfec126-5c72-4be9-ad1f-c605c63bea90" containerID="5baba50f3c9b6095727232636feab713930ac0d1e31e8adde43d0baac0399c7b" exitCode=0 Oct 31 00:32:17.963784 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:17.963700 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1019bddd-hdvj8" event={"ID":"6dfec126-5c72-4be9-ad1f-c605c63bea90","Type":"ContainerDied","Data":"5baba50f3c9b6095727232636feab713930ac0d1e31e8adde43d0baac0399c7b"} Oct 31 00:32:19.143175 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:19.143145 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da56edec-wglq9" Oct 31 00:32:19.146811 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:19.146788 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1019bddd-hdvj8" Oct 31 00:32:19.277271 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:19.277241 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/98010426-9a98-407e-ad55-7eeb3313b555-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98010426-9a98-407e-ad55-7eeb3313b555-maas-api-xks-ca\") pod \"98010426-9a98-407e-ad55-7eeb3313b555\" (UID: \"98010426-9a98-407e-ad55-7eeb3313b555\") " Oct 31 00:32:19.277486 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:19.277300 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98010426-9a98-407e-ad55-7eeb3313b555-kube-api-access-nvzw5\" (UniqueName: \"kubernetes.io/projected/98010426-9a98-407e-ad55-7eeb3313b555-kube-api-access-nvzw5\") pod \"98010426-9a98-407e-ad55-7eeb3313b555\" (UID: \"98010426-9a98-407e-ad55-7eeb3313b555\") " Oct 31 00:32:19.277486 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:19.277342 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6dfec126-5c72-4be9-ad1f-c605c63bea90-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dfec126-5c72-4be9-ad1f-c605c63bea90-maas-api-openshift-ca\") pod \"6dfec126-5c72-4be9-ad1f-c605c63bea90\" (UID: \"6dfec126-5c72-4be9-ad1f-c605c63bea90\") " Oct 31 00:32:19.277486 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:19.277389 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6dfec126-5c72-4be9-ad1f-c605c63bea90-kube-api-access-hzslp\" (UniqueName: \"kubernetes.io/projected/6dfec126-5c72-4be9-ad1f-c605c63bea90-kube-api-access-hzslp\") pod \"6dfec126-5c72-4be9-ad1f-c605c63bea90\" (UID: \"6dfec126-5c72-4be9-ad1f-c605c63bea90\") " Oct 31 00:32:19.277652 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:19.277529 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6dfec126-5c72-4be9-ad1f-c605c63bea90-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dfec126-5c72-4be9-ad1f-c605c63bea90-maas-api-xks-ca\") pod \"6dfec126-5c72-4be9-ad1f-c605c63bea90\" (UID: \"6dfec126-5c72-4be9-ad1f-c605c63bea90\") " Oct 31 00:32:19.277652 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:19.277610 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98010426-9a98-407e-ad55-7eeb3313b555-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98010426-9a98-407e-ad55-7eeb3313b555-maas-api-openshift-ca\") pod \"98010426-9a98-407e-ad55-7eeb3313b555\" (UID: \"98010426-9a98-407e-ad55-7eeb3313b555\") " Oct 31 00:32:19.278566 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:19.278452 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98010426-9a98-407e-ad55-7eeb3313b555-maas-api-openshift-ca" pod "98010426-9a98-407e-ad55-7eeb3313b555" (UID: "98010426-9a98-407e-ad55-7eeb3313b555"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:32:19.279427 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:19.278832 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98010426-9a98-407e-ad55-7eeb3313b555-maas-api-openshift-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:19.279427 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:19.279163 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6dfec126-5c72-4be9-ad1f-c605c63bea90-maas-api-openshift-ca" pod "6dfec126-5c72-4be9-ad1f-c605c63bea90" (UID: "6dfec126-5c72-4be9-ad1f-c605c63bea90"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:32:19.281844 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:19.280748 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6dfec126-5c72-4be9-ad1f-c605c63bea90-maas-api-xks-ca" pod "6dfec126-5c72-4be9-ad1f-c605c63bea90" (UID: "6dfec126-5c72-4be9-ad1f-c605c63bea90"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:32:19.281844 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:19.280765 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98010426-9a98-407e-ad55-7eeb3313b555-maas-api-xks-ca" pod "98010426-9a98-407e-ad55-7eeb3313b555" (UID: "98010426-9a98-407e-ad55-7eeb3313b555"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:32:19.281844 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:19.280788 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dfec126-5c72-4be9-ad1f-c605c63bea90-kube-api-access-hzslp" pod "6dfec126-5c72-4be9-ad1f-c605c63bea90" (UID: "6dfec126-5c72-4be9-ad1f-c605c63bea90"). InnerVolumeSpecName "kube-api-access-hzslp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:32:19.292960 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:19.292918 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98010426-9a98-407e-ad55-7eeb3313b555-kube-api-access-nvzw5" pod "98010426-9a98-407e-ad55-7eeb3313b555" (UID: "98010426-9a98-407e-ad55-7eeb3313b555"). InnerVolumeSpecName "kube-api-access-nvzw5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:32:19.379918 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:19.379879 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98010426-9a98-407e-ad55-7eeb3313b555-maas-api-xks-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:19.379918 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:19.379920 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvzw5\" (UniqueName: \"kubernetes.io/projected/98010426-9a98-407e-ad55-7eeb3313b555-kube-api-access-nvzw5\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:19.380177 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:19.379938 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dfec126-5c72-4be9-ad1f-c605c63bea90-maas-api-openshift-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:19.380177 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:19.379954 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzslp\" (UniqueName: \"kubernetes.io/projected/6dfec126-5c72-4be9-ad1f-c605c63bea90-kube-api-access-hzslp\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:19.380177 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:19.379968 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dfec126-5c72-4be9-ad1f-c605c63bea90-maas-api-xks-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:19.977475 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:19.977391 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da56edec-wglq9" Oct 31 00:32:19.977627 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:19.977359 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da56edec-wglq9" event={"ID":"98010426-9a98-407e-ad55-7eeb3313b555","Type":"ContainerDied","Data":"92ff205049c793da4bb3eeb35124a081c12a06efc0e2aacc68d62af6f1563291"} Oct 31 00:32:19.977627 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:19.977516 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="92ff205049c793da4bb3eeb35124a081c12a06efc0e2aacc68d62af6f1563291" Oct 31 00:32:19.978815 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:19.978795 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1019bddd-hdvj8" Oct 31 00:32:19.978957 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:19.978830 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1019bddd-hdvj8" event={"ID":"6dfec126-5c72-4be9-ad1f-c605c63bea90","Type":"ContainerDied","Data":"6370bf76eaf98e3b7d5897d60265b0c999dbefb4cc1efc225424e09f2bd86a17"} Oct 31 00:32:19.978957 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:19.978867 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6370bf76eaf98e3b7d5897d60265b0c999dbefb4cc1efc225424e09f2bd86a17" Oct 31 00:32:20.142787 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:20.142741 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0a93465-jv7vb"] Oct 31 00:32:20.143232 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:20.143219 5222 cpu_manager.go:414] "containerMap: removing container" podUID="98010426-9a98-407e-ad55-7eeb3313b555" containerName="revoke-keys" Oct 31 00:32:20.143554 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:20.143233 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98010426-9a98-407e-ad55-7eeb3313b555" containerName="revoke-keys" Oct 31 00:32:20.143554 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:20.143244 5222 cpu_manager.go:414] "containerMap: removing container" podUID="6dfec126-5c72-4be9-ad1f-c605c63bea90" containerName="revoke-keys" Oct 31 00:32:20.143554 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:20.143249 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dfec126-5c72-4be9-ad1f-c605c63bea90" containerName="revoke-keys" Oct 31 00:32:20.143554 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:20.143356 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="98010426-9a98-407e-ad55-7eeb3313b555" containerName="revoke-keys" Oct 31 00:32:20.143554 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:20.143364 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="6dfec126-5c72-4be9-ad1f-c605c63bea90" containerName="revoke-keys" Oct 31 00:32:20.146627 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:20.146610 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0a93465-jv7vb" Oct 31 00:32:20.200478 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:20.200444 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:32:20.200675 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:20.200493 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:32:20.200675 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:20.200591 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfz9z\"" Oct 31 00:32:20.200675 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:20.200612 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:32:20.223630 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:20.223596 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0a93465-jv7vb"] Oct 31 00:32:20.290591 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:20.290548 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7h7f\" (UniqueName: \"kubernetes.io/projected/b801f228-4d3a-4233-a4ea-a263bf53a34f-kube-api-access-t7h7f\") pod \"maas-api-revoke-sub-f0a93465-jv7vb\" (UID: \"b801f228-4d3a-4233-a4ea-a263bf53a34f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0a93465-jv7vb" Oct 31 00:32:20.290783 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:20.290712 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b801f228-4d3a-4233-a4ea-a263bf53a34f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0a93465-jv7vb\" (UID: \"b801f228-4d3a-4233-a4ea-a263bf53a34f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0a93465-jv7vb" Oct 31 00:32:20.290783 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:20.290762 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b801f228-4d3a-4233-a4ea-a263bf53a34f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0a93465-jv7vb\" (UID: \"b801f228-4d3a-4233-a4ea-a263bf53a34f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0a93465-jv7vb" Oct 31 00:32:20.391870 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:20.391835 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b801f228-4d3a-4233-a4ea-a263bf53a34f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0a93465-jv7vb\" (UID: \"b801f228-4d3a-4233-a4ea-a263bf53a34f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0a93465-jv7vb" Oct 31 00:32:20.392047 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:20.391892 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b801f228-4d3a-4233-a4ea-a263bf53a34f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0a93465-jv7vb\" (UID: \"b801f228-4d3a-4233-a4ea-a263bf53a34f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0a93465-jv7vb" Oct 31 00:32:20.392047 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:20.391964 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7h7f\" (UniqueName: \"kubernetes.io/projected/b801f228-4d3a-4233-a4ea-a263bf53a34f-kube-api-access-t7h7f\") pod \"maas-api-revoke-sub-f0a93465-jv7vb\" (UID: \"b801f228-4d3a-4233-a4ea-a263bf53a34f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0a93465-jv7vb" Oct 31 00:32:20.392650 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:20.392626 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b801f228-4d3a-4233-a4ea-a263bf53a34f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0a93465-jv7vb\" (UID: \"b801f228-4d3a-4233-a4ea-a263bf53a34f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0a93465-jv7vb" Oct 31 00:32:20.394414 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:20.394364 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b801f228-4d3a-4233-a4ea-a263bf53a34f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0a93465-jv7vb\" (UID: \"b801f228-4d3a-4233-a4ea-a263bf53a34f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0a93465-jv7vb" Oct 31 00:32:20.454607 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:20.454573 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7h7f\" (UniqueName: \"kubernetes.io/projected/b801f228-4d3a-4233-a4ea-a263bf53a34f-kube-api-access-t7h7f\") pod \"maas-api-revoke-sub-f0a93465-jv7vb\" (UID: \"b801f228-4d3a-4233-a4ea-a263bf53a34f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0a93465-jv7vb" Oct 31 00:32:20.460439 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:20.460405 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0a93465-jv7vb" Oct 31 00:32:20.880076 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:32:20.880044 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb801f228_4d3a_4233_a4ea_a263bf53a34f.slice/crio-64890883592cfe5f7b2ee34363d3fe6f5ea96bb67c15777dde78b1aae942ab36 WatchSource:0}: Error finding container 64890883592cfe5f7b2ee34363d3fe6f5ea96bb67c15777dde78b1aae942ab36: Status 404 returned error can't find the container with id 64890883592cfe5f7b2ee34363d3fe6f5ea96bb67c15777dde78b1aae942ab36 Oct 31 00:32:20.883539 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:20.883517 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0a93465-jv7vb"] Oct 31 00:32:20.988879 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:20.988842 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0a93465-jv7vb" event={"ID":"b801f228-4d3a-4233-a4ea-a263bf53a34f","Type":"ContainerStarted","Data":"64890883592cfe5f7b2ee34363d3fe6f5ea96bb67c15777dde78b1aae942ab36"} Oct 31 00:32:21.996875 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:21.996787 5222 generic.go:356] "Generic (PLEG): container finished" podID="b801f228-4d3a-4233-a4ea-a263bf53a34f" containerID="0746566f1cac3816fc98f651c7bfc0750fffbcb76bef5a51d538a0ead6891987" exitCode=0 Oct 31 00:32:21.997266 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:21.996871 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0a93465-jv7vb" event={"ID":"b801f228-4d3a-4233-a4ea-a263bf53a34f","Type":"ContainerDied","Data":"0746566f1cac3816fc98f651c7bfc0750fffbcb76bef5a51d538a0ead6891987"} Oct 31 00:32:23.149157 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:23.149129 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0a93465-jv7vb" Oct 31 00:32:23.231745 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:23.231706 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b801f228-4d3a-4233-a4ea-a263bf53a34f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b801f228-4d3a-4233-a4ea-a263bf53a34f-maas-api-openshift-ca\") pod \"b801f228-4d3a-4233-a4ea-a263bf53a34f\" (UID: \"b801f228-4d3a-4233-a4ea-a263bf53a34f\") " Oct 31 00:32:23.231923 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:23.231760 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b801f228-4d3a-4233-a4ea-a263bf53a34f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b801f228-4d3a-4233-a4ea-a263bf53a34f-maas-api-xks-ca\") pod \"b801f228-4d3a-4233-a4ea-a263bf53a34f\" (UID: \"b801f228-4d3a-4233-a4ea-a263bf53a34f\") " Oct 31 00:32:23.231923 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:23.231903 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b801f228-4d3a-4233-a4ea-a263bf53a34f-kube-api-access-t7h7f\" (UniqueName: \"kubernetes.io/projected/b801f228-4d3a-4233-a4ea-a263bf53a34f-kube-api-access-t7h7f\") pod \"b801f228-4d3a-4233-a4ea-a263bf53a34f\" (UID: \"b801f228-4d3a-4233-a4ea-a263bf53a34f\") " Oct 31 00:32:23.232145 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:23.232120 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b801f228-4d3a-4233-a4ea-a263bf53a34f-maas-api-openshift-ca" pod "b801f228-4d3a-4233-a4ea-a263bf53a34f" (UID: "b801f228-4d3a-4233-a4ea-a263bf53a34f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:32:23.232255 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:23.232232 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b801f228-4d3a-4233-a4ea-a263bf53a34f-maas-api-openshift-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:23.234009 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:23.233984 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b801f228-4d3a-4233-a4ea-a263bf53a34f-maas-api-xks-ca" pod "b801f228-4d3a-4233-a4ea-a263bf53a34f" (UID: "b801f228-4d3a-4233-a4ea-a263bf53a34f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:32:23.234119 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:23.234099 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b801f228-4d3a-4233-a4ea-a263bf53a34f-kube-api-access-t7h7f" pod "b801f228-4d3a-4233-a4ea-a263bf53a34f" (UID: "b801f228-4d3a-4233-a4ea-a263bf53a34f"). InnerVolumeSpecName "kube-api-access-t7h7f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:32:23.332787 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:23.332750 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b801f228-4d3a-4233-a4ea-a263bf53a34f-maas-api-xks-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:23.332787 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:23.332783 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7h7f\" (UniqueName: \"kubernetes.io/projected/b801f228-4d3a-4233-a4ea-a263bf53a34f-kube-api-access-t7h7f\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:24.010221 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:24.010134 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0a93465-jv7vb" Oct 31 00:32:24.010221 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:24.010171 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0a93465-jv7vb" event={"ID":"b801f228-4d3a-4233-a4ea-a263bf53a34f","Type":"ContainerDied","Data":"64890883592cfe5f7b2ee34363d3fe6f5ea96bb67c15777dde78b1aae942ab36"} Oct 31 00:32:24.010481 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:24.010238 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="64890883592cfe5f7b2ee34363d3fe6f5ea96bb67c15777dde78b1aae942ab36" Oct 31 00:32:25.847727 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:25.847691 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f83ba8e-sjjz4"] Oct 31 00:32:25.848454 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:25.848436 5222 cpu_manager.go:414] "containerMap: removing container" podUID="b801f228-4d3a-4233-a4ea-a263bf53a34f" containerName="revoke-keys" Oct 31 00:32:25.848525 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:25.848457 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b801f228-4d3a-4233-a4ea-a263bf53a34f" containerName="revoke-keys" Oct 31 00:32:25.848643 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:25.848631 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="b801f228-4d3a-4233-a4ea-a263bf53a34f" containerName="revoke-keys" Oct 31 00:32:25.852577 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:25.852547 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f83ba8e-sjjz4" Oct 31 00:32:25.855305 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:25.855279 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fe83b3b-1e24-490e-a991-f3b86577405e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f83ba8e-sjjz4\" (UID: \"9fe83b3b-1e24-490e-a991-f3b86577405e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f83ba8e-sjjz4" Oct 31 00:32:25.855475 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:25.855436 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcqkn\" (UniqueName: \"kubernetes.io/projected/9fe83b3b-1e24-490e-a991-f3b86577405e-kube-api-access-rcqkn\") pod \"maas-api-revoke-sub-5f83ba8e-sjjz4\" (UID: \"9fe83b3b-1e24-490e-a991-f3b86577405e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f83ba8e-sjjz4" Oct 31 00:32:25.855528 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:25.855475 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fe83b3b-1e24-490e-a991-f3b86577405e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f83ba8e-sjjz4\" (UID: \"9fe83b3b-1e24-490e-a991-f3b86577405e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f83ba8e-sjjz4" Oct 31 00:32:25.905641 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:25.905601 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f83ba8e-sjjz4"] Oct 31 00:32:25.953795 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:25.953765 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:32:25.953987 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:25.953815 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:32:25.956799 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:25.956767 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fe83b3b-1e24-490e-a991-f3b86577405e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f83ba8e-sjjz4\" (UID: \"9fe83b3b-1e24-490e-a991-f3b86577405e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f83ba8e-sjjz4" Oct 31 00:32:25.956945 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:25.956825 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcqkn\" (UniqueName: \"kubernetes.io/projected/9fe83b3b-1e24-490e-a991-f3b86577405e-kube-api-access-rcqkn\") pod \"maas-api-revoke-sub-5f83ba8e-sjjz4\" (UID: \"9fe83b3b-1e24-490e-a991-f3b86577405e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f83ba8e-sjjz4" Oct 31 00:32:25.956945 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:25.956847 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fe83b3b-1e24-490e-a991-f3b86577405e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f83ba8e-sjjz4\" (UID: \"9fe83b3b-1e24-490e-a991-f3b86577405e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f83ba8e-sjjz4" Oct 31 00:32:25.957406 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:25.957360 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fe83b3b-1e24-490e-a991-f3b86577405e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f83ba8e-sjjz4\" (UID: \"9fe83b3b-1e24-490e-a991-f3b86577405e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f83ba8e-sjjz4" Oct 31 00:32:25.957742 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:25.957725 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:32:25.957788 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:25.957745 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfz9z\"" Oct 31 00:32:25.970613 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:25.970579 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fe83b3b-1e24-490e-a991-f3b86577405e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f83ba8e-sjjz4\" (UID: \"9fe83b3b-1e24-490e-a991-f3b86577405e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f83ba8e-sjjz4" Oct 31 00:32:26.019509 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:26.019483 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcqkn\" (UniqueName: \"kubernetes.io/projected/9fe83b3b-1e24-490e-a991-f3b86577405e-kube-api-access-rcqkn\") pod \"maas-api-revoke-sub-5f83ba8e-sjjz4\" (UID: \"9fe83b3b-1e24-490e-a991-f3b86577405e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f83ba8e-sjjz4" Oct 31 00:32:26.166680 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:26.166574 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f83ba8e-sjjz4" Oct 31 00:32:26.541446 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:32:26.541155 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fe83b3b_1e24_490e_a991_f3b86577405e.slice/crio-5f77ece14a19669543d72ad4f7db01ee09d9a266289173a768b91973d2f4e684 WatchSource:0}: Error finding container 5f77ece14a19669543d72ad4f7db01ee09d9a266289173a768b91973d2f4e684: Status 404 returned error can't find the container with id 5f77ece14a19669543d72ad4f7db01ee09d9a266289173a768b91973d2f4e684 Oct 31 00:32:26.544100 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:26.544078 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f83ba8e-sjjz4"] Oct 31 00:32:26.850692 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:26.850555 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49bb57d4-8qrc9"] Oct 31 00:32:26.854670 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:26.854646 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49bb57d4-8qrc9" Oct 31 00:32:26.868095 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:26.868065 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad445cdf-d3a4-4363-8411-c66c7ac98ab8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49bb57d4-8qrc9\" (UID: \"ad445cdf-d3a4-4363-8411-c66c7ac98ab8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49bb57d4-8qrc9" Oct 31 00:32:26.868271 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:26.868104 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad445cdf-d3a4-4363-8411-c66c7ac98ab8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49bb57d4-8qrc9\" (UID: \"ad445cdf-d3a4-4363-8411-c66c7ac98ab8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49bb57d4-8qrc9" Oct 31 00:32:26.868271 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:26.868243 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xx6v\" (UniqueName: \"kubernetes.io/projected/ad445cdf-d3a4-4363-8411-c66c7ac98ab8-kube-api-access-9xx6v\") pod \"maas-api-revoke-sub-49bb57d4-8qrc9\" (UID: \"ad445cdf-d3a4-4363-8411-c66c7ac98ab8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49bb57d4-8qrc9" Oct 31 00:32:26.903348 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:26.903312 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eabb169b-8rpmt"] Oct 31 00:32:26.907724 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:26.907695 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eabb169b-8rpmt" Oct 31 00:32:26.911888 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:26.911857 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49bb57d4-8qrc9"] Oct 31 00:32:26.960984 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:26.960946 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eabb169b-8rpmt"] Oct 31 00:32:26.969203 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:26.969166 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a9a2e33-97eb-4882-b975-7f52b2e32dec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eabb169b-8rpmt\" (UID: \"8a9a2e33-97eb-4882-b975-7f52b2e32dec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eabb169b-8rpmt" Oct 31 00:32:26.969423 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:26.969307 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a9a2e33-97eb-4882-b975-7f52b2e32dec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eabb169b-8rpmt\" (UID: \"8a9a2e33-97eb-4882-b975-7f52b2e32dec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eabb169b-8rpmt" Oct 31 00:32:26.969423 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:26.969347 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad445cdf-d3a4-4363-8411-c66c7ac98ab8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49bb57d4-8qrc9\" (UID: \"ad445cdf-d3a4-4363-8411-c66c7ac98ab8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49bb57d4-8qrc9" Oct 31 00:32:26.969423 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:26.969396 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad445cdf-d3a4-4363-8411-c66c7ac98ab8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49bb57d4-8qrc9\" (UID: \"ad445cdf-d3a4-4363-8411-c66c7ac98ab8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49bb57d4-8qrc9" Oct 31 00:32:26.969688 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:26.969526 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9hdz\" (UniqueName: \"kubernetes.io/projected/8a9a2e33-97eb-4882-b975-7f52b2e32dec-kube-api-access-c9hdz\") pod \"maas-api-revoke-sub-eabb169b-8rpmt\" (UID: \"8a9a2e33-97eb-4882-b975-7f52b2e32dec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eabb169b-8rpmt" Oct 31 00:32:26.969688 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:26.969657 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xx6v\" (UniqueName: \"kubernetes.io/projected/ad445cdf-d3a4-4363-8411-c66c7ac98ab8-kube-api-access-9xx6v\") pod \"maas-api-revoke-sub-49bb57d4-8qrc9\" (UID: \"ad445cdf-d3a4-4363-8411-c66c7ac98ab8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49bb57d4-8qrc9" Oct 31 00:32:26.970666 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:26.970618 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad445cdf-d3a4-4363-8411-c66c7ac98ab8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49bb57d4-8qrc9\" (UID: \"ad445cdf-d3a4-4363-8411-c66c7ac98ab8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49bb57d4-8qrc9" Oct 31 00:32:26.972467 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:26.972444 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad445cdf-d3a4-4363-8411-c66c7ac98ab8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49bb57d4-8qrc9\" (UID: \"ad445cdf-d3a4-4363-8411-c66c7ac98ab8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49bb57d4-8qrc9" Oct 31 00:32:27.030040 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:27.030003 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f83ba8e-sjjz4" event={"ID":"9fe83b3b-1e24-490e-a991-f3b86577405e","Type":"ContainerStarted","Data":"50ae8c2d47a857f59d86ba7b31f953675e05abd936d50805185fb728f5a43649"} Oct 31 00:32:27.030227 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:27.030055 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f83ba8e-sjjz4" event={"ID":"9fe83b3b-1e24-490e-a991-f3b86577405e","Type":"ContainerStarted","Data":"5f77ece14a19669543d72ad4f7db01ee09d9a266289173a768b91973d2f4e684"} Oct 31 00:32:27.032740 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:27.032711 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xx6v\" (UniqueName: \"kubernetes.io/projected/ad445cdf-d3a4-4363-8411-c66c7ac98ab8-kube-api-access-9xx6v\") pod \"maas-api-revoke-sub-49bb57d4-8qrc9\" (UID: \"ad445cdf-d3a4-4363-8411-c66c7ac98ab8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49bb57d4-8qrc9" Oct 31 00:32:27.070937 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:27.070904 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9hdz\" (UniqueName: \"kubernetes.io/projected/8a9a2e33-97eb-4882-b975-7f52b2e32dec-kube-api-access-c9hdz\") pod \"maas-api-revoke-sub-eabb169b-8rpmt\" (UID: \"8a9a2e33-97eb-4882-b975-7f52b2e32dec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eabb169b-8rpmt" Oct 31 00:32:27.071108 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:27.071028 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a9a2e33-97eb-4882-b975-7f52b2e32dec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eabb169b-8rpmt\" (UID: \"8a9a2e33-97eb-4882-b975-7f52b2e32dec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eabb169b-8rpmt" Oct 31 00:32:27.071204 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:27.071123 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a9a2e33-97eb-4882-b975-7f52b2e32dec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eabb169b-8rpmt\" (UID: \"8a9a2e33-97eb-4882-b975-7f52b2e32dec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eabb169b-8rpmt" Oct 31 00:32:27.071770 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:27.071744 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a9a2e33-97eb-4882-b975-7f52b2e32dec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eabb169b-8rpmt\" (UID: \"8a9a2e33-97eb-4882-b975-7f52b2e32dec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eabb169b-8rpmt" Oct 31 00:32:27.073606 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:27.073584 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a9a2e33-97eb-4882-b975-7f52b2e32dec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eabb169b-8rpmt\" (UID: \"8a9a2e33-97eb-4882-b975-7f52b2e32dec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eabb169b-8rpmt" Oct 31 00:32:27.133997 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:27.133915 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9hdz\" (UniqueName: \"kubernetes.io/projected/8a9a2e33-97eb-4882-b975-7f52b2e32dec-kube-api-access-c9hdz\") pod \"maas-api-revoke-sub-eabb169b-8rpmt\" (UID: \"8a9a2e33-97eb-4882-b975-7f52b2e32dec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eabb169b-8rpmt" Oct 31 00:32:27.167763 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:27.167715 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49bb57d4-8qrc9" Oct 31 00:32:27.207243 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:27.207171 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f83ba8e-sjjz4" podStartSLOduration=2.207148967 podStartE2EDuration="2.207148967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:32:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:32:27.205150367 +0000 UTC m=+1039.676320116" watchObservedRunningTime="2026-10-31 00:32:27.207148967 +0000 UTC m=+1039.678318717" Oct 31 00:32:27.231719 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:27.231689 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eabb169b-8rpmt" Oct 31 00:32:27.547135 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:27.547102 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49bb57d4-8qrc9"] Oct 31 00:32:27.568413 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:32:27.568363 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad445cdf_d3a4_4363_8411_c66c7ac98ab8.slice/crio-d4ad0313ab5c3783c59048eab80be6d8957cce4a94bc71789a79db090d005192 WatchSource:0}: Error finding container d4ad0313ab5c3783c59048eab80be6d8957cce4a94bc71789a79db090d005192: Status 404 returned error can't find the container with id d4ad0313ab5c3783c59048eab80be6d8957cce4a94bc71789a79db090d005192 Oct 31 00:32:27.642527 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:32:27.642497 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a9a2e33_97eb_4882_b975_7f52b2e32dec.slice/crio-590d04432be950b30367c8e2f455644f77399b323b12687c9aaea67435a47aa6 WatchSource:0}: Error finding container 590d04432be950b30367c8e2f455644f77399b323b12687c9aaea67435a47aa6: Status 404 returned error can't find the container with id 590d04432be950b30367c8e2f455644f77399b323b12687c9aaea67435a47aa6 Oct 31 00:32:27.644685 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:27.644662 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eabb169b-8rpmt"] Oct 31 00:32:28.040183 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:28.040144 5222 generic.go:356] "Generic (PLEG): container finished" podID="9fe83b3b-1e24-490e-a991-f3b86577405e" containerID="50ae8c2d47a857f59d86ba7b31f953675e05abd936d50805185fb728f5a43649" exitCode=0 Oct 31 00:32:28.040667 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:28.040240 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f83ba8e-sjjz4" event={"ID":"9fe83b3b-1e24-490e-a991-f3b86577405e","Type":"ContainerDied","Data":"50ae8c2d47a857f59d86ba7b31f953675e05abd936d50805185fb728f5a43649"} Oct 31 00:32:28.041823 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:28.041799 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eabb169b-8rpmt" event={"ID":"8a9a2e33-97eb-4882-b975-7f52b2e32dec","Type":"ContainerStarted","Data":"590d04432be950b30367c8e2f455644f77399b323b12687c9aaea67435a47aa6"} Oct 31 00:32:28.043553 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:28.043533 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49bb57d4-8qrc9" event={"ID":"ad445cdf-d3a4-4363-8411-c66c7ac98ab8","Type":"ContainerStarted","Data":"b36524d123b292dbddd465b82268fffbad40658f525fb1620645fa50bf2bc83a"} Oct 31 00:32:28.043629 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:28.043574 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49bb57d4-8qrc9" event={"ID":"ad445cdf-d3a4-4363-8411-c66c7ac98ab8","Type":"ContainerStarted","Data":"d4ad0313ab5c3783c59048eab80be6d8957cce4a94bc71789a79db090d005192"} Oct 31 00:32:28.283667 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:28.282686 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49bb57d4-8qrc9" podStartSLOduration=2.282668477 podStartE2EDuration="2.282668477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:32:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:32:28.279198458 +0000 UTC m=+1040.750368209" watchObservedRunningTime="2026-10-31 00:32:28.282668477 +0000 UTC m=+1040.753838226" Oct 31 00:32:29.051952 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:29.051918 5222 generic.go:356] "Generic (PLEG): container finished" podID="8a9a2e33-97eb-4882-b975-7f52b2e32dec" containerID="f751d5ac35992fbfa181259fc9f6e4ef0f3e8cca8fbb6b4d86ebb2807298c7a1" exitCode=0 Oct 31 00:32:29.052497 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:29.051991 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eabb169b-8rpmt" event={"ID":"8a9a2e33-97eb-4882-b975-7f52b2e32dec","Type":"ContainerDied","Data":"f751d5ac35992fbfa181259fc9f6e4ef0f3e8cca8fbb6b4d86ebb2807298c7a1"} Oct 31 00:32:29.053859 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:29.053834 5222 generic.go:356] "Generic (PLEG): container finished" podID="ad445cdf-d3a4-4363-8411-c66c7ac98ab8" containerID="b36524d123b292dbddd465b82268fffbad40658f525fb1620645fa50bf2bc83a" exitCode=0 Oct 31 00:32:29.053993 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:29.053915 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49bb57d4-8qrc9" event={"ID":"ad445cdf-d3a4-4363-8411-c66c7ac98ab8","Type":"ContainerDied","Data":"b36524d123b292dbddd465b82268fffbad40658f525fb1620645fa50bf2bc83a"} Oct 31 00:32:29.220363 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:29.220331 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f83ba8e-sjjz4" Oct 31 00:32:29.293990 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:29.293942 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fe83b3b-1e24-490e-a991-f3b86577405e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fe83b3b-1e24-490e-a991-f3b86577405e-maas-api-openshift-ca\") pod \"9fe83b3b-1e24-490e-a991-f3b86577405e\" (UID: \"9fe83b3b-1e24-490e-a991-f3b86577405e\") " Oct 31 00:32:29.294175 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:29.294052 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fe83b3b-1e24-490e-a991-f3b86577405e-kube-api-access-rcqkn\" (UniqueName: \"kubernetes.io/projected/9fe83b3b-1e24-490e-a991-f3b86577405e-kube-api-access-rcqkn\") pod \"9fe83b3b-1e24-490e-a991-f3b86577405e\" (UID: \"9fe83b3b-1e24-490e-a991-f3b86577405e\") " Oct 31 00:32:29.294252 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:29.294173 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9fe83b3b-1e24-490e-a991-f3b86577405e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fe83b3b-1e24-490e-a991-f3b86577405e-maas-api-xks-ca\") pod \"9fe83b3b-1e24-490e-a991-f3b86577405e\" (UID: \"9fe83b3b-1e24-490e-a991-f3b86577405e\") " Oct 31 00:32:29.294345 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:29.294316 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fe83b3b-1e24-490e-a991-f3b86577405e-maas-api-openshift-ca" pod "9fe83b3b-1e24-490e-a991-f3b86577405e" (UID: "9fe83b3b-1e24-490e-a991-f3b86577405e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:32:29.294585 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:29.294565 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fe83b3b-1e24-490e-a991-f3b86577405e-maas-api-openshift-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:29.296421 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:29.296390 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fe83b3b-1e24-490e-a991-f3b86577405e-maas-api-xks-ca" pod "9fe83b3b-1e24-490e-a991-f3b86577405e" (UID: "9fe83b3b-1e24-490e-a991-f3b86577405e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:32:29.296421 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:29.296415 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fe83b3b-1e24-490e-a991-f3b86577405e-kube-api-access-rcqkn" pod "9fe83b3b-1e24-490e-a991-f3b86577405e" (UID: "9fe83b3b-1e24-490e-a991-f3b86577405e"). InnerVolumeSpecName "kube-api-access-rcqkn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:32:29.395643 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:29.395610 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fe83b3b-1e24-490e-a991-f3b86577405e-maas-api-xks-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:29.395643 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:29.395641 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcqkn\" (UniqueName: \"kubernetes.io/projected/9fe83b3b-1e24-490e-a991-f3b86577405e-kube-api-access-rcqkn\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:30.061535 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:30.061503 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f83ba8e-sjjz4" Oct 31 00:32:30.061535 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:30.061531 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f83ba8e-sjjz4" event={"ID":"9fe83b3b-1e24-490e-a991-f3b86577405e","Type":"ContainerDied","Data":"5f77ece14a19669543d72ad4f7db01ee09d9a266289173a768b91973d2f4e684"} Oct 31 00:32:30.062138 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:30.061580 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5f77ece14a19669543d72ad4f7db01ee09d9a266289173a768b91973d2f4e684" Oct 31 00:32:30.216169 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:30.216144 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49bb57d4-8qrc9" Oct 31 00:32:30.245692 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:30.245664 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eabb169b-8rpmt" Oct 31 00:32:30.304323 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:30.304292 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad445cdf-d3a4-4363-8411-c66c7ac98ab8-kube-api-access-9xx6v\" (UniqueName: \"kubernetes.io/projected/ad445cdf-d3a4-4363-8411-c66c7ac98ab8-kube-api-access-9xx6v\") pod \"ad445cdf-d3a4-4363-8411-c66c7ac98ab8\" (UID: \"ad445cdf-d3a4-4363-8411-c66c7ac98ab8\") " Oct 31 00:32:30.304565 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:30.304399 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad445cdf-d3a4-4363-8411-c66c7ac98ab8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad445cdf-d3a4-4363-8411-c66c7ac98ab8-maas-api-xks-ca\") pod \"ad445cdf-d3a4-4363-8411-c66c7ac98ab8\" (UID: \"ad445cdf-d3a4-4363-8411-c66c7ac98ab8\") " Oct 31 00:32:30.304565 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:30.304484 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a9a2e33-97eb-4882-b975-7f52b2e32dec-kube-api-access-c9hdz\" (UniqueName: \"kubernetes.io/projected/8a9a2e33-97eb-4882-b975-7f52b2e32dec-kube-api-access-c9hdz\") pod \"8a9a2e33-97eb-4882-b975-7f52b2e32dec\" (UID: \"8a9a2e33-97eb-4882-b975-7f52b2e32dec\") " Oct 31 00:32:30.304684 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:30.304571 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad445cdf-d3a4-4363-8411-c66c7ac98ab8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad445cdf-d3a4-4363-8411-c66c7ac98ab8-maas-api-openshift-ca\") pod \"ad445cdf-d3a4-4363-8411-c66c7ac98ab8\" (UID: \"ad445cdf-d3a4-4363-8411-c66c7ac98ab8\") " Oct 31 00:32:30.304684 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:30.304614 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a9a2e33-97eb-4882-b975-7f52b2e32dec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a9a2e33-97eb-4882-b975-7f52b2e32dec-maas-api-openshift-ca\") pod \"8a9a2e33-97eb-4882-b975-7f52b2e32dec\" (UID: \"8a9a2e33-97eb-4882-b975-7f52b2e32dec\") " Oct 31 00:32:30.304801 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:30.304695 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a9a2e33-97eb-4882-b975-7f52b2e32dec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a9a2e33-97eb-4882-b975-7f52b2e32dec-maas-api-xks-ca\") pod \"8a9a2e33-97eb-4882-b975-7f52b2e32dec\" (UID: \"8a9a2e33-97eb-4882-b975-7f52b2e32dec\") " Oct 31 00:32:30.305090 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:30.305055 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad445cdf-d3a4-4363-8411-c66c7ac98ab8-maas-api-openshift-ca" pod "ad445cdf-d3a4-4363-8411-c66c7ac98ab8" (UID: "ad445cdf-d3a4-4363-8411-c66c7ac98ab8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:32:30.305362 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:30.305073 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a9a2e33-97eb-4882-b975-7f52b2e32dec-maas-api-openshift-ca" pod "8a9a2e33-97eb-4882-b975-7f52b2e32dec" (UID: "8a9a2e33-97eb-4882-b975-7f52b2e32dec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:32:30.306869 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:30.306843 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad445cdf-d3a4-4363-8411-c66c7ac98ab8-kube-api-access-9xx6v" pod "ad445cdf-d3a4-4363-8411-c66c7ac98ab8" (UID: "ad445cdf-d3a4-4363-8411-c66c7ac98ab8"). InnerVolumeSpecName "kube-api-access-9xx6v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:32:30.307137 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:30.307111 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a9a2e33-97eb-4882-b975-7f52b2e32dec-maas-api-xks-ca" pod "8a9a2e33-97eb-4882-b975-7f52b2e32dec" (UID: "8a9a2e33-97eb-4882-b975-7f52b2e32dec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:32:30.307278 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:30.307254 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad445cdf-d3a4-4363-8411-c66c7ac98ab8-maas-api-xks-ca" pod "ad445cdf-d3a4-4363-8411-c66c7ac98ab8" (UID: "ad445cdf-d3a4-4363-8411-c66c7ac98ab8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:32:30.307414 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:30.307291 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a9a2e33-97eb-4882-b975-7f52b2e32dec-kube-api-access-c9hdz" pod "8a9a2e33-97eb-4882-b975-7f52b2e32dec" (UID: "8a9a2e33-97eb-4882-b975-7f52b2e32dec"). InnerVolumeSpecName "kube-api-access-c9hdz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:32:30.405805 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:30.405771 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a9a2e33-97eb-4882-b975-7f52b2e32dec-maas-api-xks-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:30.405805 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:30.405801 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xx6v\" (UniqueName: \"kubernetes.io/projected/ad445cdf-d3a4-4363-8411-c66c7ac98ab8-kube-api-access-9xx6v\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:30.406020 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:30.405813 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad445cdf-d3a4-4363-8411-c66c7ac98ab8-maas-api-xks-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:30.406020 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:30.405829 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9hdz\" (UniqueName: \"kubernetes.io/projected/8a9a2e33-97eb-4882-b975-7f52b2e32dec-kube-api-access-c9hdz\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:30.406020 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:30.405840 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad445cdf-d3a4-4363-8411-c66c7ac98ab8-maas-api-openshift-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:30.406020 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:30.405848 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a9a2e33-97eb-4882-b975-7f52b2e32dec-maas-api-openshift-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:31.069485 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:31.069462 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49bb57d4-8qrc9" Oct 31 00:32:31.069999 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:31.069460 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49bb57d4-8qrc9" event={"ID":"ad445cdf-d3a4-4363-8411-c66c7ac98ab8","Type":"ContainerDied","Data":"d4ad0313ab5c3783c59048eab80be6d8957cce4a94bc71789a79db090d005192"} Oct 31 00:32:31.069999 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:31.069605 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d4ad0313ab5c3783c59048eab80be6d8957cce4a94bc71789a79db090d005192" Oct 31 00:32:31.071008 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:31.070988 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eabb169b-8rpmt" Oct 31 00:32:31.071157 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:31.071035 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eabb169b-8rpmt" event={"ID":"8a9a2e33-97eb-4882-b975-7f52b2e32dec","Type":"ContainerDied","Data":"590d04432be950b30367c8e2f455644f77399b323b12687c9aaea67435a47aa6"} Oct 31 00:32:31.071157 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:31.071069 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="590d04432be950b30367c8e2f455644f77399b323b12687c9aaea67435a47aa6" Oct 31 00:32:35.890055 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:32:35.890013 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 00:32:42.046391 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:42.046339 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afa82603-jr9rb"] Oct 31 00:32:42.046841 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:42.046827 5222 cpu_manager.go:414] "containerMap: removing container" podUID="ad445cdf-d3a4-4363-8411-c66c7ac98ab8" containerName="revoke-keys" Oct 31 00:32:42.046889 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:42.046842 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad445cdf-d3a4-4363-8411-c66c7ac98ab8" containerName="revoke-keys" Oct 31 00:32:42.046889 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:42.046877 5222 cpu_manager.go:414] "containerMap: removing container" podUID="9fe83b3b-1e24-490e-a991-f3b86577405e" containerName="revoke-keys" Oct 31 00:32:42.046889 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:42.046881 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fe83b3b-1e24-490e-a991-f3b86577405e" containerName="revoke-keys" Oct 31 00:32:42.046889 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:42.046888 5222 cpu_manager.go:414] "containerMap: removing container" podUID="8a9a2e33-97eb-4882-b975-7f52b2e32dec" containerName="revoke-keys" Oct 31 00:32:42.047011 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:42.046893 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a9a2e33-97eb-4882-b975-7f52b2e32dec" containerName="revoke-keys" Oct 31 00:32:42.047011 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:42.046965 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a9a2e33-97eb-4882-b975-7f52b2e32dec" containerName="revoke-keys" Oct 31 00:32:42.047011 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:42.046975 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad445cdf-d3a4-4363-8411-c66c7ac98ab8" containerName="revoke-keys" Oct 31 00:32:42.047011 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:42.046983 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fe83b3b-1e24-490e-a991-f3b86577405e" containerName="revoke-keys" Oct 31 00:32:42.051697 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:42.051676 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa82603-jr9rb" Oct 31 00:32:42.104540 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:42.104501 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afa82603-jr9rb"] Oct 31 00:32:42.135568 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:42.135531 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65htw\" (UniqueName: \"kubernetes.io/projected/62140133-beb2-49fc-a4c9-4864ed0dd580-kube-api-access-65htw\") pod \"maas-api-revoke-sub-afa82603-jr9rb\" (UID: \"62140133-beb2-49fc-a4c9-4864ed0dd580\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa82603-jr9rb" Oct 31 00:32:42.135752 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:42.135668 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62140133-beb2-49fc-a4c9-4864ed0dd580-maas-api-xks-ca\") pod \"maas-api-revoke-sub-afa82603-jr9rb\" (UID: \"62140133-beb2-49fc-a4c9-4864ed0dd580\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa82603-jr9rb" Oct 31 00:32:42.135752 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:42.135709 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62140133-beb2-49fc-a4c9-4864ed0dd580-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-afa82603-jr9rb\" (UID: \"62140133-beb2-49fc-a4c9-4864ed0dd580\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa82603-jr9rb" Oct 31 00:32:42.152322 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:42.152295 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:32:42.152322 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:42.152316 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:32:42.152632 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:42.152319 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:32:42.152632 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:42.152442 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfz9z\"" Oct 31 00:32:42.236849 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:42.236802 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62140133-beb2-49fc-a4c9-4864ed0dd580-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-afa82603-jr9rb\" (UID: \"62140133-beb2-49fc-a4c9-4864ed0dd580\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa82603-jr9rb" Oct 31 00:32:42.237065 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:42.236893 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65htw\" (UniqueName: \"kubernetes.io/projected/62140133-beb2-49fc-a4c9-4864ed0dd580-kube-api-access-65htw\") pod \"maas-api-revoke-sub-afa82603-jr9rb\" (UID: \"62140133-beb2-49fc-a4c9-4864ed0dd580\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa82603-jr9rb" Oct 31 00:32:42.237065 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:42.237018 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62140133-beb2-49fc-a4c9-4864ed0dd580-maas-api-xks-ca\") pod \"maas-api-revoke-sub-afa82603-jr9rb\" (UID: \"62140133-beb2-49fc-a4c9-4864ed0dd580\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa82603-jr9rb" Oct 31 00:32:42.237574 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:42.237546 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62140133-beb2-49fc-a4c9-4864ed0dd580-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-afa82603-jr9rb\" (UID: \"62140133-beb2-49fc-a4c9-4864ed0dd580\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa82603-jr9rb" Oct 31 00:32:42.239485 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:42.239466 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62140133-beb2-49fc-a4c9-4864ed0dd580-maas-api-xks-ca\") pod \"maas-api-revoke-sub-afa82603-jr9rb\" (UID: \"62140133-beb2-49fc-a4c9-4864ed0dd580\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa82603-jr9rb" Oct 31 00:32:42.300214 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:42.300138 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65htw\" (UniqueName: \"kubernetes.io/projected/62140133-beb2-49fc-a4c9-4864ed0dd580-kube-api-access-65htw\") pod \"maas-api-revoke-sub-afa82603-jr9rb\" (UID: \"62140133-beb2-49fc-a4c9-4864ed0dd580\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa82603-jr9rb" Oct 31 00:32:42.365483 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:42.365447 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa82603-jr9rb" Oct 31 00:32:42.464344 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:42.464307 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14eec3f8-z5kkm"] Oct 31 00:32:42.469414 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:42.469386 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14eec3f8-z5kkm" Oct 31 00:32:42.523559 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:42.523526 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14eec3f8-z5kkm"] Oct 31 00:32:42.540983 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:42.540945 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sw5vk\" (UniqueName: \"kubernetes.io/projected/d22f0a86-6e20-4f79-8104-b47a4431d338-kube-api-access-sw5vk\") pod \"maas-api-revoke-sub-14eec3f8-z5kkm\" (UID: \"d22f0a86-6e20-4f79-8104-b47a4431d338\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14eec3f8-z5kkm" Oct 31 00:32:42.541190 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:42.541024 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d22f0a86-6e20-4f79-8104-b47a4431d338-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14eec3f8-z5kkm\" (UID: \"d22f0a86-6e20-4f79-8104-b47a4431d338\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14eec3f8-z5kkm" Oct 31 00:32:42.541283 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:42.541225 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d22f0a86-6e20-4f79-8104-b47a4431d338-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14eec3f8-z5kkm\" (UID: \"d22f0a86-6e20-4f79-8104-b47a4431d338\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14eec3f8-z5kkm" Oct 31 00:32:42.642690 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:42.642604 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sw5vk\" (UniqueName: \"kubernetes.io/projected/d22f0a86-6e20-4f79-8104-b47a4431d338-kube-api-access-sw5vk\") pod \"maas-api-revoke-sub-14eec3f8-z5kkm\" (UID: \"d22f0a86-6e20-4f79-8104-b47a4431d338\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14eec3f8-z5kkm" Oct 31 00:32:42.642690 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:42.642662 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d22f0a86-6e20-4f79-8104-b47a4431d338-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14eec3f8-z5kkm\" (UID: \"d22f0a86-6e20-4f79-8104-b47a4431d338\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14eec3f8-z5kkm" Oct 31 00:32:42.642885 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:42.642788 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d22f0a86-6e20-4f79-8104-b47a4431d338-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14eec3f8-z5kkm\" (UID: \"d22f0a86-6e20-4f79-8104-b47a4431d338\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14eec3f8-z5kkm" Oct 31 00:32:42.643354 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:42.643330 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d22f0a86-6e20-4f79-8104-b47a4431d338-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14eec3f8-z5kkm\" (UID: \"d22f0a86-6e20-4f79-8104-b47a4431d338\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14eec3f8-z5kkm" Oct 31 00:32:42.645325 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:42.645296 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d22f0a86-6e20-4f79-8104-b47a4431d338-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14eec3f8-z5kkm\" (UID: \"d22f0a86-6e20-4f79-8104-b47a4431d338\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14eec3f8-z5kkm" Oct 31 00:32:42.705220 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:42.705184 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sw5vk\" (UniqueName: \"kubernetes.io/projected/d22f0a86-6e20-4f79-8104-b47a4431d338-kube-api-access-sw5vk\") pod \"maas-api-revoke-sub-14eec3f8-z5kkm\" (UID: \"d22f0a86-6e20-4f79-8104-b47a4431d338\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14eec3f8-z5kkm" Oct 31 00:32:42.741802 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:32:42.741773 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62140133_beb2_49fc_a4c9_4864ed0dd580.slice/crio-504d89b9f73b3fbc39b10dc0356d84b47d41fbf51fdfff09bf5e9f92188edca9 WatchSource:0}: Error finding container 504d89b9f73b3fbc39b10dc0356d84b47d41fbf51fdfff09bf5e9f92188edca9: Status 404 returned error can't find the container with id 504d89b9f73b3fbc39b10dc0356d84b47d41fbf51fdfff09bf5e9f92188edca9 Oct 31 00:32:42.744347 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:42.744321 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afa82603-jr9rb"] Oct 31 00:32:42.786288 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:42.786259 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14eec3f8-z5kkm" Oct 31 00:32:43.143730 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:43.143697 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa82603-jr9rb" event={"ID":"62140133-beb2-49fc-a4c9-4864ed0dd580","Type":"ContainerStarted","Data":"26635f2b2b53ff11d5d104155e95ae94721a63a7cf4151fb28a01d8fc721fa79"} Oct 31 00:32:43.144129 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:43.143756 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa82603-jr9rb" event={"ID":"62140133-beb2-49fc-a4c9-4864ed0dd580","Type":"ContainerStarted","Data":"504d89b9f73b3fbc39b10dc0356d84b47d41fbf51fdfff09bf5e9f92188edca9"} Oct 31 00:32:43.164473 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:32:43.164438 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd22f0a86_6e20_4f79_8104_b47a4431d338.slice/crio-3f9c29e8a5dbe9dbd104745c3713b6b5b4ae87d291f14b72d508c8ae389a512e WatchSource:0}: Error finding container 3f9c29e8a5dbe9dbd104745c3713b6b5b4ae87d291f14b72d508c8ae389a512e: Status 404 returned error can't find the container with id 3f9c29e8a5dbe9dbd104745c3713b6b5b4ae87d291f14b72d508c8ae389a512e Oct 31 00:32:43.169723 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:43.169684 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14eec3f8-z5kkm"] Oct 31 00:32:43.263179 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:43.263123 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa82603-jr9rb" podStartSLOduration=1.263105114 podStartE2EDuration="1.263105114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:32:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:32:43.261792398 +0000 UTC m=+1055.732962150" watchObservedRunningTime="2026-10-31 00:32:43.263105114 +0000 UTC m=+1055.734274862" Oct 31 00:32:44.150961 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:44.150924 5222 generic.go:356] "Generic (PLEG): container finished" podID="62140133-beb2-49fc-a4c9-4864ed0dd580" containerID="26635f2b2b53ff11d5d104155e95ae94721a63a7cf4151fb28a01d8fc721fa79" exitCode=0 Oct 31 00:32:44.151413 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:44.151012 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa82603-jr9rb" event={"ID":"62140133-beb2-49fc-a4c9-4864ed0dd580","Type":"ContainerDied","Data":"26635f2b2b53ff11d5d104155e95ae94721a63a7cf4151fb28a01d8fc721fa79"} Oct 31 00:32:44.152678 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:44.152652 5222 generic.go:356] "Generic (PLEG): container finished" podID="d22f0a86-6e20-4f79-8104-b47a4431d338" containerID="0b96fccce54fcbb8e519b18f3d66eed5339168be902319e84064ae7d936f73b9" exitCode=0 Oct 31 00:32:44.152798 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:44.152725 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14eec3f8-z5kkm" event={"ID":"d22f0a86-6e20-4f79-8104-b47a4431d338","Type":"ContainerDied","Data":"0b96fccce54fcbb8e519b18f3d66eed5339168be902319e84064ae7d936f73b9"} Oct 31 00:32:44.152798 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:44.152776 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14eec3f8-z5kkm" event={"ID":"d22f0a86-6e20-4f79-8104-b47a4431d338","Type":"ContainerStarted","Data":"3f9c29e8a5dbe9dbd104745c3713b6b5b4ae87d291f14b72d508c8ae389a512e"} Oct 31 00:32:45.307805 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:45.307775 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14eec3f8-z5kkm" Oct 31 00:32:45.345363 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:45.345339 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa82603-jr9rb" Oct 31 00:32:45.371191 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:45.371160 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d22f0a86-6e20-4f79-8104-b47a4431d338-kube-api-access-sw5vk\" (UniqueName: \"kubernetes.io/projected/d22f0a86-6e20-4f79-8104-b47a4431d338-kube-api-access-sw5vk\") pod \"d22f0a86-6e20-4f79-8104-b47a4431d338\" (UID: \"d22f0a86-6e20-4f79-8104-b47a4431d338\") " Oct 31 00:32:45.371405 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:45.371280 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d22f0a86-6e20-4f79-8104-b47a4431d338-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d22f0a86-6e20-4f79-8104-b47a4431d338-maas-api-openshift-ca\") pod \"d22f0a86-6e20-4f79-8104-b47a4431d338\" (UID: \"d22f0a86-6e20-4f79-8104-b47a4431d338\") " Oct 31 00:32:45.371405 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:45.371308 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d22f0a86-6e20-4f79-8104-b47a4431d338-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d22f0a86-6e20-4f79-8104-b47a4431d338-maas-api-xks-ca\") pod \"d22f0a86-6e20-4f79-8104-b47a4431d338\" (UID: \"d22f0a86-6e20-4f79-8104-b47a4431d338\") " Oct 31 00:32:45.371692 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:45.371666 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d22f0a86-6e20-4f79-8104-b47a4431d338-maas-api-openshift-ca" pod "d22f0a86-6e20-4f79-8104-b47a4431d338" (UID: "d22f0a86-6e20-4f79-8104-b47a4431d338"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:32:45.373653 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:45.373624 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d22f0a86-6e20-4f79-8104-b47a4431d338-kube-api-access-sw5vk" pod "d22f0a86-6e20-4f79-8104-b47a4431d338" (UID: "d22f0a86-6e20-4f79-8104-b47a4431d338"). InnerVolumeSpecName "kube-api-access-sw5vk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:32:45.373653 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:45.373641 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d22f0a86-6e20-4f79-8104-b47a4431d338-maas-api-xks-ca" pod "d22f0a86-6e20-4f79-8104-b47a4431d338" (UID: "d22f0a86-6e20-4f79-8104-b47a4431d338"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:32:45.472710 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:45.472621 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62140133-beb2-49fc-a4c9-4864ed0dd580-kube-api-access-65htw\" (UniqueName: \"kubernetes.io/projected/62140133-beb2-49fc-a4c9-4864ed0dd580-kube-api-access-65htw\") pod \"62140133-beb2-49fc-a4c9-4864ed0dd580\" (UID: \"62140133-beb2-49fc-a4c9-4864ed0dd580\") " Oct 31 00:32:45.472886 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:45.472729 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62140133-beb2-49fc-a4c9-4864ed0dd580-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62140133-beb2-49fc-a4c9-4864ed0dd580-maas-api-openshift-ca\") pod \"62140133-beb2-49fc-a4c9-4864ed0dd580\" (UID: \"62140133-beb2-49fc-a4c9-4864ed0dd580\") " Oct 31 00:32:45.472886 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:45.472772 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62140133-beb2-49fc-a4c9-4864ed0dd580-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62140133-beb2-49fc-a4c9-4864ed0dd580-maas-api-xks-ca\") pod \"62140133-beb2-49fc-a4c9-4864ed0dd580\" (UID: \"62140133-beb2-49fc-a4c9-4864ed0dd580\") " Oct 31 00:32:45.473089 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:45.473065 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62140133-beb2-49fc-a4c9-4864ed0dd580-maas-api-openshift-ca" pod "62140133-beb2-49fc-a4c9-4864ed0dd580" (UID: "62140133-beb2-49fc-a4c9-4864ed0dd580"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:32:45.473196 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:45.473179 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sw5vk\" (UniqueName: \"kubernetes.io/projected/d22f0a86-6e20-4f79-8104-b47a4431d338-kube-api-access-sw5vk\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:45.473256 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:45.473201 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62140133-beb2-49fc-a4c9-4864ed0dd580-maas-api-openshift-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:45.473256 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:45.473217 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d22f0a86-6e20-4f79-8104-b47a4431d338-maas-api-openshift-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:45.473256 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:45.473232 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d22f0a86-6e20-4f79-8104-b47a4431d338-maas-api-xks-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:45.474857 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:45.474835 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62140133-beb2-49fc-a4c9-4864ed0dd580-maas-api-xks-ca" pod "62140133-beb2-49fc-a4c9-4864ed0dd580" (UID: "62140133-beb2-49fc-a4c9-4864ed0dd580"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:32:45.474931 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:45.474890 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62140133-beb2-49fc-a4c9-4864ed0dd580-kube-api-access-65htw" pod "62140133-beb2-49fc-a4c9-4864ed0dd580" (UID: "62140133-beb2-49fc-a4c9-4864ed0dd580"). InnerVolumeSpecName "kube-api-access-65htw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:32:45.574348 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:45.574316 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65htw\" (UniqueName: \"kubernetes.io/projected/62140133-beb2-49fc-a4c9-4864ed0dd580-kube-api-access-65htw\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:45.574348 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:45.574345 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62140133-beb2-49fc-a4c9-4864ed0dd580-maas-api-xks-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:46.166669 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:46.166642 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa82603-jr9rb" Oct 31 00:32:46.166870 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:46.166667 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa82603-jr9rb" event={"ID":"62140133-beb2-49fc-a4c9-4864ed0dd580","Type":"ContainerDied","Data":"504d89b9f73b3fbc39b10dc0356d84b47d41fbf51fdfff09bf5e9f92188edca9"} Oct 31 00:32:46.166870 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:46.166723 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="504d89b9f73b3fbc39b10dc0356d84b47d41fbf51fdfff09bf5e9f92188edca9" Oct 31 00:32:46.168151 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:46.168133 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14eec3f8-z5kkm" Oct 31 00:32:46.168269 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:46.168158 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14eec3f8-z5kkm" event={"ID":"d22f0a86-6e20-4f79-8104-b47a4431d338","Type":"ContainerDied","Data":"3f9c29e8a5dbe9dbd104745c3713b6b5b4ae87d291f14b72d508c8ae389a512e"} Oct 31 00:32:46.168269 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:46.168209 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3f9c29e8a5dbe9dbd104745c3713b6b5b4ae87d291f14b72d508c8ae389a512e" Oct 31 00:32:49.973088 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:49.973051 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-79tk8"] Oct 31 00:32:50.070812 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:50.070776 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mk8f6"] Oct 31 00:32:50.071011 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:50.070984 5222 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mk8f6" podUID="a83f0bf9-4b49-4e5f-9e11-838a1307907e" containerName="istio-proxy" containerID="cri-o://807b78c6972e752e3d52d4597ab34a5ca80314102260c84d495fda8a6f85c2aa" gracePeriod=30 Oct 31 00:32:52.743946 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:52.743906 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-101b9e/distinct-w0-101b9e-kserve-6dcc4cc94f-kwv9w"] Oct 31 00:32:52.744642 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:52.744625 5222 cpu_manager.go:414] "containerMap: removing container" podUID="d22f0a86-6e20-4f79-8104-b47a4431d338" containerName="revoke-keys" Oct 31 00:32:52.744708 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:52.744645 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d22f0a86-6e20-4f79-8104-b47a4431d338" containerName="revoke-keys" Oct 31 00:32:52.744708 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:52.744689 5222 cpu_manager.go:414] "containerMap: removing container" podUID="62140133-beb2-49fc-a4c9-4864ed0dd580" containerName="revoke-keys" Oct 31 00:32:52.744708 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:52.744698 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62140133-beb2-49fc-a4c9-4864ed0dd580" containerName="revoke-keys" Oct 31 00:32:52.744837 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:52.744825 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="d22f0a86-6e20-4f79-8104-b47a4431d338" containerName="revoke-keys" Oct 31 00:32:52.744873 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:52.744846 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="62140133-beb2-49fc-a4c9-4864ed0dd580" containerName="revoke-keys" Oct 31 00:32:52.748976 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:52.748954 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-101b9e/distinct-w0-101b9e-kserve-6dcc4cc94f-kwv9w" Oct 31 00:32:52.864862 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:52.864823 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-101b9e\"/\"distinct-w0-101b9e-kserve-self-signed-certs\"" Oct 31 00:32:52.867990 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:52.867962 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-101b9e/distinct-w0-101b9e-kserve-6dcc4cc94f-kwv9w"] Oct 31 00:32:52.872384 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:52.872337 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4598\" (UniqueName: \"kubernetes.io/projected/73487e30-1bc6-49b2-b07b-1b004137232e-kube-api-access-s4598\") pod \"distinct-w0-101b9e-kserve-6dcc4cc94f-kwv9w\" (UID: \"73487e30-1bc6-49b2-b07b-1b004137232e\") " pod="e2e-models-e2e-worker-w0-101b9e/distinct-w0-101b9e-kserve-6dcc4cc94f-kwv9w" Oct 31 00:32:52.872508 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:52.872484 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73487e30-1bc6-49b2-b07b-1b004137232e-tls-certs\") pod \"distinct-w0-101b9e-kserve-6dcc4cc94f-kwv9w\" (UID: \"73487e30-1bc6-49b2-b07b-1b004137232e\") " pod="e2e-models-e2e-worker-w0-101b9e/distinct-w0-101b9e-kserve-6dcc4cc94f-kwv9w" Oct 31 00:32:52.872678 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:52.872645 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73487e30-1bc6-49b2-b07b-1b004137232e-dshm\") pod \"distinct-w0-101b9e-kserve-6dcc4cc94f-kwv9w\" (UID: \"73487e30-1bc6-49b2-b07b-1b004137232e\") " pod="e2e-models-e2e-worker-w0-101b9e/distinct-w0-101b9e-kserve-6dcc4cc94f-kwv9w" Oct 31 00:32:52.872807 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:52.872692 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73487e30-1bc6-49b2-b07b-1b004137232e-model-cache\") pod \"distinct-w0-101b9e-kserve-6dcc4cc94f-kwv9w\" (UID: \"73487e30-1bc6-49b2-b07b-1b004137232e\") " pod="e2e-models-e2e-worker-w0-101b9e/distinct-w0-101b9e-kserve-6dcc4cc94f-kwv9w" Oct 31 00:32:52.872807 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:52.872767 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73487e30-1bc6-49b2-b07b-1b004137232e-home\") pod \"distinct-w0-101b9e-kserve-6dcc4cc94f-kwv9w\" (UID: \"73487e30-1bc6-49b2-b07b-1b004137232e\") " pod="e2e-models-e2e-worker-w0-101b9e/distinct-w0-101b9e-kserve-6dcc4cc94f-kwv9w" Oct 31 00:32:52.872905 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:52.872824 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73487e30-1bc6-49b2-b07b-1b004137232e-tmp-dir\") pod \"distinct-w0-101b9e-kserve-6dcc4cc94f-kwv9w\" (UID: \"73487e30-1bc6-49b2-b07b-1b004137232e\") " pod="e2e-models-e2e-worker-w0-101b9e/distinct-w0-101b9e-kserve-6dcc4cc94f-kwv9w" Oct 31 00:32:52.973880 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:52.973840 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73487e30-1bc6-49b2-b07b-1b004137232e-dshm\") pod \"distinct-w0-101b9e-kserve-6dcc4cc94f-kwv9w\" (UID: \"73487e30-1bc6-49b2-b07b-1b004137232e\") " pod="e2e-models-e2e-worker-w0-101b9e/distinct-w0-101b9e-kserve-6dcc4cc94f-kwv9w" Oct 31 00:32:52.973880 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:52.973885 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73487e30-1bc6-49b2-b07b-1b004137232e-model-cache\") pod \"distinct-w0-101b9e-kserve-6dcc4cc94f-kwv9w\" (UID: \"73487e30-1bc6-49b2-b07b-1b004137232e\") " pod="e2e-models-e2e-worker-w0-101b9e/distinct-w0-101b9e-kserve-6dcc4cc94f-kwv9w" Oct 31 00:32:52.974140 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:52.973903 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73487e30-1bc6-49b2-b07b-1b004137232e-home\") pod \"distinct-w0-101b9e-kserve-6dcc4cc94f-kwv9w\" (UID: \"73487e30-1bc6-49b2-b07b-1b004137232e\") " pod="e2e-models-e2e-worker-w0-101b9e/distinct-w0-101b9e-kserve-6dcc4cc94f-kwv9w" Oct 31 00:32:52.974140 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:52.973934 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73487e30-1bc6-49b2-b07b-1b004137232e-tmp-dir\") pod \"distinct-w0-101b9e-kserve-6dcc4cc94f-kwv9w\" (UID: \"73487e30-1bc6-49b2-b07b-1b004137232e\") " pod="e2e-models-e2e-worker-w0-101b9e/distinct-w0-101b9e-kserve-6dcc4cc94f-kwv9w" Oct 31 00:32:52.974140 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:52.973979 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4598\" (UniqueName: \"kubernetes.io/projected/73487e30-1bc6-49b2-b07b-1b004137232e-kube-api-access-s4598\") pod \"distinct-w0-101b9e-kserve-6dcc4cc94f-kwv9w\" (UID: \"73487e30-1bc6-49b2-b07b-1b004137232e\") " pod="e2e-models-e2e-worker-w0-101b9e/distinct-w0-101b9e-kserve-6dcc4cc94f-kwv9w" Oct 31 00:32:52.974140 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:52.974032 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73487e30-1bc6-49b2-b07b-1b004137232e-tls-certs\") pod \"distinct-w0-101b9e-kserve-6dcc4cc94f-kwv9w\" (UID: \"73487e30-1bc6-49b2-b07b-1b004137232e\") " pod="e2e-models-e2e-worker-w0-101b9e/distinct-w0-101b9e-kserve-6dcc4cc94f-kwv9w" Oct 31 00:32:52.974346 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:52.974296 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73487e30-1bc6-49b2-b07b-1b004137232e-model-cache\") pod \"distinct-w0-101b9e-kserve-6dcc4cc94f-kwv9w\" (UID: \"73487e30-1bc6-49b2-b07b-1b004137232e\") " pod="e2e-models-e2e-worker-w0-101b9e/distinct-w0-101b9e-kserve-6dcc4cc94f-kwv9w" Oct 31 00:32:52.974447 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:52.974396 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73487e30-1bc6-49b2-b07b-1b004137232e-home\") pod \"distinct-w0-101b9e-kserve-6dcc4cc94f-kwv9w\" (UID: \"73487e30-1bc6-49b2-b07b-1b004137232e\") " pod="e2e-models-e2e-worker-w0-101b9e/distinct-w0-101b9e-kserve-6dcc4cc94f-kwv9w" Oct 31 00:32:52.974512 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:52.974490 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73487e30-1bc6-49b2-b07b-1b004137232e-tmp-dir\") pod \"distinct-w0-101b9e-kserve-6dcc4cc94f-kwv9w\" (UID: \"73487e30-1bc6-49b2-b07b-1b004137232e\") " pod="e2e-models-e2e-worker-w0-101b9e/distinct-w0-101b9e-kserve-6dcc4cc94f-kwv9w" Oct 31 00:32:52.976406 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:52.976358 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73487e30-1bc6-49b2-b07b-1b004137232e-dshm\") pod \"distinct-w0-101b9e-kserve-6dcc4cc94f-kwv9w\" (UID: \"73487e30-1bc6-49b2-b07b-1b004137232e\") " pod="e2e-models-e2e-worker-w0-101b9e/distinct-w0-101b9e-kserve-6dcc4cc94f-kwv9w" Oct 31 00:32:52.976653 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:52.976637 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73487e30-1bc6-49b2-b07b-1b004137232e-tls-certs\") pod \"distinct-w0-101b9e-kserve-6dcc4cc94f-kwv9w\" (UID: \"73487e30-1bc6-49b2-b07b-1b004137232e\") " pod="e2e-models-e2e-worker-w0-101b9e/distinct-w0-101b9e-kserve-6dcc4cc94f-kwv9w" Oct 31 00:32:53.037860 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:53.037825 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4598\" (UniqueName: \"kubernetes.io/projected/73487e30-1bc6-49b2-b07b-1b004137232e-kube-api-access-s4598\") pod \"distinct-w0-101b9e-kserve-6dcc4cc94f-kwv9w\" (UID: \"73487e30-1bc6-49b2-b07b-1b004137232e\") " pod="e2e-models-e2e-worker-w0-101b9e/distinct-w0-101b9e-kserve-6dcc4cc94f-kwv9w" Oct 31 00:32:53.061441 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:53.061400 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-101b9e/distinct-w0-101b9e-kserve-6dcc4cc94f-kwv9w" Oct 31 00:32:53.460046 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:32:53.460014 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod73487e30_1bc6_49b2_b07b_1b004137232e.slice/crio-507449e4d357c1c7f493084b6278c275c890cab59d60a41488cbc6669d87a312 WatchSource:0}: Error finding container 507449e4d357c1c7f493084b6278c275c890cab59d60a41488cbc6669d87a312: Status 404 returned error can't find the container with id 507449e4d357c1c7f493084b6278c275c890cab59d60a41488cbc6669d87a312 Oct 31 00:32:53.463533 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:53.463513 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-101b9e/distinct-w0-101b9e-kserve-6dcc4cc94f-kwv9w"] Oct 31 00:32:54.220561 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:54.220527 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-101b9e/distinct-w0-101b9e-kserve-6dcc4cc94f-kwv9w" event={"ID":"73487e30-1bc6-49b2-b07b-1b004137232e","Type":"ContainerStarted","Data":"edb07311cd170b00ceaadc005648108ea352ce910f7179de453809f9ebc1ad5d"} Oct 31 00:32:54.220561 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:54.220567 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-101b9e/distinct-w0-101b9e-kserve-6dcc4cc94f-kwv9w" event={"ID":"73487e30-1bc6-49b2-b07b-1b004137232e","Type":"ContainerStarted","Data":"507449e4d357c1c7f493084b6278c275c890cab59d60a41488cbc6669d87a312"} Oct 31 00:32:54.356601 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:54.356535 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-101b9e/distinct-w0-101b9e-kserve-6dcc4cc94f-kwv9w" podStartSLOduration=2.356520437 podStartE2EDuration="2.356520437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:32:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:32:54.354202 +0000 UTC m=+1066.825371749" watchObservedRunningTime="2026-10-31 00:32:54.356520437 +0000 UTC m=+1066.827690190" Oct 31 00:32:55.062242 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:55.062186 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-101b9e/distinct-w0-101b9e-kserve-6dcc4cc94f-kwv9w" Oct 31 00:32:55.228871 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:55.228842 5222 generic.go:356] "Generic (PLEG): container finished" podID="a83f0bf9-4b49-4e5f-9e11-838a1307907e" containerID="807b78c6972e752e3d52d4597ab34a5ca80314102260c84d495fda8a6f85c2aa" exitCode=0 Oct 31 00:32:55.229274 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:55.228926 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mk8f6" event={"ID":"a83f0bf9-4b49-4e5f-9e11-838a1307907e","Type":"ContainerDied","Data":"807b78c6972e752e3d52d4597ab34a5ca80314102260c84d495fda8a6f85c2aa"} Oct 31 00:32:55.338187 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:55.338165 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mk8f6" Oct 31 00:32:55.504070 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:55.504037 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a83f0bf9-4b49-4e5f-9e11-838a1307907e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a83f0bf9-4b49-4e5f-9e11-838a1307907e-istio-envoy\") pod \"a83f0bf9-4b49-4e5f-9e11-838a1307907e\" (UID: \"a83f0bf9-4b49-4e5f-9e11-838a1307907e\") " Oct 31 00:32:55.504272 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:55.504105 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a83f0bf9-4b49-4e5f-9e11-838a1307907e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a83f0bf9-4b49-4e5f-9e11-838a1307907e-istio-data\") pod \"a83f0bf9-4b49-4e5f-9e11-838a1307907e\" (UID: \"a83f0bf9-4b49-4e5f-9e11-838a1307907e\") " Oct 31 00:32:55.504272 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:55.504174 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a83f0bf9-4b49-4e5f-9e11-838a1307907e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a83f0bf9-4b49-4e5f-9e11-838a1307907e-istio-podinfo\") pod \"a83f0bf9-4b49-4e5f-9e11-838a1307907e\" (UID: \"a83f0bf9-4b49-4e5f-9e11-838a1307907e\") " Oct 31 00:32:55.504272 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:55.504238 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a83f0bf9-4b49-4e5f-9e11-838a1307907e-istio-token\" (UniqueName: \"kubernetes.io/projected/a83f0bf9-4b49-4e5f-9e11-838a1307907e-istio-token\") pod \"a83f0bf9-4b49-4e5f-9e11-838a1307907e\" (UID: \"a83f0bf9-4b49-4e5f-9e11-838a1307907e\") " Oct 31 00:32:55.504483 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:55.504276 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a83f0bf9-4b49-4e5f-9e11-838a1307907e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a83f0bf9-4b49-4e5f-9e11-838a1307907e-workload-socket\") pod \"a83f0bf9-4b49-4e5f-9e11-838a1307907e\" (UID: \"a83f0bf9-4b49-4e5f-9e11-838a1307907e\") " Oct 31 00:32:55.504483 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:55.504353 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a83f0bf9-4b49-4e5f-9e11-838a1307907e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a83f0bf9-4b49-4e5f-9e11-838a1307907e-istiod-ca-cert\") pod \"a83f0bf9-4b49-4e5f-9e11-838a1307907e\" (UID: \"a83f0bf9-4b49-4e5f-9e11-838a1307907e\") " Oct 31 00:32:55.504483 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:55.504401 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a83f0bf9-4b49-4e5f-9e11-838a1307907e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a83f0bf9-4b49-4e5f-9e11-838a1307907e-workload-certs\") pod \"a83f0bf9-4b49-4e5f-9e11-838a1307907e\" (UID: \"a83f0bf9-4b49-4e5f-9e11-838a1307907e\") " Oct 31 00:32:55.504483 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:55.504418 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a83f0bf9-4b49-4e5f-9e11-838a1307907e-istio-data" pod "a83f0bf9-4b49-4e5f-9e11-838a1307907e" (UID: "a83f0bf9-4b49-4e5f-9e11-838a1307907e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:32:55.504483 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:55.504450 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a83f0bf9-4b49-4e5f-9e11-838a1307907e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a83f0bf9-4b49-4e5f-9e11-838a1307907e-credential-socket\") pod \"a83f0bf9-4b49-4e5f-9e11-838a1307907e\" (UID: \"a83f0bf9-4b49-4e5f-9e11-838a1307907e\") " Oct 31 00:32:55.504726 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:55.504493 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a83f0bf9-4b49-4e5f-9e11-838a1307907e-kube-api-access-vl8hm\" (UniqueName: \"kubernetes.io/projected/a83f0bf9-4b49-4e5f-9e11-838a1307907e-kube-api-access-vl8hm\") pod \"a83f0bf9-4b49-4e5f-9e11-838a1307907e\" (UID: \"a83f0bf9-4b49-4e5f-9e11-838a1307907e\") " Oct 31 00:32:55.504782 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:55.504717 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a83f0bf9-4b49-4e5f-9e11-838a1307907e-credential-socket" pod "a83f0bf9-4b49-4e5f-9e11-838a1307907e" (UID: "a83f0bf9-4b49-4e5f-9e11-838a1307907e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:32:55.504782 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:55.504728 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a83f0bf9-4b49-4e5f-9e11-838a1307907e-workload-certs" pod "a83f0bf9-4b49-4e5f-9e11-838a1307907e" (UID: "a83f0bf9-4b49-4e5f-9e11-838a1307907e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:32:55.504782 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:55.504764 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a83f0bf9-4b49-4e5f-9e11-838a1307907e-istiod-ca-cert" pod "a83f0bf9-4b49-4e5f-9e11-838a1307907e" (UID: "a83f0bf9-4b49-4e5f-9e11-838a1307907e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:32:55.505019 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:55.505000 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a83f0bf9-4b49-4e5f-9e11-838a1307907e-workload-socket" pod "a83f0bf9-4b49-4e5f-9e11-838a1307907e" (UID: "a83f0bf9-4b49-4e5f-9e11-838a1307907e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:32:55.505110 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:55.505016 5222 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a83f0bf9-4b49-4e5f-9e11-838a1307907e-istio-data\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:55.505110 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:55.505039 5222 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a83f0bf9-4b49-4e5f-9e11-838a1307907e-istiod-ca-cert\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:55.505110 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:55.505054 5222 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a83f0bf9-4b49-4e5f-9e11-838a1307907e-workload-certs\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:55.505110 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:55.505067 5222 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a83f0bf9-4b49-4e5f-9e11-838a1307907e-credential-socket\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:55.506712 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:55.506684 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a83f0bf9-4b49-4e5f-9e11-838a1307907e-istio-envoy" pod "a83f0bf9-4b49-4e5f-9e11-838a1307907e" (UID: "a83f0bf9-4b49-4e5f-9e11-838a1307907e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:32:55.507200 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:55.507173 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a83f0bf9-4b49-4e5f-9e11-838a1307907e-istio-podinfo" pod "a83f0bf9-4b49-4e5f-9e11-838a1307907e" (UID: "a83f0bf9-4b49-4e5f-9e11-838a1307907e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:32:55.507200 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:55.507191 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a83f0bf9-4b49-4e5f-9e11-838a1307907e-kube-api-access-vl8hm" pod "a83f0bf9-4b49-4e5f-9e11-838a1307907e" (UID: "a83f0bf9-4b49-4e5f-9e11-838a1307907e"). InnerVolumeSpecName "kube-api-access-vl8hm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:32:55.507721 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:55.507695 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a83f0bf9-4b49-4e5f-9e11-838a1307907e-istio-token" pod "a83f0bf9-4b49-4e5f-9e11-838a1307907e" (UID: "a83f0bf9-4b49-4e5f-9e11-838a1307907e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:32:55.606274 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:55.606160 5222 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a83f0bf9-4b49-4e5f-9e11-838a1307907e-istio-podinfo\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:55.606274 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:55.606205 5222 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a83f0bf9-4b49-4e5f-9e11-838a1307907e-istio-token\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:55.606274 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:55.606220 5222 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a83f0bf9-4b49-4e5f-9e11-838a1307907e-workload-socket\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:55.606274 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:55.606233 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vl8hm\" (UniqueName: \"kubernetes.io/projected/a83f0bf9-4b49-4e5f-9e11-838a1307907e-kube-api-access-vl8hm\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:55.606274 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:55.606245 5222 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a83f0bf9-4b49-4e5f-9e11-838a1307907e-istio-envoy\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:56.236851 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:56.236817 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mk8f6" Oct 31 00:32:56.236851 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:56.236844 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mk8f6" event={"ID":"a83f0bf9-4b49-4e5f-9e11-838a1307907e","Type":"ContainerDied","Data":"2aa1b90556ac90a3e96b0ae1931b50723763b817b21b205b8819fad996481da0"} Oct 31 00:32:56.237358 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:56.236900 5222 scope.go:122] "RemoveContainer" containerID="807b78c6972e752e3d52d4597ab34a5ca80314102260c84d495fda8a6f85c2aa" Oct 31 00:32:56.367957 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:56.367914 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-j4tb5"] Oct 31 00:32:56.368634 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:56.368614 5222 cpu_manager.go:414] "containerMap: removing container" podUID="a83f0bf9-4b49-4e5f-9e11-838a1307907e" containerName="istio-proxy" Oct 31 00:32:56.368634 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:56.368635 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a83f0bf9-4b49-4e5f-9e11-838a1307907e" containerName="istio-proxy" Oct 31 00:32:56.368768 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:56.368709 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="a83f0bf9-4b49-4e5f-9e11-838a1307907e" containerName="istio-proxy" Oct 31 00:32:56.373716 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:56.373700 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-j4tb5" Oct 31 00:32:56.429364 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:56.429330 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-da9d07-openshift-default-dockercfg-ckgpr\"" Oct 31 00:32:56.431312 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:56.431286 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-j4tb5"] Oct 31 00:32:56.461616 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:56.461586 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mk8f6"] Oct 31 00:32:56.481728 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:56.481691 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mk8f6"] Oct 31 00:32:56.516503 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:56.516470 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d359a4aa-8996-4615-9b33-fccc33762a6a-workload-certs\") pod \"e2e-worker-w4-da9d07-openshift-default-798d9948bc-j4tb5\" (UID: \"d359a4aa-8996-4615-9b33-fccc33762a6a\") " pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-j4tb5" Oct 31 00:32:56.516672 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:56.516511 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d359a4aa-8996-4615-9b33-fccc33762a6a-istio-token\") pod \"e2e-worker-w4-da9d07-openshift-default-798d9948bc-j4tb5\" (UID: \"d359a4aa-8996-4615-9b33-fccc33762a6a\") " pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-j4tb5" Oct 31 00:32:56.516672 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:56.516536 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74htg\" (UniqueName: \"kubernetes.io/projected/d359a4aa-8996-4615-9b33-fccc33762a6a-kube-api-access-74htg\") pod \"e2e-worker-w4-da9d07-openshift-default-798d9948bc-j4tb5\" (UID: \"d359a4aa-8996-4615-9b33-fccc33762a6a\") " pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-j4tb5" Oct 31 00:32:56.516672 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:56.516648 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d359a4aa-8996-4615-9b33-fccc33762a6a-credential-socket\") pod \"e2e-worker-w4-da9d07-openshift-default-798d9948bc-j4tb5\" (UID: \"d359a4aa-8996-4615-9b33-fccc33762a6a\") " pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-j4tb5" Oct 31 00:32:56.516783 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:56.516712 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d359a4aa-8996-4615-9b33-fccc33762a6a-istio-envoy\") pod \"e2e-worker-w4-da9d07-openshift-default-798d9948bc-j4tb5\" (UID: \"d359a4aa-8996-4615-9b33-fccc33762a6a\") " pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-j4tb5" Oct 31 00:32:56.516817 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:56.516796 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d359a4aa-8996-4615-9b33-fccc33762a6a-workload-socket\") pod \"e2e-worker-w4-da9d07-openshift-default-798d9948bc-j4tb5\" (UID: \"d359a4aa-8996-4615-9b33-fccc33762a6a\") " pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-j4tb5" Oct 31 00:32:56.516878 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:56.516860 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d359a4aa-8996-4615-9b33-fccc33762a6a-istio-podinfo\") pod \"e2e-worker-w4-da9d07-openshift-default-798d9948bc-j4tb5\" (UID: \"d359a4aa-8996-4615-9b33-fccc33762a6a\") " pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-j4tb5" Oct 31 00:32:56.516916 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:56.516899 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d359a4aa-8996-4615-9b33-fccc33762a6a-istiod-ca-cert\") pod \"e2e-worker-w4-da9d07-openshift-default-798d9948bc-j4tb5\" (UID: \"d359a4aa-8996-4615-9b33-fccc33762a6a\") " pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-j4tb5" Oct 31 00:32:56.517020 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:56.517004 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d359a4aa-8996-4615-9b33-fccc33762a6a-istio-data\") pod \"e2e-worker-w4-da9d07-openshift-default-798d9948bc-j4tb5\" (UID: \"d359a4aa-8996-4615-9b33-fccc33762a6a\") " pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-j4tb5" Oct 31 00:32:56.618118 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:56.618080 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d359a4aa-8996-4615-9b33-fccc33762a6a-istio-data\") pod \"e2e-worker-w4-da9d07-openshift-default-798d9948bc-j4tb5\" (UID: \"d359a4aa-8996-4615-9b33-fccc33762a6a\") " pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-j4tb5" Oct 31 00:32:56.618327 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:56.618134 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d359a4aa-8996-4615-9b33-fccc33762a6a-workload-certs\") pod \"e2e-worker-w4-da9d07-openshift-default-798d9948bc-j4tb5\" (UID: \"d359a4aa-8996-4615-9b33-fccc33762a6a\") " pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-j4tb5" Oct 31 00:32:56.618327 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:56.618164 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d359a4aa-8996-4615-9b33-fccc33762a6a-istio-token\") pod \"e2e-worker-w4-da9d07-openshift-default-798d9948bc-j4tb5\" (UID: \"d359a4aa-8996-4615-9b33-fccc33762a6a\") " pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-j4tb5" Oct 31 00:32:56.618327 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:56.618187 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74htg\" (UniqueName: \"kubernetes.io/projected/d359a4aa-8996-4615-9b33-fccc33762a6a-kube-api-access-74htg\") pod \"e2e-worker-w4-da9d07-openshift-default-798d9948bc-j4tb5\" (UID: \"d359a4aa-8996-4615-9b33-fccc33762a6a\") " pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-j4tb5" Oct 31 00:32:56.618327 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:56.618209 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d359a4aa-8996-4615-9b33-fccc33762a6a-credential-socket\") pod \"e2e-worker-w4-da9d07-openshift-default-798d9948bc-j4tb5\" (UID: \"d359a4aa-8996-4615-9b33-fccc33762a6a\") " pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-j4tb5" Oct 31 00:32:56.618327 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:56.618266 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d359a4aa-8996-4615-9b33-fccc33762a6a-istio-envoy\") pod \"e2e-worker-w4-da9d07-openshift-default-798d9948bc-j4tb5\" (UID: \"d359a4aa-8996-4615-9b33-fccc33762a6a\") " pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-j4tb5" Oct 31 00:32:56.618327 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:56.618309 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d359a4aa-8996-4615-9b33-fccc33762a6a-workload-socket\") pod \"e2e-worker-w4-da9d07-openshift-default-798d9948bc-j4tb5\" (UID: \"d359a4aa-8996-4615-9b33-fccc33762a6a\") " pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-j4tb5" Oct 31 00:32:56.618662 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:56.618359 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d359a4aa-8996-4615-9b33-fccc33762a6a-istio-podinfo\") pod \"e2e-worker-w4-da9d07-openshift-default-798d9948bc-j4tb5\" (UID: \"d359a4aa-8996-4615-9b33-fccc33762a6a\") " pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-j4tb5" Oct 31 00:32:56.618662 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:56.618409 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d359a4aa-8996-4615-9b33-fccc33762a6a-istiod-ca-cert\") pod \"e2e-worker-w4-da9d07-openshift-default-798d9948bc-j4tb5\" (UID: \"d359a4aa-8996-4615-9b33-fccc33762a6a\") " pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-j4tb5" Oct 31 00:32:56.618662 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:56.618575 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d359a4aa-8996-4615-9b33-fccc33762a6a-istio-data\") pod \"e2e-worker-w4-da9d07-openshift-default-798d9948bc-j4tb5\" (UID: \"d359a4aa-8996-4615-9b33-fccc33762a6a\") " pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-j4tb5" Oct 31 00:32:56.618662 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:56.618625 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d359a4aa-8996-4615-9b33-fccc33762a6a-workload-certs\") pod \"e2e-worker-w4-da9d07-openshift-default-798d9948bc-j4tb5\" (UID: \"d359a4aa-8996-4615-9b33-fccc33762a6a\") " pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-j4tb5" Oct 31 00:32:56.618906 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:56.618883 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d359a4aa-8996-4615-9b33-fccc33762a6a-credential-socket\") pod \"e2e-worker-w4-da9d07-openshift-default-798d9948bc-j4tb5\" (UID: \"d359a4aa-8996-4615-9b33-fccc33762a6a\") " pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-j4tb5" Oct 31 00:32:56.619039 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:56.619019 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d359a4aa-8996-4615-9b33-fccc33762a6a-workload-socket\") pod \"e2e-worker-w4-da9d07-openshift-default-798d9948bc-j4tb5\" (UID: \"d359a4aa-8996-4615-9b33-fccc33762a6a\") " pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-j4tb5" Oct 31 00:32:56.619264 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:56.619238 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d359a4aa-8996-4615-9b33-fccc33762a6a-istiod-ca-cert\") pod \"e2e-worker-w4-da9d07-openshift-default-798d9948bc-j4tb5\" (UID: \"d359a4aa-8996-4615-9b33-fccc33762a6a\") " pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-j4tb5" Oct 31 00:32:56.621158 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:56.621135 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d359a4aa-8996-4615-9b33-fccc33762a6a-istio-envoy\") pod \"e2e-worker-w4-da9d07-openshift-default-798d9948bc-j4tb5\" (UID: \"d359a4aa-8996-4615-9b33-fccc33762a6a\") " pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-j4tb5" Oct 31 00:32:56.621575 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:56.621558 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d359a4aa-8996-4615-9b33-fccc33762a6a-istio-podinfo\") pod \"e2e-worker-w4-da9d07-openshift-default-798d9948bc-j4tb5\" (UID: \"d359a4aa-8996-4615-9b33-fccc33762a6a\") " pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-j4tb5" Oct 31 00:32:56.681531 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:56.681494 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74htg\" (UniqueName: \"kubernetes.io/projected/d359a4aa-8996-4615-9b33-fccc33762a6a-kube-api-access-74htg\") pod \"e2e-worker-w4-da9d07-openshift-default-798d9948bc-j4tb5\" (UID: \"d359a4aa-8996-4615-9b33-fccc33762a6a\") " pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-j4tb5" Oct 31 00:32:56.686806 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:56.686777 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d359a4aa-8996-4615-9b33-fccc33762a6a-istio-token\") pod \"e2e-worker-w4-da9d07-openshift-default-798d9948bc-j4tb5\" (UID: \"d359a4aa-8996-4615-9b33-fccc33762a6a\") " pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-j4tb5" Oct 31 00:32:56.687552 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:56.687526 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-j4tb5" Oct 31 00:32:57.119974 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:32:57.119944 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd359a4aa_8996_4615_9b33_fccc33762a6a.slice/crio-0b015d3eff42faf1a44f398a3aa0bb686ca4ae687ee23c68ed225263f98d571f WatchSource:0}: Error finding container 0b015d3eff42faf1a44f398a3aa0bb686ca4ae687ee23c68ed225263f98d571f: Status 404 returned error can't find the container with id 0b015d3eff42faf1a44f398a3aa0bb686ca4ae687ee23c68ed225263f98d571f Oct 31 00:32:57.122527 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:57.122488 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:32:57.122641 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:57.122559 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:32:57.122641 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:57.122592 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:32:57.123856 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:57.123833 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-j4tb5"] Oct 31 00:32:57.246691 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:57.246647 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-j4tb5" event={"ID":"d359a4aa-8996-4615-9b33-fccc33762a6a","Type":"ContainerStarted","Data":"5d7a265874f1ee0b61e4e566f8de0fce151565afb36530586afa5e7b0790de22"} Oct 31 00:32:57.247167 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:57.246717 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-j4tb5" event={"ID":"d359a4aa-8996-4615-9b33-fccc33762a6a","Type":"ContainerStarted","Data":"0b015d3eff42faf1a44f398a3aa0bb686ca4ae687ee23c68ed225263f98d571f"} Oct 31 00:32:57.373202 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:57.373087 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-j4tb5" podStartSLOduration=1.373065189 podStartE2EDuration="1.373065189s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:32:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:32:57.369968317 +0000 UTC m=+1069.841138072" watchObservedRunningTime="2026-10-31 00:32:57.373065189 +0000 UTC m=+1069.844234940" Oct 31 00:32:57.664316 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:57.664207 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a83f0bf9-4b49-4e5f-9e11-838a1307907e" path="/var/lib/kubelet/pods/a83f0bf9-4b49-4e5f-9e11-838a1307907e/volumes" Oct 31 00:32:57.688088 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:57.688044 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-j4tb5" Oct 31 00:32:58.690056 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:58.690021 5222 patch_prober.go:29] interesting pod/e2e-worker-w4-da9d07-openshift-default-798d9948bc-j4tb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 31 00:32:58.690456 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:58.690076 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-j4tb5" podUID="d359a4aa-8996-4615-9b33-fccc33762a6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 31 00:32:59.688331 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:59.688291 5222 patch_prober.go:29] interesting pod/e2e-worker-w4-da9d07-openshift-default-798d9948bc-j4tb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 31 00:32:59.688571 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:32:59.688349 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-j4tb5" podUID="d359a4aa-8996-4615-9b33-fccc33762a6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 31 00:33:00.688625 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:00.688590 5222 patch_prober.go:29] interesting pod/e2e-worker-w4-da9d07-openshift-default-798d9948bc-j4tb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 31 00:33:00.689038 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:00.688649 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-j4tb5" podUID="d359a4aa-8996-4615-9b33-fccc33762a6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 31 00:33:01.688144 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:01.688104 5222 patch_prober.go:29] interesting pod/e2e-worker-w4-da9d07-openshift-default-798d9948bc-j4tb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 31 00:33:01.688444 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:01.688173 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-j4tb5" podUID="d359a4aa-8996-4615-9b33-fccc33762a6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 31 00:33:02.688663 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:02.688628 5222 patch_prober.go:29] interesting pod/e2e-worker-w4-da9d07-openshift-default-798d9948bc-j4tb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 31 00:33:02.689087 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:02.688694 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-j4tb5" podUID="d359a4aa-8996-4615-9b33-fccc33762a6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 31 00:33:03.061623 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:03.061587 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-101b9e/distinct-w0-101b9e-kserve-6dcc4cc94f-kwv9w" Oct 31 00:33:03.074618 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:03.074588 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-101b9e/distinct-w0-101b9e-kserve-6dcc4cc94f-kwv9w" Oct 31 00:33:03.311893 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:03.311812 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-101b9e/distinct-w0-101b9e-kserve-6dcc4cc94f-kwv9w" Oct 31 00:33:03.814823 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:03.814792 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-j4tb5" Oct 31 00:33:03.815330 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:03.815285 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-j4tb5" Oct 31 00:33:03.816112 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:03.816094 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-j4tb5" Oct 31 00:33:05.890140 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:33:05.890103 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 00:33:11.464023 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:11.463989 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-1e5e75/distinct-w3-1e5e75-kserve-7b55d58bbb-hchmb"] Oct 31 00:33:11.469465 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:11.469443 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-1e5e75/distinct-w3-1e5e75-kserve-7b55d58bbb-hchmb" Oct 31 00:33:11.523546 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:11.523516 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-1e5e75/distinct-w3-1e5e75-kserve-7b55d58bbb-hchmb"] Oct 31 00:33:11.570639 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:11.570601 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-1e5e75\"/\"distinct-w3-1e5e75-kserve-self-signed-certs\"" Oct 31 00:33:11.610169 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:11.610131 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79ffb21b-10c3-4174-89be-05ba9e852a98-tmp-dir\") pod \"distinct-w3-1e5e75-kserve-7b55d58bbb-hchmb\" (UID: \"79ffb21b-10c3-4174-89be-05ba9e852a98\") " pod="e2e-models-e2e-worker-w3-1e5e75/distinct-w3-1e5e75-kserve-7b55d58bbb-hchmb" Oct 31 00:33:11.610596 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:11.610570 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79ffb21b-10c3-4174-89be-05ba9e852a98-dshm\") pod \"distinct-w3-1e5e75-kserve-7b55d58bbb-hchmb\" (UID: \"79ffb21b-10c3-4174-89be-05ba9e852a98\") " pod="e2e-models-e2e-worker-w3-1e5e75/distinct-w3-1e5e75-kserve-7b55d58bbb-hchmb" Oct 31 00:33:11.610780 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:11.610766 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79ffb21b-10c3-4174-89be-05ba9e852a98-home\") pod \"distinct-w3-1e5e75-kserve-7b55d58bbb-hchmb\" (UID: \"79ffb21b-10c3-4174-89be-05ba9e852a98\") " pod="e2e-models-e2e-worker-w3-1e5e75/distinct-w3-1e5e75-kserve-7b55d58bbb-hchmb" Oct 31 00:33:11.611078 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:11.611044 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79ffb21b-10c3-4174-89be-05ba9e852a98-model-cache\") pod \"distinct-w3-1e5e75-kserve-7b55d58bbb-hchmb\" (UID: \"79ffb21b-10c3-4174-89be-05ba9e852a98\") " pod="e2e-models-e2e-worker-w3-1e5e75/distinct-w3-1e5e75-kserve-7b55d58bbb-hchmb" Oct 31 00:33:11.611221 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:11.611201 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6blpj\" (UniqueName: \"kubernetes.io/projected/79ffb21b-10c3-4174-89be-05ba9e852a98-kube-api-access-6blpj\") pod \"distinct-w3-1e5e75-kserve-7b55d58bbb-hchmb\" (UID: \"79ffb21b-10c3-4174-89be-05ba9e852a98\") " pod="e2e-models-e2e-worker-w3-1e5e75/distinct-w3-1e5e75-kserve-7b55d58bbb-hchmb" Oct 31 00:33:11.611432 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:11.611412 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79ffb21b-10c3-4174-89be-05ba9e852a98-tls-certs\") pod \"distinct-w3-1e5e75-kserve-7b55d58bbb-hchmb\" (UID: \"79ffb21b-10c3-4174-89be-05ba9e852a98\") " pod="e2e-models-e2e-worker-w3-1e5e75/distinct-w3-1e5e75-kserve-7b55d58bbb-hchmb" Oct 31 00:33:11.713288 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:11.713253 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79ffb21b-10c3-4174-89be-05ba9e852a98-model-cache\") pod \"distinct-w3-1e5e75-kserve-7b55d58bbb-hchmb\" (UID: \"79ffb21b-10c3-4174-89be-05ba9e852a98\") " pod="e2e-models-e2e-worker-w3-1e5e75/distinct-w3-1e5e75-kserve-7b55d58bbb-hchmb" Oct 31 00:33:11.713544 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:11.713361 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6blpj\" (UniqueName: \"kubernetes.io/projected/79ffb21b-10c3-4174-89be-05ba9e852a98-kube-api-access-6blpj\") pod \"distinct-w3-1e5e75-kserve-7b55d58bbb-hchmb\" (UID: \"79ffb21b-10c3-4174-89be-05ba9e852a98\") " pod="e2e-models-e2e-worker-w3-1e5e75/distinct-w3-1e5e75-kserve-7b55d58bbb-hchmb" Oct 31 00:33:11.713544 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:11.713443 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79ffb21b-10c3-4174-89be-05ba9e852a98-tls-certs\") pod \"distinct-w3-1e5e75-kserve-7b55d58bbb-hchmb\" (UID: \"79ffb21b-10c3-4174-89be-05ba9e852a98\") " pod="e2e-models-e2e-worker-w3-1e5e75/distinct-w3-1e5e75-kserve-7b55d58bbb-hchmb" Oct 31 00:33:11.713680 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:11.713591 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79ffb21b-10c3-4174-89be-05ba9e852a98-tmp-dir\") pod \"distinct-w3-1e5e75-kserve-7b55d58bbb-hchmb\" (UID: \"79ffb21b-10c3-4174-89be-05ba9e852a98\") " pod="e2e-models-e2e-worker-w3-1e5e75/distinct-w3-1e5e75-kserve-7b55d58bbb-hchmb" Oct 31 00:33:11.713680 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:11.713626 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79ffb21b-10c3-4174-89be-05ba9e852a98-dshm\") pod \"distinct-w3-1e5e75-kserve-7b55d58bbb-hchmb\" (UID: \"79ffb21b-10c3-4174-89be-05ba9e852a98\") " pod="e2e-models-e2e-worker-w3-1e5e75/distinct-w3-1e5e75-kserve-7b55d58bbb-hchmb" Oct 31 00:33:11.713680 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:11.713658 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79ffb21b-10c3-4174-89be-05ba9e852a98-home\") pod \"distinct-w3-1e5e75-kserve-7b55d58bbb-hchmb\" (UID: \"79ffb21b-10c3-4174-89be-05ba9e852a98\") " pod="e2e-models-e2e-worker-w3-1e5e75/distinct-w3-1e5e75-kserve-7b55d58bbb-hchmb" Oct 31 00:33:11.713828 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:11.713741 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79ffb21b-10c3-4174-89be-05ba9e852a98-model-cache\") pod \"distinct-w3-1e5e75-kserve-7b55d58bbb-hchmb\" (UID: \"79ffb21b-10c3-4174-89be-05ba9e852a98\") " pod="e2e-models-e2e-worker-w3-1e5e75/distinct-w3-1e5e75-kserve-7b55d58bbb-hchmb" Oct 31 00:33:11.713962 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:11.713943 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79ffb21b-10c3-4174-89be-05ba9e852a98-tmp-dir\") pod \"distinct-w3-1e5e75-kserve-7b55d58bbb-hchmb\" (UID: \"79ffb21b-10c3-4174-89be-05ba9e852a98\") " pod="e2e-models-e2e-worker-w3-1e5e75/distinct-w3-1e5e75-kserve-7b55d58bbb-hchmb" Oct 31 00:33:11.714135 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:11.714088 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79ffb21b-10c3-4174-89be-05ba9e852a98-home\") pod \"distinct-w3-1e5e75-kserve-7b55d58bbb-hchmb\" (UID: \"79ffb21b-10c3-4174-89be-05ba9e852a98\") " pod="e2e-models-e2e-worker-w3-1e5e75/distinct-w3-1e5e75-kserve-7b55d58bbb-hchmb" Oct 31 00:33:11.715891 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:11.715867 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79ffb21b-10c3-4174-89be-05ba9e852a98-dshm\") pod \"distinct-w3-1e5e75-kserve-7b55d58bbb-hchmb\" (UID: \"79ffb21b-10c3-4174-89be-05ba9e852a98\") " pod="e2e-models-e2e-worker-w3-1e5e75/distinct-w3-1e5e75-kserve-7b55d58bbb-hchmb" Oct 31 00:33:11.716004 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:11.715987 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79ffb21b-10c3-4174-89be-05ba9e852a98-tls-certs\") pod \"distinct-w3-1e5e75-kserve-7b55d58bbb-hchmb\" (UID: \"79ffb21b-10c3-4174-89be-05ba9e852a98\") " pod="e2e-models-e2e-worker-w3-1e5e75/distinct-w3-1e5e75-kserve-7b55d58bbb-hchmb" Oct 31 00:33:11.776537 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:11.776505 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6blpj\" (UniqueName: \"kubernetes.io/projected/79ffb21b-10c3-4174-89be-05ba9e852a98-kube-api-access-6blpj\") pod \"distinct-w3-1e5e75-kserve-7b55d58bbb-hchmb\" (UID: \"79ffb21b-10c3-4174-89be-05ba9e852a98\") " pod="e2e-models-e2e-worker-w3-1e5e75/distinct-w3-1e5e75-kserve-7b55d58bbb-hchmb" Oct 31 00:33:11.782336 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:11.782313 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-1e5e75/distinct-w3-1e5e75-kserve-7b55d58bbb-hchmb" Oct 31 00:33:12.182817 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:33:12.182777 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod79ffb21b_10c3_4174_89be_05ba9e852a98.slice/crio-a39bf91a09bf8326ddb64e46e0cc3afabd1edd214d0a415b941bb924d0f96567 WatchSource:0}: Error finding container a39bf91a09bf8326ddb64e46e0cc3afabd1edd214d0a415b941bb924d0f96567: Status 404 returned error can't find the container with id a39bf91a09bf8326ddb64e46e0cc3afabd1edd214d0a415b941bb924d0f96567 Oct 31 00:33:12.186959 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:12.186931 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-1e5e75/distinct-w3-1e5e75-kserve-7b55d58bbb-hchmb"] Oct 31 00:33:12.368953 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:12.368915 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-1e5e75/distinct-w3-1e5e75-kserve-7b55d58bbb-hchmb" event={"ID":"79ffb21b-10c3-4174-89be-05ba9e852a98","Type":"ContainerStarted","Data":"f419e42661b29d96499018f9421f7cb63bc92d160a55ca9fd87635f7892d5ba8"} Oct 31 00:33:12.369127 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:12.368965 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-1e5e75/distinct-w3-1e5e75-kserve-7b55d58bbb-hchmb" event={"ID":"79ffb21b-10c3-4174-89be-05ba9e852a98","Type":"ContainerStarted","Data":"a39bf91a09bf8326ddb64e46e0cc3afabd1edd214d0a415b941bb924d0f96567"} Oct 31 00:33:12.498860 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:12.498759 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-1e5e75/distinct-w3-1e5e75-kserve-7b55d58bbb-hchmb" podStartSLOduration=1.498746692 podStartE2EDuration="1.498746692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:33:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:33:12.498059627 +0000 UTC m=+1084.969229376" watchObservedRunningTime="2026-10-31 00:33:12.498746692 +0000 UTC m=+1084.969916441" Oct 31 00:33:12.678792 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:12.678754 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-412743-openshift-default-7bddc8b5df-thfvg"] Oct 31 00:33:12.682734 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:12.682716 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-412743-openshift-default-7bddc8b5df-thfvg" Oct 31 00:33:12.736777 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:12.736741 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-412743-openshift-default-dockercfg-wqjw6\"" Oct 31 00:33:12.741067 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:12.741036 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-412743-openshift-default-7bddc8b5df-thfvg"] Oct 31 00:33:12.783467 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:12.783434 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-1e5e75/distinct-w3-1e5e75-kserve-7b55d58bbb-hchmb" Oct 31 00:33:12.825333 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:12.825300 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ede619b3-54d8-4ba1-9c2e-237e54461390-workload-socket\") pod \"e2e-worker-w5-412743-openshift-default-7bddc8b5df-thfvg\" (UID: \"ede619b3-54d8-4ba1-9c2e-237e54461390\") " pod="openshift-ingress/e2e-worker-w5-412743-openshift-default-7bddc8b5df-thfvg" Oct 31 00:33:12.825536 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:12.825350 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ede619b3-54d8-4ba1-9c2e-237e54461390-istio-token\") pod \"e2e-worker-w5-412743-openshift-default-7bddc8b5df-thfvg\" (UID: \"ede619b3-54d8-4ba1-9c2e-237e54461390\") " pod="openshift-ingress/e2e-worker-w5-412743-openshift-default-7bddc8b5df-thfvg" Oct 31 00:33:12.825536 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:12.825505 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ph4lj\" (UniqueName: \"kubernetes.io/projected/ede619b3-54d8-4ba1-9c2e-237e54461390-kube-api-access-ph4lj\") pod \"e2e-worker-w5-412743-openshift-default-7bddc8b5df-thfvg\" (UID: \"ede619b3-54d8-4ba1-9c2e-237e54461390\") " pod="openshift-ingress/e2e-worker-w5-412743-openshift-default-7bddc8b5df-thfvg" Oct 31 00:33:12.825618 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:12.825559 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ede619b3-54d8-4ba1-9c2e-237e54461390-credential-socket\") pod \"e2e-worker-w5-412743-openshift-default-7bddc8b5df-thfvg\" (UID: \"ede619b3-54d8-4ba1-9c2e-237e54461390\") " pod="openshift-ingress/e2e-worker-w5-412743-openshift-default-7bddc8b5df-thfvg" Oct 31 00:33:12.825655 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:12.825629 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ede619b3-54d8-4ba1-9c2e-237e54461390-workload-certs\") pod \"e2e-worker-w5-412743-openshift-default-7bddc8b5df-thfvg\" (UID: \"ede619b3-54d8-4ba1-9c2e-237e54461390\") " pod="openshift-ingress/e2e-worker-w5-412743-openshift-default-7bddc8b5df-thfvg" Oct 31 00:33:12.825689 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:12.825669 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ede619b3-54d8-4ba1-9c2e-237e54461390-istio-data\") pod \"e2e-worker-w5-412743-openshift-default-7bddc8b5df-thfvg\" (UID: \"ede619b3-54d8-4ba1-9c2e-237e54461390\") " pod="openshift-ingress/e2e-worker-w5-412743-openshift-default-7bddc8b5df-thfvg" Oct 31 00:33:12.825722 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:12.825696 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ede619b3-54d8-4ba1-9c2e-237e54461390-istio-envoy\") pod \"e2e-worker-w5-412743-openshift-default-7bddc8b5df-thfvg\" (UID: \"ede619b3-54d8-4ba1-9c2e-237e54461390\") " pod="openshift-ingress/e2e-worker-w5-412743-openshift-default-7bddc8b5df-thfvg" Oct 31 00:33:12.825722 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:12.825715 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ede619b3-54d8-4ba1-9c2e-237e54461390-istiod-ca-cert\") pod \"e2e-worker-w5-412743-openshift-default-7bddc8b5df-thfvg\" (UID: \"ede619b3-54d8-4ba1-9c2e-237e54461390\") " pod="openshift-ingress/e2e-worker-w5-412743-openshift-default-7bddc8b5df-thfvg" Oct 31 00:33:12.825816 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:12.825803 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ede619b3-54d8-4ba1-9c2e-237e54461390-istio-podinfo\") pod \"e2e-worker-w5-412743-openshift-default-7bddc8b5df-thfvg\" (UID: \"ede619b3-54d8-4ba1-9c2e-237e54461390\") " pod="openshift-ingress/e2e-worker-w5-412743-openshift-default-7bddc8b5df-thfvg" Oct 31 00:33:12.927163 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:12.927125 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ede619b3-54d8-4ba1-9c2e-237e54461390-workload-socket\") pod \"e2e-worker-w5-412743-openshift-default-7bddc8b5df-thfvg\" (UID: \"ede619b3-54d8-4ba1-9c2e-237e54461390\") " pod="openshift-ingress/e2e-worker-w5-412743-openshift-default-7bddc8b5df-thfvg" Oct 31 00:33:12.927391 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:12.927178 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ede619b3-54d8-4ba1-9c2e-237e54461390-istio-token\") pod \"e2e-worker-w5-412743-openshift-default-7bddc8b5df-thfvg\" (UID: \"ede619b3-54d8-4ba1-9c2e-237e54461390\") " pod="openshift-ingress/e2e-worker-w5-412743-openshift-default-7bddc8b5df-thfvg" Oct 31 00:33:12.927391 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:12.927223 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ph4lj\" (UniqueName: \"kubernetes.io/projected/ede619b3-54d8-4ba1-9c2e-237e54461390-kube-api-access-ph4lj\") pod \"e2e-worker-w5-412743-openshift-default-7bddc8b5df-thfvg\" (UID: \"ede619b3-54d8-4ba1-9c2e-237e54461390\") " pod="openshift-ingress/e2e-worker-w5-412743-openshift-default-7bddc8b5df-thfvg" Oct 31 00:33:12.927391 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:12.927256 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ede619b3-54d8-4ba1-9c2e-237e54461390-credential-socket\") pod \"e2e-worker-w5-412743-openshift-default-7bddc8b5df-thfvg\" (UID: \"ede619b3-54d8-4ba1-9c2e-237e54461390\") " pod="openshift-ingress/e2e-worker-w5-412743-openshift-default-7bddc8b5df-thfvg" Oct 31 00:33:12.927391 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:12.927274 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ede619b3-54d8-4ba1-9c2e-237e54461390-workload-certs\") pod \"e2e-worker-w5-412743-openshift-default-7bddc8b5df-thfvg\" (UID: \"ede619b3-54d8-4ba1-9c2e-237e54461390\") " pod="openshift-ingress/e2e-worker-w5-412743-openshift-default-7bddc8b5df-thfvg" Oct 31 00:33:12.927391 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:12.927304 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ede619b3-54d8-4ba1-9c2e-237e54461390-istio-data\") pod \"e2e-worker-w5-412743-openshift-default-7bddc8b5df-thfvg\" (UID: \"ede619b3-54d8-4ba1-9c2e-237e54461390\") " pod="openshift-ingress/e2e-worker-w5-412743-openshift-default-7bddc8b5df-thfvg" Oct 31 00:33:12.927391 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:12.927340 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ede619b3-54d8-4ba1-9c2e-237e54461390-istio-envoy\") pod \"e2e-worker-w5-412743-openshift-default-7bddc8b5df-thfvg\" (UID: \"ede619b3-54d8-4ba1-9c2e-237e54461390\") " pod="openshift-ingress/e2e-worker-w5-412743-openshift-default-7bddc8b5df-thfvg" Oct 31 00:33:12.927731 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:12.927402 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ede619b3-54d8-4ba1-9c2e-237e54461390-istiod-ca-cert\") pod \"e2e-worker-w5-412743-openshift-default-7bddc8b5df-thfvg\" (UID: \"ede619b3-54d8-4ba1-9c2e-237e54461390\") " pod="openshift-ingress/e2e-worker-w5-412743-openshift-default-7bddc8b5df-thfvg" Oct 31 00:33:12.927731 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:12.927448 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ede619b3-54d8-4ba1-9c2e-237e54461390-istio-podinfo\") pod \"e2e-worker-w5-412743-openshift-default-7bddc8b5df-thfvg\" (UID: \"ede619b3-54d8-4ba1-9c2e-237e54461390\") " pod="openshift-ingress/e2e-worker-w5-412743-openshift-default-7bddc8b5df-thfvg" Oct 31 00:33:12.927731 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:12.927677 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ede619b3-54d8-4ba1-9c2e-237e54461390-workload-socket\") pod \"e2e-worker-w5-412743-openshift-default-7bddc8b5df-thfvg\" (UID: \"ede619b3-54d8-4ba1-9c2e-237e54461390\") " pod="openshift-ingress/e2e-worker-w5-412743-openshift-default-7bddc8b5df-thfvg" Oct 31 00:33:12.927903 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:12.927722 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ede619b3-54d8-4ba1-9c2e-237e54461390-credential-socket\") pod \"e2e-worker-w5-412743-openshift-default-7bddc8b5df-thfvg\" (UID: \"ede619b3-54d8-4ba1-9c2e-237e54461390\") " pod="openshift-ingress/e2e-worker-w5-412743-openshift-default-7bddc8b5df-thfvg" Oct 31 00:33:12.927903 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:12.927758 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ede619b3-54d8-4ba1-9c2e-237e54461390-istio-data\") pod \"e2e-worker-w5-412743-openshift-default-7bddc8b5df-thfvg\" (UID: \"ede619b3-54d8-4ba1-9c2e-237e54461390\") " pod="openshift-ingress/e2e-worker-w5-412743-openshift-default-7bddc8b5df-thfvg" Oct 31 00:33:12.927903 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:12.927830 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ede619b3-54d8-4ba1-9c2e-237e54461390-workload-certs\") pod \"e2e-worker-w5-412743-openshift-default-7bddc8b5df-thfvg\" (UID: \"ede619b3-54d8-4ba1-9c2e-237e54461390\") " pod="openshift-ingress/e2e-worker-w5-412743-openshift-default-7bddc8b5df-thfvg" Oct 31 00:33:12.928115 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:12.928085 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ede619b3-54d8-4ba1-9c2e-237e54461390-istiod-ca-cert\") pod \"e2e-worker-w5-412743-openshift-default-7bddc8b5df-thfvg\" (UID: \"ede619b3-54d8-4ba1-9c2e-237e54461390\") " pod="openshift-ingress/e2e-worker-w5-412743-openshift-default-7bddc8b5df-thfvg" Oct 31 00:33:12.930016 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:12.929991 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ede619b3-54d8-4ba1-9c2e-237e54461390-istio-envoy\") pod \"e2e-worker-w5-412743-openshift-default-7bddc8b5df-thfvg\" (UID: \"ede619b3-54d8-4ba1-9c2e-237e54461390\") " pod="openshift-ingress/e2e-worker-w5-412743-openshift-default-7bddc8b5df-thfvg" Oct 31 00:33:12.930244 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:12.930226 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ede619b3-54d8-4ba1-9c2e-237e54461390-istio-podinfo\") pod \"e2e-worker-w5-412743-openshift-default-7bddc8b5df-thfvg\" (UID: \"ede619b3-54d8-4ba1-9c2e-237e54461390\") " pod="openshift-ingress/e2e-worker-w5-412743-openshift-default-7bddc8b5df-thfvg" Oct 31 00:33:12.990384 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:12.990329 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ede619b3-54d8-4ba1-9c2e-237e54461390-istio-token\") pod \"e2e-worker-w5-412743-openshift-default-7bddc8b5df-thfvg\" (UID: \"ede619b3-54d8-4ba1-9c2e-237e54461390\") " pod="openshift-ingress/e2e-worker-w5-412743-openshift-default-7bddc8b5df-thfvg" Oct 31 00:33:12.990560 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:12.990508 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ph4lj\" (UniqueName: \"kubernetes.io/projected/ede619b3-54d8-4ba1-9c2e-237e54461390-kube-api-access-ph4lj\") pod \"e2e-worker-w5-412743-openshift-default-7bddc8b5df-thfvg\" (UID: \"ede619b3-54d8-4ba1-9c2e-237e54461390\") " pod="openshift-ingress/e2e-worker-w5-412743-openshift-default-7bddc8b5df-thfvg" Oct 31 00:33:12.997294 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:12.997264 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-412743-openshift-default-7bddc8b5df-thfvg" Oct 31 00:33:13.431549 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:33:13.431516 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podede619b3_54d8_4ba1_9c2e_237e54461390.slice/crio-a2f1d7f37f593fffc225daf3fe7daacb82e32fb8ceaaecce0ab0c6d08aec4cea WatchSource:0}: Error finding container a2f1d7f37f593fffc225daf3fe7daacb82e32fb8ceaaecce0ab0c6d08aec4cea: Status 404 returned error can't find the container with id a2f1d7f37f593fffc225daf3fe7daacb82e32fb8ceaaecce0ab0c6d08aec4cea Oct 31 00:33:13.434208 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:13.434173 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:33:13.434323 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:13.434245 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:33:13.434323 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:13.434274 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:33:13.435363 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:13.435340 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-412743-openshift-default-7bddc8b5df-thfvg"] Oct 31 00:33:14.382825 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:14.382751 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-412743-openshift-default-7bddc8b5df-thfvg" event={"ID":"ede619b3-54d8-4ba1-9c2e-237e54461390","Type":"ContainerStarted","Data":"7e82c1d645c963553ad4634ab5649719ff344bbf6ef1096bf3960fea44b98eec"} Oct 31 00:33:14.382825 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:14.382805 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-412743-openshift-default-7bddc8b5df-thfvg" event={"ID":"ede619b3-54d8-4ba1-9c2e-237e54461390","Type":"ContainerStarted","Data":"a2f1d7f37f593fffc225daf3fe7daacb82e32fb8ceaaecce0ab0c6d08aec4cea"} Oct 31 00:33:14.567482 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:14.567428 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-412743-openshift-default-7bddc8b5df-thfvg" podStartSLOduration=2.567415023 podStartE2EDuration="2.567415023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:33:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:33:14.565156099 +0000 UTC m=+1087.036325849" watchObservedRunningTime="2026-10-31 00:33:14.567415023 +0000 UTC m=+1087.038584771" Oct 31 00:33:14.998357 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:14.998318 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-412743-openshift-default-7bddc8b5df-thfvg" Oct 31 00:33:14.999912 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:14.999887 5222 patch_prober.go:29] interesting pod/e2e-worker-w5-412743-openshift-default-7bddc8b5df-thfvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 31 00:33:15.000027 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:14.999938 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-412743-openshift-default-7bddc8b5df-thfvg" podUID="ede619b3-54d8-4ba1-9c2e-237e54461390" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 31 00:33:15.997743 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:15.997704 5222 patch_prober.go:29] interesting pod/e2e-worker-w5-412743-openshift-default-7bddc8b5df-thfvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 31 00:33:15.998240 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:15.997764 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-412743-openshift-default-7bddc8b5df-thfvg" podUID="ede619b3-54d8-4ba1-9c2e-237e54461390" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 31 00:33:16.998737 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:16.998695 5222 patch_prober.go:29] interesting pod/e2e-worker-w5-412743-openshift-default-7bddc8b5df-thfvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 31 00:33:16.999137 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:16.998767 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-412743-openshift-default-7bddc8b5df-thfvg" podUID="ede619b3-54d8-4ba1-9c2e-237e54461390" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 31 00:33:17.998609 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:17.998569 5222 patch_prober.go:29] interesting pod/e2e-worker-w5-412743-openshift-default-7bddc8b5df-thfvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 31 00:33:17.998797 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:17.998630 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-412743-openshift-default-7bddc8b5df-thfvg" podUID="ede619b3-54d8-4ba1-9c2e-237e54461390" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 31 00:33:18.998073 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:18.998028 5222 patch_prober.go:29] interesting pod/e2e-worker-w5-412743-openshift-default-7bddc8b5df-thfvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 31 00:33:18.998287 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:18.998111 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-412743-openshift-default-7bddc8b5df-thfvg" podUID="ede619b3-54d8-4ba1-9c2e-237e54461390" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 31 00:33:19.095140 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:19.095102 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-79tk8"] Oct 31 00:33:20.200228 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:20.200196 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-412743-openshift-default-7bddc8b5df-thfvg" Oct 31 00:33:20.200975 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:20.200945 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-412743-openshift-default-7bddc8b5df-thfvg" Oct 31 00:33:20.201815 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:20.201796 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-412743-openshift-default-7bddc8b5df-thfvg" Oct 31 00:33:21.782990 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:21.782948 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-1e5e75/distinct-w3-1e5e75-kserve-7b55d58bbb-hchmb" Oct 31 00:33:21.795823 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:21.795793 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-1e5e75/distinct-w3-1e5e75-kserve-7b55d58bbb-hchmb" Oct 31 00:33:22.446231 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:22.446196 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-1e5e75/distinct-w3-1e5e75-kserve-7b55d58bbb-hchmb" Oct 31 00:33:35.890307 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:33:35.890273 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 00:33:52.725674 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:52.725632 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-79tk8"] Oct 31 00:33:58.826329 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:58.826294 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-412743/distinct-w5-412743-kserve-64d65f7d8f-c4tfs"] Oct 31 00:33:58.832132 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:58.832109 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-412743/distinct-w5-412743-kserve-64d65f7d8f-c4tfs" Oct 31 00:33:58.879905 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:58.879871 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0e8d5c5-aa08-41f6-badf-792648a20a68-home\") pod \"distinct-w5-412743-kserve-64d65f7d8f-c4tfs\" (UID: \"d0e8d5c5-aa08-41f6-badf-792648a20a68\") " pod="e2e-models-e2e-worker-w5-412743/distinct-w5-412743-kserve-64d65f7d8f-c4tfs" Oct 31 00:33:58.880068 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:58.879922 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0e8d5c5-aa08-41f6-badf-792648a20a68-tmp-dir\") pod \"distinct-w5-412743-kserve-64d65f7d8f-c4tfs\" (UID: \"d0e8d5c5-aa08-41f6-badf-792648a20a68\") " pod="e2e-models-e2e-worker-w5-412743/distinct-w5-412743-kserve-64d65f7d8f-c4tfs" Oct 31 00:33:58.880114 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:58.880087 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0e8d5c5-aa08-41f6-badf-792648a20a68-dshm\") pod \"distinct-w5-412743-kserve-64d65f7d8f-c4tfs\" (UID: \"d0e8d5c5-aa08-41f6-badf-792648a20a68\") " pod="e2e-models-e2e-worker-w5-412743/distinct-w5-412743-kserve-64d65f7d8f-c4tfs" Oct 31 00:33:58.880155 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:58.880124 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0e8d5c5-aa08-41f6-badf-792648a20a68-tls-certs\") pod \"distinct-w5-412743-kserve-64d65f7d8f-c4tfs\" (UID: \"d0e8d5c5-aa08-41f6-badf-792648a20a68\") " pod="e2e-models-e2e-worker-w5-412743/distinct-w5-412743-kserve-64d65f7d8f-c4tfs" Oct 31 00:33:58.880193 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:58.880154 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0e8d5c5-aa08-41f6-badf-792648a20a68-model-cache\") pod \"distinct-w5-412743-kserve-64d65f7d8f-c4tfs\" (UID: \"d0e8d5c5-aa08-41f6-badf-792648a20a68\") " pod="e2e-models-e2e-worker-w5-412743/distinct-w5-412743-kserve-64d65f7d8f-c4tfs" Oct 31 00:33:58.880237 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:58.880216 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxzgk\" (UniqueName: \"kubernetes.io/projected/d0e8d5c5-aa08-41f6-badf-792648a20a68-kube-api-access-lxzgk\") pod \"distinct-w5-412743-kserve-64d65f7d8f-c4tfs\" (UID: \"d0e8d5c5-aa08-41f6-badf-792648a20a68\") " pod="e2e-models-e2e-worker-w5-412743/distinct-w5-412743-kserve-64d65f7d8f-c4tfs" Oct 31 00:33:58.887279 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:58.887249 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-412743/distinct-w5-412743-kserve-64d65f7d8f-c4tfs"] Oct 31 00:33:58.934655 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:58.934621 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-412743\"/\"distinct-w5-412743-kserve-self-signed-certs\"" Oct 31 00:33:58.988652 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:58.986489 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0e8d5c5-aa08-41f6-badf-792648a20a68-home\") pod \"distinct-w5-412743-kserve-64d65f7d8f-c4tfs\" (UID: \"d0e8d5c5-aa08-41f6-badf-792648a20a68\") " pod="e2e-models-e2e-worker-w5-412743/distinct-w5-412743-kserve-64d65f7d8f-c4tfs" Oct 31 00:33:58.988652 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:58.986566 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0e8d5c5-aa08-41f6-badf-792648a20a68-tmp-dir\") pod \"distinct-w5-412743-kserve-64d65f7d8f-c4tfs\" (UID: \"d0e8d5c5-aa08-41f6-badf-792648a20a68\") " pod="e2e-models-e2e-worker-w5-412743/distinct-w5-412743-kserve-64d65f7d8f-c4tfs" Oct 31 00:33:58.988652 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:58.987418 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0e8d5c5-aa08-41f6-badf-792648a20a68-dshm\") pod \"distinct-w5-412743-kserve-64d65f7d8f-c4tfs\" (UID: \"d0e8d5c5-aa08-41f6-badf-792648a20a68\") " pod="e2e-models-e2e-worker-w5-412743/distinct-w5-412743-kserve-64d65f7d8f-c4tfs" Oct 31 00:33:58.988652 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:58.987564 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0e8d5c5-aa08-41f6-badf-792648a20a68-tls-certs\") pod \"distinct-w5-412743-kserve-64d65f7d8f-c4tfs\" (UID: \"d0e8d5c5-aa08-41f6-badf-792648a20a68\") " pod="e2e-models-e2e-worker-w5-412743/distinct-w5-412743-kserve-64d65f7d8f-c4tfs" Oct 31 00:33:58.988652 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:58.987595 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0e8d5c5-aa08-41f6-badf-792648a20a68-model-cache\") pod \"distinct-w5-412743-kserve-64d65f7d8f-c4tfs\" (UID: \"d0e8d5c5-aa08-41f6-badf-792648a20a68\") " pod="e2e-models-e2e-worker-w5-412743/distinct-w5-412743-kserve-64d65f7d8f-c4tfs" Oct 31 00:33:58.988652 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:58.987724 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxzgk\" (UniqueName: \"kubernetes.io/projected/d0e8d5c5-aa08-41f6-badf-792648a20a68-kube-api-access-lxzgk\") pod \"distinct-w5-412743-kserve-64d65f7d8f-c4tfs\" (UID: \"d0e8d5c5-aa08-41f6-badf-792648a20a68\") " pod="e2e-models-e2e-worker-w5-412743/distinct-w5-412743-kserve-64d65f7d8f-c4tfs" Oct 31 00:33:58.988652 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:58.987781 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0e8d5c5-aa08-41f6-badf-792648a20a68-home\") pod \"distinct-w5-412743-kserve-64d65f7d8f-c4tfs\" (UID: \"d0e8d5c5-aa08-41f6-badf-792648a20a68\") " pod="e2e-models-e2e-worker-w5-412743/distinct-w5-412743-kserve-64d65f7d8f-c4tfs" Oct 31 00:33:58.988652 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:58.987057 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0e8d5c5-aa08-41f6-badf-792648a20a68-tmp-dir\") pod \"distinct-w5-412743-kserve-64d65f7d8f-c4tfs\" (UID: \"d0e8d5c5-aa08-41f6-badf-792648a20a68\") " pod="e2e-models-e2e-worker-w5-412743/distinct-w5-412743-kserve-64d65f7d8f-c4tfs" Oct 31 00:33:58.994701 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:58.992914 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0e8d5c5-aa08-41f6-badf-792648a20a68-model-cache\") pod \"distinct-w5-412743-kserve-64d65f7d8f-c4tfs\" (UID: \"d0e8d5c5-aa08-41f6-badf-792648a20a68\") " pod="e2e-models-e2e-worker-w5-412743/distinct-w5-412743-kserve-64d65f7d8f-c4tfs" Oct 31 00:33:58.994701 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:58.994417 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0e8d5c5-aa08-41f6-badf-792648a20a68-dshm\") pod \"distinct-w5-412743-kserve-64d65f7d8f-c4tfs\" (UID: \"d0e8d5c5-aa08-41f6-badf-792648a20a68\") " pod="e2e-models-e2e-worker-w5-412743/distinct-w5-412743-kserve-64d65f7d8f-c4tfs" Oct 31 00:33:58.994701 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:58.994528 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0e8d5c5-aa08-41f6-badf-792648a20a68-tls-certs\") pod \"distinct-w5-412743-kserve-64d65f7d8f-c4tfs\" (UID: \"d0e8d5c5-aa08-41f6-badf-792648a20a68\") " pod="e2e-models-e2e-worker-w5-412743/distinct-w5-412743-kserve-64d65f7d8f-c4tfs" Oct 31 00:33:59.055728 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:59.055691 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxzgk\" (UniqueName: \"kubernetes.io/projected/d0e8d5c5-aa08-41f6-badf-792648a20a68-kube-api-access-lxzgk\") pod \"distinct-w5-412743-kserve-64d65f7d8f-c4tfs\" (UID: \"d0e8d5c5-aa08-41f6-badf-792648a20a68\") " pod="e2e-models-e2e-worker-w5-412743/distinct-w5-412743-kserve-64d65f7d8f-c4tfs" Oct 31 00:33:59.145657 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:59.145555 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-412743/distinct-w5-412743-kserve-64d65f7d8f-c4tfs" Oct 31 00:33:59.550013 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:33:59.549970 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0e8d5c5_aa08_41f6_badf_792648a20a68.slice/crio-d7f5e2a1b1245344f532b700b30fb938a3bbddce02e33e32bea18c49f5e18b1f WatchSource:0}: Error finding container d7f5e2a1b1245344f532b700b30fb938a3bbddce02e33e32bea18c49f5e18b1f: Status 404 returned error can't find the container with id d7f5e2a1b1245344f532b700b30fb938a3bbddce02e33e32bea18c49f5e18b1f Oct 31 00:33:59.553324 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:59.553304 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-412743/distinct-w5-412743-kserve-64d65f7d8f-c4tfs"] Oct 31 00:33:59.672011 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:59.671976 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-412743/distinct-w5-412743-kserve-64d65f7d8f-c4tfs" event={"ID":"d0e8d5c5-aa08-41f6-badf-792648a20a68","Type":"ContainerStarted","Data":"af8a8bb0d4914d263c0a1b9186a10009c551064f85aca4355365e584cba73091"} Oct 31 00:33:59.672222 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:59.672022 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-412743/distinct-w5-412743-kserve-64d65f7d8f-c4tfs" event={"ID":"d0e8d5c5-aa08-41f6-badf-792648a20a68","Type":"ContainerStarted","Data":"d7f5e2a1b1245344f532b700b30fb938a3bbddce02e33e32bea18c49f5e18b1f"} Oct 31 00:33:59.794297 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:33:59.794244 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-412743/distinct-w5-412743-kserve-64d65f7d8f-c4tfs" podStartSLOduration=1.7942294749999999 podStartE2EDuration="1.794229475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:33:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:33:59.793190975 +0000 UTC m=+1132.264360725" watchObservedRunningTime="2026-10-31 00:33:59.794229475 +0000 UTC m=+1132.265399224" Oct 31 00:34:00.146482 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:00.146446 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-412743/distinct-w5-412743-kserve-64d65f7d8f-c4tfs" Oct 31 00:34:02.341931 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:02.341897 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-982ae3/distinct-w1-982ae3-kserve-84b6f97b84-4hhxk"] Oct 31 00:34:02.346180 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:02.346154 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-982ae3/distinct-w1-982ae3-kserve-84b6f97b84-4hhxk" Oct 31 00:34:02.400506 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:02.400474 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-982ae3/distinct-w1-982ae3-kserve-84b6f97b84-4hhxk"] Oct 31 00:34:02.441691 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:02.441648 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4ffad70-a133-4d21-aa05-c90cf8d5b6e9-tls-certs\") pod \"distinct-w1-982ae3-kserve-84b6f97b84-4hhxk\" (UID: \"c4ffad70-a133-4d21-aa05-c90cf8d5b6e9\") " pod="e2e-models-e2e-worker-w1-982ae3/distinct-w1-982ae3-kserve-84b6f97b84-4hhxk" Oct 31 00:34:02.441865 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:02.441700 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4ffad70-a133-4d21-aa05-c90cf8d5b6e9-home\") pod \"distinct-w1-982ae3-kserve-84b6f97b84-4hhxk\" (UID: \"c4ffad70-a133-4d21-aa05-c90cf8d5b6e9\") " pod="e2e-models-e2e-worker-w1-982ae3/distinct-w1-982ae3-kserve-84b6f97b84-4hhxk" Oct 31 00:34:02.441865 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:02.441838 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4ffad70-a133-4d21-aa05-c90cf8d5b6e9-tmp-dir\") pod \"distinct-w1-982ae3-kserve-84b6f97b84-4hhxk\" (UID: \"c4ffad70-a133-4d21-aa05-c90cf8d5b6e9\") " pod="e2e-models-e2e-worker-w1-982ae3/distinct-w1-982ae3-kserve-84b6f97b84-4hhxk" Oct 31 00:34:02.441954 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:02.441899 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcgqj\" (UniqueName: \"kubernetes.io/projected/c4ffad70-a133-4d21-aa05-c90cf8d5b6e9-kube-api-access-mcgqj\") pod \"distinct-w1-982ae3-kserve-84b6f97b84-4hhxk\" (UID: \"c4ffad70-a133-4d21-aa05-c90cf8d5b6e9\") " pod="e2e-models-e2e-worker-w1-982ae3/distinct-w1-982ae3-kserve-84b6f97b84-4hhxk" Oct 31 00:34:02.442001 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:02.441990 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4ffad70-a133-4d21-aa05-c90cf8d5b6e9-dshm\") pod \"distinct-w1-982ae3-kserve-84b6f97b84-4hhxk\" (UID: \"c4ffad70-a133-4d21-aa05-c90cf8d5b6e9\") " pod="e2e-models-e2e-worker-w1-982ae3/distinct-w1-982ae3-kserve-84b6f97b84-4hhxk" Oct 31 00:34:02.442115 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:02.442099 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4ffad70-a133-4d21-aa05-c90cf8d5b6e9-model-cache\") pod \"distinct-w1-982ae3-kserve-84b6f97b84-4hhxk\" (UID: \"c4ffad70-a133-4d21-aa05-c90cf8d5b6e9\") " pod="e2e-models-e2e-worker-w1-982ae3/distinct-w1-982ae3-kserve-84b6f97b84-4hhxk" Oct 31 00:34:02.447703 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:02.447677 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-982ae3\"/\"distinct-w1-982ae3-kserve-self-signed-certs\"" Oct 31 00:34:02.543620 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:02.543584 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4ffad70-a133-4d21-aa05-c90cf8d5b6e9-tls-certs\") pod \"distinct-w1-982ae3-kserve-84b6f97b84-4hhxk\" (UID: \"c4ffad70-a133-4d21-aa05-c90cf8d5b6e9\") " pod="e2e-models-e2e-worker-w1-982ae3/distinct-w1-982ae3-kserve-84b6f97b84-4hhxk" Oct 31 00:34:02.543620 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:02.543624 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4ffad70-a133-4d21-aa05-c90cf8d5b6e9-home\") pod \"distinct-w1-982ae3-kserve-84b6f97b84-4hhxk\" (UID: \"c4ffad70-a133-4d21-aa05-c90cf8d5b6e9\") " pod="e2e-models-e2e-worker-w1-982ae3/distinct-w1-982ae3-kserve-84b6f97b84-4hhxk" Oct 31 00:34:02.543855 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:02.543666 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4ffad70-a133-4d21-aa05-c90cf8d5b6e9-tmp-dir\") pod \"distinct-w1-982ae3-kserve-84b6f97b84-4hhxk\" (UID: \"c4ffad70-a133-4d21-aa05-c90cf8d5b6e9\") " pod="e2e-models-e2e-worker-w1-982ae3/distinct-w1-982ae3-kserve-84b6f97b84-4hhxk" Oct 31 00:34:02.543855 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:02.543693 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcgqj\" (UniqueName: \"kubernetes.io/projected/c4ffad70-a133-4d21-aa05-c90cf8d5b6e9-kube-api-access-mcgqj\") pod \"distinct-w1-982ae3-kserve-84b6f97b84-4hhxk\" (UID: \"c4ffad70-a133-4d21-aa05-c90cf8d5b6e9\") " pod="e2e-models-e2e-worker-w1-982ae3/distinct-w1-982ae3-kserve-84b6f97b84-4hhxk" Oct 31 00:34:02.543855 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:02.543733 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4ffad70-a133-4d21-aa05-c90cf8d5b6e9-dshm\") pod \"distinct-w1-982ae3-kserve-84b6f97b84-4hhxk\" (UID: \"c4ffad70-a133-4d21-aa05-c90cf8d5b6e9\") " pod="e2e-models-e2e-worker-w1-982ae3/distinct-w1-982ae3-kserve-84b6f97b84-4hhxk" Oct 31 00:34:02.543855 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:02.543833 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4ffad70-a133-4d21-aa05-c90cf8d5b6e9-model-cache\") pod \"distinct-w1-982ae3-kserve-84b6f97b84-4hhxk\" (UID: \"c4ffad70-a133-4d21-aa05-c90cf8d5b6e9\") " pod="e2e-models-e2e-worker-w1-982ae3/distinct-w1-982ae3-kserve-84b6f97b84-4hhxk" Oct 31 00:34:02.544090 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:02.544068 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4ffad70-a133-4d21-aa05-c90cf8d5b6e9-home\") pod \"distinct-w1-982ae3-kserve-84b6f97b84-4hhxk\" (UID: \"c4ffad70-a133-4d21-aa05-c90cf8d5b6e9\") " pod="e2e-models-e2e-worker-w1-982ae3/distinct-w1-982ae3-kserve-84b6f97b84-4hhxk" Oct 31 00:34:02.544131 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:02.544097 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4ffad70-a133-4d21-aa05-c90cf8d5b6e9-tmp-dir\") pod \"distinct-w1-982ae3-kserve-84b6f97b84-4hhxk\" (UID: \"c4ffad70-a133-4d21-aa05-c90cf8d5b6e9\") " pod="e2e-models-e2e-worker-w1-982ae3/distinct-w1-982ae3-kserve-84b6f97b84-4hhxk" Oct 31 00:34:02.544361 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:02.544341 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4ffad70-a133-4d21-aa05-c90cf8d5b6e9-model-cache\") pod \"distinct-w1-982ae3-kserve-84b6f97b84-4hhxk\" (UID: \"c4ffad70-a133-4d21-aa05-c90cf8d5b6e9\") " pod="e2e-models-e2e-worker-w1-982ae3/distinct-w1-982ae3-kserve-84b6f97b84-4hhxk" Oct 31 00:34:02.546257 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:02.546234 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4ffad70-a133-4d21-aa05-c90cf8d5b6e9-dshm\") pod \"distinct-w1-982ae3-kserve-84b6f97b84-4hhxk\" (UID: \"c4ffad70-a133-4d21-aa05-c90cf8d5b6e9\") " pod="e2e-models-e2e-worker-w1-982ae3/distinct-w1-982ae3-kserve-84b6f97b84-4hhxk" Oct 31 00:34:02.546348 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:02.546290 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4ffad70-a133-4d21-aa05-c90cf8d5b6e9-tls-certs\") pod \"distinct-w1-982ae3-kserve-84b6f97b84-4hhxk\" (UID: \"c4ffad70-a133-4d21-aa05-c90cf8d5b6e9\") " pod="e2e-models-e2e-worker-w1-982ae3/distinct-w1-982ae3-kserve-84b6f97b84-4hhxk" Oct 31 00:34:02.607043 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:02.606967 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcgqj\" (UniqueName: \"kubernetes.io/projected/c4ffad70-a133-4d21-aa05-c90cf8d5b6e9-kube-api-access-mcgqj\") pod \"distinct-w1-982ae3-kserve-84b6f97b84-4hhxk\" (UID: \"c4ffad70-a133-4d21-aa05-c90cf8d5b6e9\") " pod="e2e-models-e2e-worker-w1-982ae3/distinct-w1-982ae3-kserve-84b6f97b84-4hhxk" Oct 31 00:34:02.659736 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:02.659686 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-982ae3/distinct-w1-982ae3-kserve-84b6f97b84-4hhxk" Oct 31 00:34:03.078410 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:34:03.078359 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc4ffad70_a133_4d21_aa05_c90cf8d5b6e9.slice/crio-6201dbb72c8650fdbe32974a5986f22d741104431323a0bf389c3e580c44111a WatchSource:0}: Error finding container 6201dbb72c8650fdbe32974a5986f22d741104431323a0bf389c3e580c44111a: Status 404 returned error can't find the container with id 6201dbb72c8650fdbe32974a5986f22d741104431323a0bf389c3e580c44111a Oct 31 00:34:03.081438 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:03.081418 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-982ae3/distinct-w1-982ae3-kserve-84b6f97b84-4hhxk"] Oct 31 00:34:03.701527 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:03.701486 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-982ae3/distinct-w1-982ae3-kserve-84b6f97b84-4hhxk" event={"ID":"c4ffad70-a133-4d21-aa05-c90cf8d5b6e9","Type":"ContainerStarted","Data":"7c42913245dd294852346c67d91e6f36c6149d2540daa7580575bb8a00481aa7"} Oct 31 00:34:03.702034 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:03.701541 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-982ae3/distinct-w1-982ae3-kserve-84b6f97b84-4hhxk" event={"ID":"c4ffad70-a133-4d21-aa05-c90cf8d5b6e9","Type":"ContainerStarted","Data":"6201dbb72c8650fdbe32974a5986f22d741104431323a0bf389c3e580c44111a"} Oct 31 00:34:03.823612 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:03.823553 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-982ae3/distinct-w1-982ae3-kserve-84b6f97b84-4hhxk" podStartSLOduration=1.823537273 podStartE2EDuration="1.823537273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:34:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:34:03.822097967 +0000 UTC m=+1136.293267718" watchObservedRunningTime="2026-10-31 00:34:03.823537273 +0000 UTC m=+1136.294707022" Oct 31 00:34:04.660050 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:04.660005 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-982ae3/distinct-w1-982ae3-kserve-84b6f97b84-4hhxk" Oct 31 00:34:05.891014 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:34:05.890976 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 00:34:09.145949 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:09.145913 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-412743/distinct-w5-412743-kserve-64d65f7d8f-c4tfs" Oct 31 00:34:09.158832 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:09.158792 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-412743/distinct-w5-412743-kserve-64d65f7d8f-c4tfs" Oct 31 00:34:09.752360 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:09.752324 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-412743/distinct-w5-412743-kserve-64d65f7d8f-c4tfs" Oct 31 00:34:12.659936 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:12.659900 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-982ae3/distinct-w1-982ae3-kserve-84b6f97b84-4hhxk" Oct 31 00:34:12.672685 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:12.672656 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-982ae3/distinct-w1-982ae3-kserve-84b6f97b84-4hhxk" Oct 31 00:34:12.772759 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:12.772732 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-982ae3/distinct-w1-982ae3-kserve-84b6f97b84-4hhxk" Oct 31 00:34:17.512679 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:17.512640 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-79tk8"] Oct 31 00:34:27.618192 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:27.618159 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-101b9e/distinct-2-w0-101b9e-kserve-74548f6d6-v5phc"] Oct 31 00:34:27.622459 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:27.622425 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-101b9e/distinct-2-w0-101b9e-kserve-74548f6d6-v5phc" Oct 31 00:34:27.727837 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:27.727807 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-101b9e\"/\"distinct-2-w0-101b9e-kserve-self-signed-certs\"" Oct 31 00:34:27.731716 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:27.731691 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-101b9e/distinct-2-w0-101b9e-kserve-74548f6d6-v5phc"] Oct 31 00:34:27.762925 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:27.762886 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8049531-bcf6-4f19-84d2-b974b9236223-tls-certs\") pod \"distinct-2-w0-101b9e-kserve-74548f6d6-v5phc\" (UID: \"b8049531-bcf6-4f19-84d2-b974b9236223\") " pod="e2e-models-e2e-worker-w0-101b9e/distinct-2-w0-101b9e-kserve-74548f6d6-v5phc" Oct 31 00:34:27.765750 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:27.765713 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8049531-bcf6-4f19-84d2-b974b9236223-home\") pod \"distinct-2-w0-101b9e-kserve-74548f6d6-v5phc\" (UID: \"b8049531-bcf6-4f19-84d2-b974b9236223\") " pod="e2e-models-e2e-worker-w0-101b9e/distinct-2-w0-101b9e-kserve-74548f6d6-v5phc" Oct 31 00:34:27.765936 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:27.765828 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8049531-bcf6-4f19-84d2-b974b9236223-dshm\") pod \"distinct-2-w0-101b9e-kserve-74548f6d6-v5phc\" (UID: \"b8049531-bcf6-4f19-84d2-b974b9236223\") " pod="e2e-models-e2e-worker-w0-101b9e/distinct-2-w0-101b9e-kserve-74548f6d6-v5phc" Oct 31 00:34:27.765936 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:27.765858 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqdb8\" (UniqueName: \"kubernetes.io/projected/b8049531-bcf6-4f19-84d2-b974b9236223-kube-api-access-kqdb8\") pod \"distinct-2-w0-101b9e-kserve-74548f6d6-v5phc\" (UID: \"b8049531-bcf6-4f19-84d2-b974b9236223\") " pod="e2e-models-e2e-worker-w0-101b9e/distinct-2-w0-101b9e-kserve-74548f6d6-v5phc" Oct 31 00:34:27.765936 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:27.765891 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8049531-bcf6-4f19-84d2-b974b9236223-tmp-dir\") pod \"distinct-2-w0-101b9e-kserve-74548f6d6-v5phc\" (UID: \"b8049531-bcf6-4f19-84d2-b974b9236223\") " pod="e2e-models-e2e-worker-w0-101b9e/distinct-2-w0-101b9e-kserve-74548f6d6-v5phc" Oct 31 00:34:27.766130 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:27.765944 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8049531-bcf6-4f19-84d2-b974b9236223-model-cache\") pod \"distinct-2-w0-101b9e-kserve-74548f6d6-v5phc\" (UID: \"b8049531-bcf6-4f19-84d2-b974b9236223\") " pod="e2e-models-e2e-worker-w0-101b9e/distinct-2-w0-101b9e-kserve-74548f6d6-v5phc" Oct 31 00:34:27.867756 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:27.867717 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8049531-bcf6-4f19-84d2-b974b9236223-tls-certs\") pod \"distinct-2-w0-101b9e-kserve-74548f6d6-v5phc\" (UID: \"b8049531-bcf6-4f19-84d2-b974b9236223\") " pod="e2e-models-e2e-worker-w0-101b9e/distinct-2-w0-101b9e-kserve-74548f6d6-v5phc" Oct 31 00:34:27.867972 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:27.867846 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8049531-bcf6-4f19-84d2-b974b9236223-home\") pod \"distinct-2-w0-101b9e-kserve-74548f6d6-v5phc\" (UID: \"b8049531-bcf6-4f19-84d2-b974b9236223\") " pod="e2e-models-e2e-worker-w0-101b9e/distinct-2-w0-101b9e-kserve-74548f6d6-v5phc" Oct 31 00:34:27.867972 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:27.867930 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8049531-bcf6-4f19-84d2-b974b9236223-dshm\") pod \"distinct-2-w0-101b9e-kserve-74548f6d6-v5phc\" (UID: \"b8049531-bcf6-4f19-84d2-b974b9236223\") " pod="e2e-models-e2e-worker-w0-101b9e/distinct-2-w0-101b9e-kserve-74548f6d6-v5phc" Oct 31 00:34:27.867972 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:27.867957 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqdb8\" (UniqueName: \"kubernetes.io/projected/b8049531-bcf6-4f19-84d2-b974b9236223-kube-api-access-kqdb8\") pod \"distinct-2-w0-101b9e-kserve-74548f6d6-v5phc\" (UID: \"b8049531-bcf6-4f19-84d2-b974b9236223\") " pod="e2e-models-e2e-worker-w0-101b9e/distinct-2-w0-101b9e-kserve-74548f6d6-v5phc" Oct 31 00:34:27.868644 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:27.867987 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8049531-bcf6-4f19-84d2-b974b9236223-tmp-dir\") pod \"distinct-2-w0-101b9e-kserve-74548f6d6-v5phc\" (UID: \"b8049531-bcf6-4f19-84d2-b974b9236223\") " pod="e2e-models-e2e-worker-w0-101b9e/distinct-2-w0-101b9e-kserve-74548f6d6-v5phc" Oct 31 00:34:27.868644 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:27.868033 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8049531-bcf6-4f19-84d2-b974b9236223-model-cache\") pod \"distinct-2-w0-101b9e-kserve-74548f6d6-v5phc\" (UID: \"b8049531-bcf6-4f19-84d2-b974b9236223\") " pod="e2e-models-e2e-worker-w0-101b9e/distinct-2-w0-101b9e-kserve-74548f6d6-v5phc" Oct 31 00:34:27.868644 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:27.868357 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8049531-bcf6-4f19-84d2-b974b9236223-home\") pod \"distinct-2-w0-101b9e-kserve-74548f6d6-v5phc\" (UID: \"b8049531-bcf6-4f19-84d2-b974b9236223\") " pod="e2e-models-e2e-worker-w0-101b9e/distinct-2-w0-101b9e-kserve-74548f6d6-v5phc" Oct 31 00:34:27.868644 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:27.868468 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8049531-bcf6-4f19-84d2-b974b9236223-tmp-dir\") pod \"distinct-2-w0-101b9e-kserve-74548f6d6-v5phc\" (UID: \"b8049531-bcf6-4f19-84d2-b974b9236223\") " pod="e2e-models-e2e-worker-w0-101b9e/distinct-2-w0-101b9e-kserve-74548f6d6-v5phc" Oct 31 00:34:27.868644 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:27.868515 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8049531-bcf6-4f19-84d2-b974b9236223-model-cache\") pod \"distinct-2-w0-101b9e-kserve-74548f6d6-v5phc\" (UID: \"b8049531-bcf6-4f19-84d2-b974b9236223\") " pod="e2e-models-e2e-worker-w0-101b9e/distinct-2-w0-101b9e-kserve-74548f6d6-v5phc" Oct 31 00:34:27.870357 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:27.870335 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8049531-bcf6-4f19-84d2-b974b9236223-dshm\") pod \"distinct-2-w0-101b9e-kserve-74548f6d6-v5phc\" (UID: \"b8049531-bcf6-4f19-84d2-b974b9236223\") " pod="e2e-models-e2e-worker-w0-101b9e/distinct-2-w0-101b9e-kserve-74548f6d6-v5phc" Oct 31 00:34:27.870702 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:27.870681 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8049531-bcf6-4f19-84d2-b974b9236223-tls-certs\") pod \"distinct-2-w0-101b9e-kserve-74548f6d6-v5phc\" (UID: \"b8049531-bcf6-4f19-84d2-b974b9236223\") " pod="e2e-models-e2e-worker-w0-101b9e/distinct-2-w0-101b9e-kserve-74548f6d6-v5phc" Oct 31 00:34:27.931319 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:27.931284 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqdb8\" (UniqueName: \"kubernetes.io/projected/b8049531-bcf6-4f19-84d2-b974b9236223-kube-api-access-kqdb8\") pod \"distinct-2-w0-101b9e-kserve-74548f6d6-v5phc\" (UID: \"b8049531-bcf6-4f19-84d2-b974b9236223\") " pod="e2e-models-e2e-worker-w0-101b9e/distinct-2-w0-101b9e-kserve-74548f6d6-v5phc" Oct 31 00:34:27.938163 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:27.938133 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-101b9e/distinct-2-w0-101b9e-kserve-74548f6d6-v5phc" Oct 31 00:34:28.353609 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:34:28.353578 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb8049531_bcf6_4f19_84d2_b974b9236223.slice/crio-2db2c97e6643f3877978cccc02df5d74e80f8398c7d9940245d2e7abbc55298d WatchSource:0}: Error finding container 2db2c97e6643f3877978cccc02df5d74e80f8398c7d9940245d2e7abbc55298d: Status 404 returned error can't find the container with id 2db2c97e6643f3877978cccc02df5d74e80f8398c7d9940245d2e7abbc55298d Oct 31 00:34:28.356899 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:28.356874 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-101b9e/distinct-2-w0-101b9e-kserve-74548f6d6-v5phc"] Oct 31 00:34:28.894521 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:28.894482 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-101b9e/distinct-2-w0-101b9e-kserve-74548f6d6-v5phc" event={"ID":"b8049531-bcf6-4f19-84d2-b974b9236223","Type":"ContainerStarted","Data":"3de9a26b71ba3e866fdf0760764d8ec12776a0d1d10a27cd71ede4821cad7543"} Oct 31 00:34:28.894916 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:28.894530 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-101b9e/distinct-2-w0-101b9e-kserve-74548f6d6-v5phc" event={"ID":"b8049531-bcf6-4f19-84d2-b974b9236223","Type":"ContainerStarted","Data":"2db2c97e6643f3877978cccc02df5d74e80f8398c7d9940245d2e7abbc55298d"} Oct 31 00:34:28.938542 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:28.938509 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-101b9e/distinct-2-w0-101b9e-kserve-74548f6d6-v5phc" Oct 31 00:34:29.014967 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:29.014910 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-101b9e/distinct-2-w0-101b9e-kserve-74548f6d6-v5phc" podStartSLOduration=2.014895815 podStartE2EDuration="2.014895815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:34:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:34:29.013763733 +0000 UTC m=+1161.484933483" watchObservedRunningTime="2026-10-31 00:34:29.014895815 +0000 UTC m=+1161.486065564" Oct 31 00:34:30.198405 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:30.198357 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e72197c4-vmwck"] Oct 31 00:34:30.202486 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:30.202463 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e72197c4-vmwck" Oct 31 00:34:30.259142 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:30.259107 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e72197c4-vmwck"] Oct 31 00:34:30.293278 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:30.293245 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/731b409e-e01b-42e0-b20b-ce7cfc1edd67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e72197c4-vmwck\" (UID: \"731b409e-e01b-42e0-b20b-ce7cfc1edd67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e72197c4-vmwck" Oct 31 00:34:30.293455 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:30.293359 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/731b409e-e01b-42e0-b20b-ce7cfc1edd67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e72197c4-vmwck\" (UID: \"731b409e-e01b-42e0-b20b-ce7cfc1edd67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e72197c4-vmwck" Oct 31 00:34:30.293455 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:30.293434 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffr6f\" (UniqueName: \"kubernetes.io/projected/731b409e-e01b-42e0-b20b-ce7cfc1edd67-kube-api-access-ffr6f\") pod \"maas-api-revoke-sub-e72197c4-vmwck\" (UID: \"731b409e-e01b-42e0-b20b-ce7cfc1edd67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e72197c4-vmwck" Oct 31 00:34:30.306862 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:30.306825 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:34:30.307552 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:30.307531 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:34:30.307552 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:30.307532 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfz9z\"" Oct 31 00:34:30.307716 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:30.307590 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:34:30.394753 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:30.394713 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/731b409e-e01b-42e0-b20b-ce7cfc1edd67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e72197c4-vmwck\" (UID: \"731b409e-e01b-42e0-b20b-ce7cfc1edd67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e72197c4-vmwck" Oct 31 00:34:30.394962 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:30.394778 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffr6f\" (UniqueName: \"kubernetes.io/projected/731b409e-e01b-42e0-b20b-ce7cfc1edd67-kube-api-access-ffr6f\") pod \"maas-api-revoke-sub-e72197c4-vmwck\" (UID: \"731b409e-e01b-42e0-b20b-ce7cfc1edd67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e72197c4-vmwck" Oct 31 00:34:30.394962 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:30.394852 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/731b409e-e01b-42e0-b20b-ce7cfc1edd67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e72197c4-vmwck\" (UID: \"731b409e-e01b-42e0-b20b-ce7cfc1edd67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e72197c4-vmwck" Oct 31 00:34:30.395518 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:30.395484 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/731b409e-e01b-42e0-b20b-ce7cfc1edd67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e72197c4-vmwck\" (UID: \"731b409e-e01b-42e0-b20b-ce7cfc1edd67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e72197c4-vmwck" Oct 31 00:34:30.397411 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:30.397388 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/731b409e-e01b-42e0-b20b-ce7cfc1edd67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e72197c4-vmwck\" (UID: \"731b409e-e01b-42e0-b20b-ce7cfc1edd67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e72197c4-vmwck" Oct 31 00:34:30.458000 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:30.457924 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffr6f\" (UniqueName: \"kubernetes.io/projected/731b409e-e01b-42e0-b20b-ce7cfc1edd67-kube-api-access-ffr6f\") pod \"maas-api-revoke-sub-e72197c4-vmwck\" (UID: \"731b409e-e01b-42e0-b20b-ce7cfc1edd67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e72197c4-vmwck" Oct 31 00:34:30.517057 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:30.517018 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e72197c4-vmwck" Oct 31 00:34:30.891663 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:34:30.891629 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod731b409e_e01b_42e0_b20b_ce7cfc1edd67.slice/crio-a89a29ab03a49fee73184302121fc13167431c4facb417cdb61f6304914ece05 WatchSource:0}: Error finding container a89a29ab03a49fee73184302121fc13167431c4facb417cdb61f6304914ece05: Status 404 returned error can't find the container with id a89a29ab03a49fee73184302121fc13167431c4facb417cdb61f6304914ece05 Oct 31 00:34:30.895633 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:30.895606 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e72197c4-vmwck"] Oct 31 00:34:30.909939 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:30.909913 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e72197c4-vmwck" event={"ID":"731b409e-e01b-42e0-b20b-ce7cfc1edd67","Type":"ContainerStarted","Data":"a89a29ab03a49fee73184302121fc13167431c4facb417cdb61f6304914ece05"} Oct 31 00:34:31.918510 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:31.918474 5222 generic.go:356] "Generic (PLEG): container finished" podID="731b409e-e01b-42e0-b20b-ce7cfc1edd67" containerID="1b39813a2154ebf791ea20527256467f4c439e06f3cf38a44e2696c9238c2bc2" exitCode=0 Oct 31 00:34:31.918922 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:31.918554 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e72197c4-vmwck" event={"ID":"731b409e-e01b-42e0-b20b-ce7cfc1edd67","Type":"ContainerDied","Data":"1b39813a2154ebf791ea20527256467f4c439e06f3cf38a44e2696c9238c2bc2"} Oct 31 00:34:33.078170 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:33.078140 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e72197c4-vmwck" Oct 31 00:34:33.123671 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:33.123638 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/731b409e-e01b-42e0-b20b-ce7cfc1edd67-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/731b409e-e01b-42e0-b20b-ce7cfc1edd67-maas-api-openshift-ca\") pod \"731b409e-e01b-42e0-b20b-ce7cfc1edd67\" (UID: \"731b409e-e01b-42e0-b20b-ce7cfc1edd67\") " Oct 31 00:34:33.123838 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:33.123708 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/731b409e-e01b-42e0-b20b-ce7cfc1edd67-kube-api-access-ffr6f\" (UniqueName: \"kubernetes.io/projected/731b409e-e01b-42e0-b20b-ce7cfc1edd67-kube-api-access-ffr6f\") pod \"731b409e-e01b-42e0-b20b-ce7cfc1edd67\" (UID: \"731b409e-e01b-42e0-b20b-ce7cfc1edd67\") " Oct 31 00:34:33.123838 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:33.123757 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/731b409e-e01b-42e0-b20b-ce7cfc1edd67-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/731b409e-e01b-42e0-b20b-ce7cfc1edd67-maas-api-xks-ca\") pod \"731b409e-e01b-42e0-b20b-ce7cfc1edd67\" (UID: \"731b409e-e01b-42e0-b20b-ce7cfc1edd67\") " Oct 31 00:34:33.124075 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:33.124051 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/731b409e-e01b-42e0-b20b-ce7cfc1edd67-maas-api-openshift-ca" pod "731b409e-e01b-42e0-b20b-ce7cfc1edd67" (UID: "731b409e-e01b-42e0-b20b-ce7cfc1edd67"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:34:33.124302 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:33.124286 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/731b409e-e01b-42e0-b20b-ce7cfc1edd67-maas-api-openshift-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:34:33.125964 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:33.125940 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/731b409e-e01b-42e0-b20b-ce7cfc1edd67-maas-api-xks-ca" pod "731b409e-e01b-42e0-b20b-ce7cfc1edd67" (UID: "731b409e-e01b-42e0-b20b-ce7cfc1edd67"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:34:33.126051 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:33.126031 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/731b409e-e01b-42e0-b20b-ce7cfc1edd67-kube-api-access-ffr6f" pod "731b409e-e01b-42e0-b20b-ce7cfc1edd67" (UID: "731b409e-e01b-42e0-b20b-ce7cfc1edd67"). InnerVolumeSpecName "kube-api-access-ffr6f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:34:33.225472 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:33.225352 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffr6f\" (UniqueName: \"kubernetes.io/projected/731b409e-e01b-42e0-b20b-ce7cfc1edd67-kube-api-access-ffr6f\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:34:33.225472 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:33.225424 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/731b409e-e01b-42e0-b20b-ce7cfc1edd67-maas-api-xks-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:34:33.933685 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:33.933597 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e72197c4-vmwck" event={"ID":"731b409e-e01b-42e0-b20b-ce7cfc1edd67","Type":"ContainerDied","Data":"a89a29ab03a49fee73184302121fc13167431c4facb417cdb61f6304914ece05"} Oct 31 00:34:33.933685 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:33.933634 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e72197c4-vmwck" Oct 31 00:34:33.933685 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:33.933646 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a89a29ab03a49fee73184302121fc13167431c4facb417cdb61f6304914ece05" Oct 31 00:34:35.891057 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:34:35.891011 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 00:34:37.939002 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:37.938961 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-101b9e/distinct-2-w0-101b9e-kserve-74548f6d6-v5phc" Oct 31 00:34:37.952500 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:37.952472 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-101b9e/distinct-2-w0-101b9e-kserve-74548f6d6-v5phc" Oct 31 00:34:37.971014 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:37.970986 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-101b9e/distinct-2-w0-101b9e-kserve-74548f6d6-v5phc" Oct 31 00:34:44.921600 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:44.921516 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e6ec7b6-g6jfm"] Oct 31 00:34:44.922295 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:44.922277 5222 cpu_manager.go:414] "containerMap: removing container" podUID="731b409e-e01b-42e0-b20b-ce7cfc1edd67" containerName="revoke-keys" Oct 31 00:34:44.922347 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:44.922299 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="731b409e-e01b-42e0-b20b-ce7cfc1edd67" containerName="revoke-keys" Oct 31 00:34:44.922474 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:44.922461 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="731b409e-e01b-42e0-b20b-ce7cfc1edd67" containerName="revoke-keys" Oct 31 00:34:44.926157 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:44.926142 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e6ec7b6-g6jfm" Oct 31 00:34:44.979481 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:44.979448 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e6ec7b6-g6jfm"] Oct 31 00:34:45.026758 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:45.026726 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:34:45.026758 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:45.026745 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:34:45.026758 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:45.026753 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:34:45.031151 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:45.031134 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfz9z\"" Oct 31 00:34:45.060857 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:45.060823 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjpw2\" (UniqueName: \"kubernetes.io/projected/34925a08-5519-4b30-b8b7-30c4681d709b-kube-api-access-cjpw2\") pod \"maas-api-revoke-sub-5e6ec7b6-g6jfm\" (UID: \"34925a08-5519-4b30-b8b7-30c4681d709b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e6ec7b6-g6jfm" Oct 31 00:34:45.060977 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:45.060900 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34925a08-5519-4b30-b8b7-30c4681d709b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e6ec7b6-g6jfm\" (UID: \"34925a08-5519-4b30-b8b7-30c4681d709b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e6ec7b6-g6jfm" Oct 31 00:34:45.061014 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:45.060991 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34925a08-5519-4b30-b8b7-30c4681d709b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e6ec7b6-g6jfm\" (UID: \"34925a08-5519-4b30-b8b7-30c4681d709b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e6ec7b6-g6jfm" Oct 31 00:34:45.162495 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:45.162460 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34925a08-5519-4b30-b8b7-30c4681d709b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e6ec7b6-g6jfm\" (UID: \"34925a08-5519-4b30-b8b7-30c4681d709b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e6ec7b6-g6jfm" Oct 31 00:34:45.162673 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:45.162583 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjpw2\" (UniqueName: \"kubernetes.io/projected/34925a08-5519-4b30-b8b7-30c4681d709b-kube-api-access-cjpw2\") pod \"maas-api-revoke-sub-5e6ec7b6-g6jfm\" (UID: \"34925a08-5519-4b30-b8b7-30c4681d709b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e6ec7b6-g6jfm" Oct 31 00:34:45.162673 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:45.162627 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34925a08-5519-4b30-b8b7-30c4681d709b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e6ec7b6-g6jfm\" (UID: \"34925a08-5519-4b30-b8b7-30c4681d709b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e6ec7b6-g6jfm" Oct 31 00:34:45.163139 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:45.163116 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34925a08-5519-4b30-b8b7-30c4681d709b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e6ec7b6-g6jfm\" (UID: \"34925a08-5519-4b30-b8b7-30c4681d709b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e6ec7b6-g6jfm" Oct 31 00:34:45.165099 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:45.165076 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34925a08-5519-4b30-b8b7-30c4681d709b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e6ec7b6-g6jfm\" (UID: \"34925a08-5519-4b30-b8b7-30c4681d709b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e6ec7b6-g6jfm" Oct 31 00:34:45.225531 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:45.225456 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjpw2\" (UniqueName: \"kubernetes.io/projected/34925a08-5519-4b30-b8b7-30c4681d709b-kube-api-access-cjpw2\") pod \"maas-api-revoke-sub-5e6ec7b6-g6jfm\" (UID: \"34925a08-5519-4b30-b8b7-30c4681d709b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e6ec7b6-g6jfm" Oct 31 00:34:45.239333 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:45.239306 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e6ec7b6-g6jfm" Oct 31 00:34:45.616745 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:34:45.616700 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34925a08_5519_4b30_b8b7_30c4681d709b.slice/crio-1687695bbadfb0a637e14512ec30798116533e2d746674b1118955e2c1746232 WatchSource:0}: Error finding container 1687695bbadfb0a637e14512ec30798116533e2d746674b1118955e2c1746232: Status 404 returned error can't find the container with id 1687695bbadfb0a637e14512ec30798116533e2d746674b1118955e2c1746232 Oct 31 00:34:45.620592 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:45.620568 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e6ec7b6-g6jfm"] Oct 31 00:34:46.018446 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:46.018410 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e6ec7b6-g6jfm" event={"ID":"34925a08-5519-4b30-b8b7-30c4681d709b","Type":"ContainerStarted","Data":"cbb96a653344fea37fcb6d94c557504565a05b542eb2794fa546cd7ccf7a7cb2"} Oct 31 00:34:46.018916 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:46.018469 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e6ec7b6-g6jfm" event={"ID":"34925a08-5519-4b30-b8b7-30c4681d709b","Type":"ContainerStarted","Data":"1687695bbadfb0a637e14512ec30798116533e2d746674b1118955e2c1746232"} Oct 31 00:34:46.139859 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:46.139792 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e6ec7b6-g6jfm" podStartSLOduration=2.139776982 podStartE2EDuration="2.139776982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:34:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:34:46.137078403 +0000 UTC m=+1178.608248146" watchObservedRunningTime="2026-10-31 00:34:46.139776982 +0000 UTC m=+1178.610946757" Oct 31 00:34:47.027074 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:47.027032 5222 generic.go:356] "Generic (PLEG): container finished" podID="34925a08-5519-4b30-b8b7-30c4681d709b" containerID="cbb96a653344fea37fcb6d94c557504565a05b542eb2794fa546cd7ccf7a7cb2" exitCode=0 Oct 31 00:34:47.027580 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:47.027118 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e6ec7b6-g6jfm" event={"ID":"34925a08-5519-4b30-b8b7-30c4681d709b","Type":"ContainerDied","Data":"cbb96a653344fea37fcb6d94c557504565a05b542eb2794fa546cd7ccf7a7cb2"} Oct 31 00:34:48.199662 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:48.199630 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e6ec7b6-g6jfm" Oct 31 00:34:48.300590 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:48.300506 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34925a08-5519-4b30-b8b7-30c4681d709b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34925a08-5519-4b30-b8b7-30c4681d709b-maas-api-openshift-ca\") pod \"34925a08-5519-4b30-b8b7-30c4681d709b\" (UID: \"34925a08-5519-4b30-b8b7-30c4681d709b\") " Oct 31 00:34:48.300751 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:48.300649 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34925a08-5519-4b30-b8b7-30c4681d709b-kube-api-access-cjpw2\" (UniqueName: \"kubernetes.io/projected/34925a08-5519-4b30-b8b7-30c4681d709b-kube-api-access-cjpw2\") pod \"34925a08-5519-4b30-b8b7-30c4681d709b\" (UID: \"34925a08-5519-4b30-b8b7-30c4681d709b\") " Oct 31 00:34:48.300751 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:48.300669 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34925a08-5519-4b30-b8b7-30c4681d709b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34925a08-5519-4b30-b8b7-30c4681d709b-maas-api-xks-ca\") pod \"34925a08-5519-4b30-b8b7-30c4681d709b\" (UID: \"34925a08-5519-4b30-b8b7-30c4681d709b\") " Oct 31 00:34:48.300906 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:48.300879 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34925a08-5519-4b30-b8b7-30c4681d709b-maas-api-openshift-ca" pod "34925a08-5519-4b30-b8b7-30c4681d709b" (UID: "34925a08-5519-4b30-b8b7-30c4681d709b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:34:48.301140 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:48.301117 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34925a08-5519-4b30-b8b7-30c4681d709b-maas-api-openshift-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:34:48.302969 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:48.302945 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34925a08-5519-4b30-b8b7-30c4681d709b-maas-api-xks-ca" pod "34925a08-5519-4b30-b8b7-30c4681d709b" (UID: "34925a08-5519-4b30-b8b7-30c4681d709b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:34:48.303084 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:48.302974 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34925a08-5519-4b30-b8b7-30c4681d709b-kube-api-access-cjpw2" pod "34925a08-5519-4b30-b8b7-30c4681d709b" (UID: "34925a08-5519-4b30-b8b7-30c4681d709b"). InnerVolumeSpecName "kube-api-access-cjpw2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:34:48.402682 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:48.402647 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjpw2\" (UniqueName: \"kubernetes.io/projected/34925a08-5519-4b30-b8b7-30c4681d709b-kube-api-access-cjpw2\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:34:48.402682 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:48.402681 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34925a08-5519-4b30-b8b7-30c4681d709b-maas-api-xks-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:34:49.042214 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:49.042182 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e6ec7b6-g6jfm" Oct 31 00:34:49.042393 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:49.042214 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e6ec7b6-g6jfm" event={"ID":"34925a08-5519-4b30-b8b7-30c4681d709b","Type":"ContainerDied","Data":"1687695bbadfb0a637e14512ec30798116533e2d746674b1118955e2c1746232"} Oct 31 00:34:49.042393 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:49.042265 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1687695bbadfb0a637e14512ec30798116533e2d746674b1118955e2c1746232" Oct 31 00:34:52.827171 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:34:52.827137 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-79tk8"] Oct 31 00:35:05.890215 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:35:05.890172 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 00:35:07.806471 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:07.806434 5222 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h9hsf_0f8c3c03-a1fc-4376-98c6-feeeefa08068/ovn-acl-logging/0.log" Oct 31 00:35:07.809797 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:07.809774 5222 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h9hsf_0f8c3c03-a1fc-4376-98c6-feeeefa08068/ovn-acl-logging/0.log" Oct 31 00:35:12.013244 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:12.013207 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b8bd811-82hkb"] Oct 31 00:35:12.013754 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:12.013740 5222 cpu_manager.go:414] "containerMap: removing container" podUID="34925a08-5519-4b30-b8b7-30c4681d709b" containerName="revoke-keys" Oct 31 00:35:12.013800 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:12.013755 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34925a08-5519-4b30-b8b7-30c4681d709b" containerName="revoke-keys" Oct 31 00:35:12.013881 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:12.013872 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="34925a08-5519-4b30-b8b7-30c4681d709b" containerName="revoke-keys" Oct 31 00:35:12.018724 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:12.018706 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b8bd811-82hkb" Oct 31 00:35:12.072127 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:12.072092 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b8bd811-82hkb"] Oct 31 00:35:12.110721 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:12.110682 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4348f9bc-3109-48a5-a918-4511cf6d0fec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b8bd811-82hkb\" (UID: \"4348f9bc-3109-48a5-a918-4511cf6d0fec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b8bd811-82hkb" Oct 31 00:35:12.110721 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:12.110720 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2pp9\" (UniqueName: \"kubernetes.io/projected/4348f9bc-3109-48a5-a918-4511cf6d0fec-kube-api-access-x2pp9\") pod \"maas-api-revoke-sub-3b8bd811-82hkb\" (UID: \"4348f9bc-3109-48a5-a918-4511cf6d0fec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b8bd811-82hkb" Oct 31 00:35:12.110928 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:12.110921 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4348f9bc-3109-48a5-a918-4511cf6d0fec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b8bd811-82hkb\" (UID: \"4348f9bc-3109-48a5-a918-4511cf6d0fec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b8bd811-82hkb" Oct 31 00:35:12.118694 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:12.118667 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:35:12.118837 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:12.118706 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:35:12.118882 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:12.118834 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:35:12.123889 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:12.123870 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfz9z\"" Oct 31 00:35:12.212586 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:12.212549 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4348f9bc-3109-48a5-a918-4511cf6d0fec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b8bd811-82hkb\" (UID: \"4348f9bc-3109-48a5-a918-4511cf6d0fec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b8bd811-82hkb" Oct 31 00:35:12.212586 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:12.212586 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2pp9\" (UniqueName: \"kubernetes.io/projected/4348f9bc-3109-48a5-a918-4511cf6d0fec-kube-api-access-x2pp9\") pod \"maas-api-revoke-sub-3b8bd811-82hkb\" (UID: \"4348f9bc-3109-48a5-a918-4511cf6d0fec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b8bd811-82hkb" Oct 31 00:35:12.212837 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:12.212703 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4348f9bc-3109-48a5-a918-4511cf6d0fec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b8bd811-82hkb\" (UID: \"4348f9bc-3109-48a5-a918-4511cf6d0fec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b8bd811-82hkb" Oct 31 00:35:12.213285 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:12.213264 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4348f9bc-3109-48a5-a918-4511cf6d0fec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b8bd811-82hkb\" (UID: \"4348f9bc-3109-48a5-a918-4511cf6d0fec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b8bd811-82hkb" Oct 31 00:35:12.215132 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:12.215101 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4348f9bc-3109-48a5-a918-4511cf6d0fec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b8bd811-82hkb\" (UID: \"4348f9bc-3109-48a5-a918-4511cf6d0fec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b8bd811-82hkb" Oct 31 00:35:12.275511 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:12.275474 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2pp9\" (UniqueName: \"kubernetes.io/projected/4348f9bc-3109-48a5-a918-4511cf6d0fec-kube-api-access-x2pp9\") pod \"maas-api-revoke-sub-3b8bd811-82hkb\" (UID: \"4348f9bc-3109-48a5-a918-4511cf6d0fec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b8bd811-82hkb" Oct 31 00:35:12.332196 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:12.332158 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b8bd811-82hkb" Oct 31 00:35:12.709900 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:35:12.709866 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4348f9bc_3109_48a5_a918_4511cf6d0fec.slice/crio-bd78f581c0307a2a7b8bca2fc1adc7a2d1839bb561c20170880a6357370c80ea WatchSource:0}: Error finding container bd78f581c0307a2a7b8bca2fc1adc7a2d1839bb561c20170880a6357370c80ea: Status 404 returned error can't find the container with id bd78f581c0307a2a7b8bca2fc1adc7a2d1839bb561c20170880a6357370c80ea Oct 31 00:35:12.713361 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:12.713334 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b8bd811-82hkb"] Oct 31 00:35:13.200693 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:13.200651 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b8bd811-82hkb" event={"ID":"4348f9bc-3109-48a5-a918-4511cf6d0fec","Type":"ContainerStarted","Data":"2bbfba490a9fb4970d0d1601e91fe18a109870bf73ae03bc1cfd8a2ff6fbea22"} Oct 31 00:35:13.201087 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:13.200746 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b8bd811-82hkb" event={"ID":"4348f9bc-3109-48a5-a918-4511cf6d0fec","Type":"ContainerStarted","Data":"bd78f581c0307a2a7b8bca2fc1adc7a2d1839bb561c20170880a6357370c80ea"} Oct 31 00:35:13.320948 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:13.320835 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b8bd811-82hkb" podStartSLOduration=2.320821569 podStartE2EDuration="2.320821569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:35:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:35:13.318578108 +0000 UTC m=+1205.789747856" watchObservedRunningTime="2026-10-31 00:35:13.320821569 +0000 UTC m=+1205.791991318" Oct 31 00:35:14.209153 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:14.209116 5222 generic.go:356] "Generic (PLEG): container finished" podID="4348f9bc-3109-48a5-a918-4511cf6d0fec" containerID="2bbfba490a9fb4970d0d1601e91fe18a109870bf73ae03bc1cfd8a2ff6fbea22" exitCode=0 Oct 31 00:35:14.209573 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:14.209201 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b8bd811-82hkb" event={"ID":"4348f9bc-3109-48a5-a918-4511cf6d0fec","Type":"ContainerDied","Data":"2bbfba490a9fb4970d0d1601e91fe18a109870bf73ae03bc1cfd8a2ff6fbea22"} Oct 31 00:35:15.377239 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:15.377207 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b8bd811-82hkb" Oct 31 00:35:15.453076 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:15.453041 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4348f9bc-3109-48a5-a918-4511cf6d0fec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4348f9bc-3109-48a5-a918-4511cf6d0fec-maas-api-openshift-ca\") pod \"4348f9bc-3109-48a5-a918-4511cf6d0fec\" (UID: \"4348f9bc-3109-48a5-a918-4511cf6d0fec\") " Oct 31 00:35:15.453283 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:15.453188 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4348f9bc-3109-48a5-a918-4511cf6d0fec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4348f9bc-3109-48a5-a918-4511cf6d0fec-maas-api-xks-ca\") pod \"4348f9bc-3109-48a5-a918-4511cf6d0fec\" (UID: \"4348f9bc-3109-48a5-a918-4511cf6d0fec\") " Oct 31 00:35:15.453283 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:15.453265 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4348f9bc-3109-48a5-a918-4511cf6d0fec-kube-api-access-x2pp9\" (UniqueName: \"kubernetes.io/projected/4348f9bc-3109-48a5-a918-4511cf6d0fec-kube-api-access-x2pp9\") pod \"4348f9bc-3109-48a5-a918-4511cf6d0fec\" (UID: \"4348f9bc-3109-48a5-a918-4511cf6d0fec\") " Oct 31 00:35:15.453543 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:15.453516 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4348f9bc-3109-48a5-a918-4511cf6d0fec-maas-api-openshift-ca" pod "4348f9bc-3109-48a5-a918-4511cf6d0fec" (UID: "4348f9bc-3109-48a5-a918-4511cf6d0fec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:35:15.453844 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:15.453823 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4348f9bc-3109-48a5-a918-4511cf6d0fec-maas-api-openshift-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:35:15.455573 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:15.455547 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4348f9bc-3109-48a5-a918-4511cf6d0fec-kube-api-access-x2pp9" pod "4348f9bc-3109-48a5-a918-4511cf6d0fec" (UID: "4348f9bc-3109-48a5-a918-4511cf6d0fec"). InnerVolumeSpecName "kube-api-access-x2pp9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:35:15.455676 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:15.455631 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4348f9bc-3109-48a5-a918-4511cf6d0fec-maas-api-xks-ca" pod "4348f9bc-3109-48a5-a918-4511cf6d0fec" (UID: "4348f9bc-3109-48a5-a918-4511cf6d0fec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:35:15.555486 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:15.555447 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4348f9bc-3109-48a5-a918-4511cf6d0fec-maas-api-xks-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:35:15.555486 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:15.555480 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2pp9\" (UniqueName: \"kubernetes.io/projected/4348f9bc-3109-48a5-a918-4511cf6d0fec-kube-api-access-x2pp9\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:35:16.224688 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:16.224659 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b8bd811-82hkb" Oct 31 00:35:16.224955 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:16.224696 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b8bd811-82hkb" event={"ID":"4348f9bc-3109-48a5-a918-4511cf6d0fec","Type":"ContainerDied","Data":"bd78f581c0307a2a7b8bca2fc1adc7a2d1839bb561c20170880a6357370c80ea"} Oct 31 00:35:16.224955 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:16.224745 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bd78f581c0307a2a7b8bca2fc1adc7a2d1839bb561c20170880a6357370c80ea" Oct 31 00:35:27.699702 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:27.699657 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5cc9374-m57hz"] Oct 31 00:35:27.700464 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:27.700443 5222 cpu_manager.go:414] "containerMap: removing container" podUID="4348f9bc-3109-48a5-a918-4511cf6d0fec" containerName="revoke-keys" Oct 31 00:35:27.700527 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:27.700468 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4348f9bc-3109-48a5-a918-4511cf6d0fec" containerName="revoke-keys" Oct 31 00:35:27.700646 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:27.700632 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="4348f9bc-3109-48a5-a918-4511cf6d0fec" containerName="revoke-keys" Oct 31 00:35:27.704194 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:27.704173 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5cc9374-m57hz" Oct 31 00:35:27.757577 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:27.757543 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5cc9374-m57hz"] Oct 31 00:35:27.804984 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:27.804950 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:35:27.804984 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:27.804979 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:35:27.804984 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:27.804991 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:35:27.805615 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:27.805590 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c55e12e-e99d-48c7-b2cd-d094309b4035-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5cc9374-m57hz\" (UID: \"3c55e12e-e99d-48c7-b2cd-d094309b4035\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5cc9374-m57hz" Oct 31 00:35:27.805737 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:27.805646 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c55e12e-e99d-48c7-b2cd-d094309b4035-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5cc9374-m57hz\" (UID: \"3c55e12e-e99d-48c7-b2cd-d094309b4035\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5cc9374-m57hz" Oct 31 00:35:27.805836 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:27.805821 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7d2n\" (UniqueName: \"kubernetes.io/projected/3c55e12e-e99d-48c7-b2cd-d094309b4035-kube-api-access-r7d2n\") pod \"maas-api-revoke-sub-e5cc9374-m57hz\" (UID: \"3c55e12e-e99d-48c7-b2cd-d094309b4035\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5cc9374-m57hz" Oct 31 00:35:27.808758 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:27.808742 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfz9z\"" Oct 31 00:35:27.906932 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:27.906894 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7d2n\" (UniqueName: \"kubernetes.io/projected/3c55e12e-e99d-48c7-b2cd-d094309b4035-kube-api-access-r7d2n\") pod \"maas-api-revoke-sub-e5cc9374-m57hz\" (UID: \"3c55e12e-e99d-48c7-b2cd-d094309b4035\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5cc9374-m57hz" Oct 31 00:35:27.907120 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:27.906993 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c55e12e-e99d-48c7-b2cd-d094309b4035-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5cc9374-m57hz\" (UID: \"3c55e12e-e99d-48c7-b2cd-d094309b4035\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5cc9374-m57hz" Oct 31 00:35:27.907120 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:27.907025 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c55e12e-e99d-48c7-b2cd-d094309b4035-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5cc9374-m57hz\" (UID: \"3c55e12e-e99d-48c7-b2cd-d094309b4035\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5cc9374-m57hz" Oct 31 00:35:27.907665 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:27.907647 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c55e12e-e99d-48c7-b2cd-d094309b4035-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5cc9374-m57hz\" (UID: \"3c55e12e-e99d-48c7-b2cd-d094309b4035\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5cc9374-m57hz" Oct 31 00:35:27.909552 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:27.909534 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c55e12e-e99d-48c7-b2cd-d094309b4035-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5cc9374-m57hz\" (UID: \"3c55e12e-e99d-48c7-b2cd-d094309b4035\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5cc9374-m57hz" Oct 31 00:35:27.969727 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:27.969656 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7d2n\" (UniqueName: \"kubernetes.io/projected/3c55e12e-e99d-48c7-b2cd-d094309b4035-kube-api-access-r7d2n\") pod \"maas-api-revoke-sub-e5cc9374-m57hz\" (UID: \"3c55e12e-e99d-48c7-b2cd-d094309b4035\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5cc9374-m57hz" Oct 31 00:35:28.018713 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:28.018681 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5cc9374-m57hz" Oct 31 00:35:28.401442 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:35:28.401406 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c55e12e_e99d_48c7_b2cd_d094309b4035.slice/crio-2089a2aa8c5cbb30ff086d6c48c9401c4887a18fefe3ac63893d4c7c88b5e5a4 WatchSource:0}: Error finding container 2089a2aa8c5cbb30ff086d6c48c9401c4887a18fefe3ac63893d4c7c88b5e5a4: Status 404 returned error can't find the container with id 2089a2aa8c5cbb30ff086d6c48c9401c4887a18fefe3ac63893d4c7c88b5e5a4 Oct 31 00:35:28.405083 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:28.404936 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5cc9374-m57hz"] Oct 31 00:35:29.314055 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:29.314016 5222 generic.go:356] "Generic (PLEG): container finished" podID="3c55e12e-e99d-48c7-b2cd-d094309b4035" containerID="9f08237735c945c768fa68436434d5ffb4da83b3e33022ab018dc1c205e0cef4" exitCode=0 Oct 31 00:35:29.314055 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:29.314054 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5cc9374-m57hz" event={"ID":"3c55e12e-e99d-48c7-b2cd-d094309b4035","Type":"ContainerDied","Data":"9f08237735c945c768fa68436434d5ffb4da83b3e33022ab018dc1c205e0cef4"} Oct 31 00:35:29.314531 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:29.314106 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5cc9374-m57hz" event={"ID":"3c55e12e-e99d-48c7-b2cd-d094309b4035","Type":"ContainerStarted","Data":"2089a2aa8c5cbb30ff086d6c48c9401c4887a18fefe3ac63893d4c7c88b5e5a4"} Oct 31 00:35:30.463713 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:30.463686 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5cc9374-m57hz" Oct 31 00:35:30.638903 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:30.638809 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c55e12e-e99d-48c7-b2cd-d094309b4035-kube-api-access-r7d2n\" (UniqueName: \"kubernetes.io/projected/3c55e12e-e99d-48c7-b2cd-d094309b4035-kube-api-access-r7d2n\") pod \"3c55e12e-e99d-48c7-b2cd-d094309b4035\" (UID: \"3c55e12e-e99d-48c7-b2cd-d094309b4035\") " Oct 31 00:35:30.639091 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:30.638914 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c55e12e-e99d-48c7-b2cd-d094309b4035-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c55e12e-e99d-48c7-b2cd-d094309b4035-maas-api-xks-ca\") pod \"3c55e12e-e99d-48c7-b2cd-d094309b4035\" (UID: \"3c55e12e-e99d-48c7-b2cd-d094309b4035\") " Oct 31 00:35:30.639091 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:30.638939 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c55e12e-e99d-48c7-b2cd-d094309b4035-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c55e12e-e99d-48c7-b2cd-d094309b4035-maas-api-openshift-ca\") pod \"3c55e12e-e99d-48c7-b2cd-d094309b4035\" (UID: \"3c55e12e-e99d-48c7-b2cd-d094309b4035\") " Oct 31 00:35:30.639360 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:30.639335 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c55e12e-e99d-48c7-b2cd-d094309b4035-maas-api-openshift-ca" pod "3c55e12e-e99d-48c7-b2cd-d094309b4035" (UID: "3c55e12e-e99d-48c7-b2cd-d094309b4035"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:35:30.641143 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:30.641120 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c55e12e-e99d-48c7-b2cd-d094309b4035-maas-api-xks-ca" pod "3c55e12e-e99d-48c7-b2cd-d094309b4035" (UID: "3c55e12e-e99d-48c7-b2cd-d094309b4035"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:35:30.641278 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:30.641257 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c55e12e-e99d-48c7-b2cd-d094309b4035-kube-api-access-r7d2n" pod "3c55e12e-e99d-48c7-b2cd-d094309b4035" (UID: "3c55e12e-e99d-48c7-b2cd-d094309b4035"). InnerVolumeSpecName "kube-api-access-r7d2n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:35:30.740564 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:30.740523 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c55e12e-e99d-48c7-b2cd-d094309b4035-maas-api-xks-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:35:30.740564 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:30.740557 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c55e12e-e99d-48c7-b2cd-d094309b4035-maas-api-openshift-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:35:30.740564 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:30.740567 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7d2n\" (UniqueName: \"kubernetes.io/projected/3c55e12e-e99d-48c7-b2cd-d094309b4035-kube-api-access-r7d2n\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:35:31.329708 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:31.329671 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5cc9374-m57hz" Oct 31 00:35:31.329904 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:31.329713 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5cc9374-m57hz" event={"ID":"3c55e12e-e99d-48c7-b2cd-d094309b4035","Type":"ContainerDied","Data":"2089a2aa8c5cbb30ff086d6c48c9401c4887a18fefe3ac63893d4c7c88b5e5a4"} Oct 31 00:35:31.329904 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:31.329798 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2089a2aa8c5cbb30ff086d6c48c9401c4887a18fefe3ac63893d4c7c88b5e5a4" Oct 31 00:35:35.059263 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:35.059226 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6b8b0b9-ddc5k"] Oct 31 00:35:35.108750 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:35.108702 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6b8b0b9-ddc5k"] Oct 31 00:35:35.656660 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:35.656623 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c47303f2-efc8-4a4e-bffd-fb123dfc9707" path="/var/lib/kubelet/pods/c47303f2-efc8-4a4e-bffd-fb123dfc9707/volumes" Oct 31 00:35:35.891085 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:35:35.891047 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 00:35:46.653571 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:46.653539 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-79tk8"] Oct 31 00:35:49.149255 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:49.149220 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-412743/distinct-2-w5-412743-kserve-5b679b6b66-59prz"] Oct 31 00:35:49.149863 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:49.149720 5222 cpu_manager.go:414] "containerMap: removing container" podUID="3c55e12e-e99d-48c7-b2cd-d094309b4035" containerName="revoke-keys" Oct 31 00:35:49.149863 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:49.149732 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c55e12e-e99d-48c7-b2cd-d094309b4035" containerName="revoke-keys" Oct 31 00:35:49.149863 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:49.149851 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c55e12e-e99d-48c7-b2cd-d094309b4035" containerName="revoke-keys" Oct 31 00:35:49.157193 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:49.157162 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-412743/distinct-2-w5-412743-kserve-5b679b6b66-59prz" Oct 31 00:35:49.210116 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:49.210074 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-412743/distinct-2-w5-412743-kserve-5b679b6b66-59prz"] Oct 31 00:35:49.255262 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:49.255231 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-412743\"/\"distinct-2-w5-412743-kserve-self-signed-certs\"" Oct 31 00:35:49.294222 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:49.294187 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f6cbca5-d586-4bc2-9114-7212e9cc8a01-dshm\") pod \"distinct-2-w5-412743-kserve-5b679b6b66-59prz\" (UID: \"2f6cbca5-d586-4bc2-9114-7212e9cc8a01\") " pod="e2e-models-e2e-worker-w5-412743/distinct-2-w5-412743-kserve-5b679b6b66-59prz" Oct 31 00:35:49.296493 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:49.296463 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f6cbca5-d586-4bc2-9114-7212e9cc8a01-tls-certs\") pod \"distinct-2-w5-412743-kserve-5b679b6b66-59prz\" (UID: \"2f6cbca5-d586-4bc2-9114-7212e9cc8a01\") " pod="e2e-models-e2e-worker-w5-412743/distinct-2-w5-412743-kserve-5b679b6b66-59prz" Oct 31 00:35:49.296766 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:49.296746 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54djx\" (UniqueName: \"kubernetes.io/projected/2f6cbca5-d586-4bc2-9114-7212e9cc8a01-kube-api-access-54djx\") pod \"distinct-2-w5-412743-kserve-5b679b6b66-59prz\" (UID: \"2f6cbca5-d586-4bc2-9114-7212e9cc8a01\") " pod="e2e-models-e2e-worker-w5-412743/distinct-2-w5-412743-kserve-5b679b6b66-59prz" Oct 31 00:35:49.297231 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:49.297210 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f6cbca5-d586-4bc2-9114-7212e9cc8a01-model-cache\") pod \"distinct-2-w5-412743-kserve-5b679b6b66-59prz\" (UID: \"2f6cbca5-d586-4bc2-9114-7212e9cc8a01\") " pod="e2e-models-e2e-worker-w5-412743/distinct-2-w5-412743-kserve-5b679b6b66-59prz" Oct 31 00:35:49.297409 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:49.297392 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f6cbca5-d586-4bc2-9114-7212e9cc8a01-home\") pod \"distinct-2-w5-412743-kserve-5b679b6b66-59prz\" (UID: \"2f6cbca5-d586-4bc2-9114-7212e9cc8a01\") " pod="e2e-models-e2e-worker-w5-412743/distinct-2-w5-412743-kserve-5b679b6b66-59prz" Oct 31 00:35:49.297614 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:49.297587 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f6cbca5-d586-4bc2-9114-7212e9cc8a01-tmp-dir\") pod \"distinct-2-w5-412743-kserve-5b679b6b66-59prz\" (UID: \"2f6cbca5-d586-4bc2-9114-7212e9cc8a01\") " pod="e2e-models-e2e-worker-w5-412743/distinct-2-w5-412743-kserve-5b679b6b66-59prz" Oct 31 00:35:49.398935 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:49.398877 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f6cbca5-d586-4bc2-9114-7212e9cc8a01-dshm\") pod \"distinct-2-w5-412743-kserve-5b679b6b66-59prz\" (UID: \"2f6cbca5-d586-4bc2-9114-7212e9cc8a01\") " pod="e2e-models-e2e-worker-w5-412743/distinct-2-w5-412743-kserve-5b679b6b66-59prz" Oct 31 00:35:49.398935 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:49.398934 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f6cbca5-d586-4bc2-9114-7212e9cc8a01-tls-certs\") pod \"distinct-2-w5-412743-kserve-5b679b6b66-59prz\" (UID: \"2f6cbca5-d586-4bc2-9114-7212e9cc8a01\") " pod="e2e-models-e2e-worker-w5-412743/distinct-2-w5-412743-kserve-5b679b6b66-59prz" Oct 31 00:35:49.399207 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:49.398964 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54djx\" (UniqueName: \"kubernetes.io/projected/2f6cbca5-d586-4bc2-9114-7212e9cc8a01-kube-api-access-54djx\") pod \"distinct-2-w5-412743-kserve-5b679b6b66-59prz\" (UID: \"2f6cbca5-d586-4bc2-9114-7212e9cc8a01\") " pod="e2e-models-e2e-worker-w5-412743/distinct-2-w5-412743-kserve-5b679b6b66-59prz" Oct 31 00:35:49.399207 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:49.398988 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f6cbca5-d586-4bc2-9114-7212e9cc8a01-model-cache\") pod \"distinct-2-w5-412743-kserve-5b679b6b66-59prz\" (UID: \"2f6cbca5-d586-4bc2-9114-7212e9cc8a01\") " pod="e2e-models-e2e-worker-w5-412743/distinct-2-w5-412743-kserve-5b679b6b66-59prz" Oct 31 00:35:49.399207 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:49.399117 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f6cbca5-d586-4bc2-9114-7212e9cc8a01-home\") pod \"distinct-2-w5-412743-kserve-5b679b6b66-59prz\" (UID: \"2f6cbca5-d586-4bc2-9114-7212e9cc8a01\") " pod="e2e-models-e2e-worker-w5-412743/distinct-2-w5-412743-kserve-5b679b6b66-59prz" Oct 31 00:35:49.399417 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:49.399211 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f6cbca5-d586-4bc2-9114-7212e9cc8a01-tmp-dir\") pod \"distinct-2-w5-412743-kserve-5b679b6b66-59prz\" (UID: \"2f6cbca5-d586-4bc2-9114-7212e9cc8a01\") " pod="e2e-models-e2e-worker-w5-412743/distinct-2-w5-412743-kserve-5b679b6b66-59prz" Oct 31 00:35:49.399580 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:49.399558 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f6cbca5-d586-4bc2-9114-7212e9cc8a01-model-cache\") pod \"distinct-2-w5-412743-kserve-5b679b6b66-59prz\" (UID: \"2f6cbca5-d586-4bc2-9114-7212e9cc8a01\") " pod="e2e-models-e2e-worker-w5-412743/distinct-2-w5-412743-kserve-5b679b6b66-59prz" Oct 31 00:35:49.399640 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:49.399586 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f6cbca5-d586-4bc2-9114-7212e9cc8a01-tmp-dir\") pod \"distinct-2-w5-412743-kserve-5b679b6b66-59prz\" (UID: \"2f6cbca5-d586-4bc2-9114-7212e9cc8a01\") " pod="e2e-models-e2e-worker-w5-412743/distinct-2-w5-412743-kserve-5b679b6b66-59prz" Oct 31 00:35:49.399809 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:49.399752 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f6cbca5-d586-4bc2-9114-7212e9cc8a01-home\") pod \"distinct-2-w5-412743-kserve-5b679b6b66-59prz\" (UID: \"2f6cbca5-d586-4bc2-9114-7212e9cc8a01\") " pod="e2e-models-e2e-worker-w5-412743/distinct-2-w5-412743-kserve-5b679b6b66-59prz" Oct 31 00:35:49.401673 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:49.401649 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f6cbca5-d586-4bc2-9114-7212e9cc8a01-dshm\") pod \"distinct-2-w5-412743-kserve-5b679b6b66-59prz\" (UID: \"2f6cbca5-d586-4bc2-9114-7212e9cc8a01\") " pod="e2e-models-e2e-worker-w5-412743/distinct-2-w5-412743-kserve-5b679b6b66-59prz" Oct 31 00:35:49.401783 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:49.401656 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f6cbca5-d586-4bc2-9114-7212e9cc8a01-tls-certs\") pod \"distinct-2-w5-412743-kserve-5b679b6b66-59prz\" (UID: \"2f6cbca5-d586-4bc2-9114-7212e9cc8a01\") " pod="e2e-models-e2e-worker-w5-412743/distinct-2-w5-412743-kserve-5b679b6b66-59prz" Oct 31 00:35:49.463410 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:49.463338 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54djx\" (UniqueName: \"kubernetes.io/projected/2f6cbca5-d586-4bc2-9114-7212e9cc8a01-kube-api-access-54djx\") pod \"distinct-2-w5-412743-kserve-5b679b6b66-59prz\" (UID: \"2f6cbca5-d586-4bc2-9114-7212e9cc8a01\") " pod="e2e-models-e2e-worker-w5-412743/distinct-2-w5-412743-kserve-5b679b6b66-59prz" Oct 31 00:35:49.473467 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:49.473440 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-412743/distinct-2-w5-412743-kserve-5b679b6b66-59prz" Oct 31 00:35:49.869692 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:35:49.869659 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2f6cbca5_d586_4bc2_9114_7212e9cc8a01.slice/crio-a9ba501a899a961eaef0739b040f8cca89566f82296780737d7901e55af6fe0a WatchSource:0}: Error finding container a9ba501a899a961eaef0739b040f8cca89566f82296780737d7901e55af6fe0a: Status 404 returned error can't find the container with id a9ba501a899a961eaef0739b040f8cca89566f82296780737d7901e55af6fe0a Oct 31 00:35:49.872964 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:49.872942 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-412743/distinct-2-w5-412743-kserve-5b679b6b66-59prz"] Oct 31 00:35:50.483671 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:50.483628 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-412743/distinct-2-w5-412743-kserve-5b679b6b66-59prz" event={"ID":"2f6cbca5-d586-4bc2-9114-7212e9cc8a01","Type":"ContainerStarted","Data":"d294543ced999ad1705a2300a6ba610d93c3b8724d4247fe4587a2316de63800"} Oct 31 00:35:50.484082 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:50.483682 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-412743/distinct-2-w5-412743-kserve-5b679b6b66-59prz" event={"ID":"2f6cbca5-d586-4bc2-9114-7212e9cc8a01","Type":"ContainerStarted","Data":"a9ba501a899a961eaef0739b040f8cca89566f82296780737d7901e55af6fe0a"} Oct 31 00:35:50.608129 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:50.608077 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-412743/distinct-2-w5-412743-kserve-5b679b6b66-59prz" podStartSLOduration=1.608055723 podStartE2EDuration="1.608055723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:35:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:35:50.604048869 +0000 UTC m=+1243.075218618" watchObservedRunningTime="2026-10-31 00:35:50.608055723 +0000 UTC m=+1243.079225472" Oct 31 00:35:51.059496 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:51.059459 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-faf6816a-8t8tg"] Oct 31 00:35:51.065480 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:51.065454 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-faf6816a-8t8tg"] Oct 31 00:35:51.474614 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:51.474526 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-412743/distinct-2-w5-412743-kserve-5b679b6b66-59prz" Oct 31 00:35:51.656991 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:51.656952 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31d1a760-c354-43e8-b130-eb62fcb9f2f1" path="/var/lib/kubelet/pods/31d1a760-c354-43e8-b130-eb62fcb9f2f1/volumes" Oct 31 00:35:52.242972 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:52.242941 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ddabde0d-46kf9"] Oct 31 00:35:52.247185 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:52.247167 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddabde0d-46kf9" Oct 31 00:35:52.301778 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:52.301748 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:35:52.303115 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:52.303085 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ddabde0d-46kf9"] Oct 31 00:35:52.335730 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:52.335687 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jndjh\" (UniqueName: \"kubernetes.io/projected/a346f5b5-f696-43cd-a709-be781e0a5d18-kube-api-access-jndjh\") pod \"maas-api-revoke-sub-ddabde0d-46kf9\" (UID: \"a346f5b5-f696-43cd-a709-be781e0a5d18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddabde0d-46kf9" Oct 31 00:35:52.335927 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:52.335748 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a346f5b5-f696-43cd-a709-be781e0a5d18-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ddabde0d-46kf9\" (UID: \"a346f5b5-f696-43cd-a709-be781e0a5d18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddabde0d-46kf9" Oct 31 00:35:52.335927 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:52.335868 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a346f5b5-f696-43cd-a709-be781e0a5d18-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ddabde0d-46kf9\" (UID: \"a346f5b5-f696-43cd-a709-be781e0a5d18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddabde0d-46kf9" Oct 31 00:35:52.349798 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:52.349768 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:35:52.349983 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:52.349833 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:35:52.349983 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:52.349850 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfz9z\"" Oct 31 00:35:52.437620 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:52.437578 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jndjh\" (UniqueName: \"kubernetes.io/projected/a346f5b5-f696-43cd-a709-be781e0a5d18-kube-api-access-jndjh\") pod \"maas-api-revoke-sub-ddabde0d-46kf9\" (UID: \"a346f5b5-f696-43cd-a709-be781e0a5d18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddabde0d-46kf9" Oct 31 00:35:52.437833 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:52.437640 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a346f5b5-f696-43cd-a709-be781e0a5d18-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ddabde0d-46kf9\" (UID: \"a346f5b5-f696-43cd-a709-be781e0a5d18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddabde0d-46kf9" Oct 31 00:35:52.437833 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:52.437677 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a346f5b5-f696-43cd-a709-be781e0a5d18-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ddabde0d-46kf9\" (UID: \"a346f5b5-f696-43cd-a709-be781e0a5d18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddabde0d-46kf9" Oct 31 00:35:52.438328 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:52.438304 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a346f5b5-f696-43cd-a709-be781e0a5d18-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ddabde0d-46kf9\" (UID: \"a346f5b5-f696-43cd-a709-be781e0a5d18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddabde0d-46kf9" Oct 31 00:35:52.440290 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:52.440257 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a346f5b5-f696-43cd-a709-be781e0a5d18-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ddabde0d-46kf9\" (UID: \"a346f5b5-f696-43cd-a709-be781e0a5d18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddabde0d-46kf9" Oct 31 00:35:52.500150 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:52.500078 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jndjh\" (UniqueName: \"kubernetes.io/projected/a346f5b5-f696-43cd-a709-be781e0a5d18-kube-api-access-jndjh\") pod \"maas-api-revoke-sub-ddabde0d-46kf9\" (UID: \"a346f5b5-f696-43cd-a709-be781e0a5d18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddabde0d-46kf9" Oct 31 00:35:52.560789 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:52.560752 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddabde0d-46kf9" Oct 31 00:35:52.946127 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:52.945588 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ddabde0d-46kf9"] Oct 31 00:35:52.953937 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:35:52.953900 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda346f5b5_f696_43cd_a709_be781e0a5d18.slice/crio-5cf7651f38df58740e11e294870420a01743c462651600ac4b8a6d8940f4c2cf WatchSource:0}: Error finding container 5cf7651f38df58740e11e294870420a01743c462651600ac4b8a6d8940f4c2cf: Status 404 returned error can't find the container with id 5cf7651f38df58740e11e294870420a01743c462651600ac4b8a6d8940f4c2cf Oct 31 00:35:53.505259 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:53.505167 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddabde0d-46kf9" event={"ID":"a346f5b5-f696-43cd-a709-be781e0a5d18","Type":"ContainerStarted","Data":"976764b4462b3bc02b6bc9df61bdecf06f095371451da245985fee27d13351bc"} Oct 31 00:35:53.505259 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:53.505229 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddabde0d-46kf9" event={"ID":"a346f5b5-f696-43cd-a709-be781e0a5d18","Type":"ContainerStarted","Data":"5cf7651f38df58740e11e294870420a01743c462651600ac4b8a6d8940f4c2cf"} Oct 31 00:35:53.624997 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:53.624936 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddabde0d-46kf9" podStartSLOduration=1.624917978 podStartE2EDuration="1.624917978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:35:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:35:53.623873825 +0000 UTC m=+1246.095043573" watchObservedRunningTime="2026-10-31 00:35:53.624917978 +0000 UTC m=+1246.096088152" Oct 31 00:35:54.514470 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:54.514429 5222 generic.go:356] "Generic (PLEG): container finished" podID="a346f5b5-f696-43cd-a709-be781e0a5d18" containerID="976764b4462b3bc02b6bc9df61bdecf06f095371451da245985fee27d13351bc" exitCode=0 Oct 31 00:35:54.514925 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:54.514508 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddabde0d-46kf9" event={"ID":"a346f5b5-f696-43cd-a709-be781e0a5d18","Type":"ContainerDied","Data":"976764b4462b3bc02b6bc9df61bdecf06f095371451da245985fee27d13351bc"} Oct 31 00:35:55.678115 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:55.678089 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddabde0d-46kf9" Oct 31 00:35:55.778126 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:55.778027 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a346f5b5-f696-43cd-a709-be781e0a5d18-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a346f5b5-f696-43cd-a709-be781e0a5d18-maas-api-openshift-ca\") pod \"a346f5b5-f696-43cd-a709-be781e0a5d18\" (UID: \"a346f5b5-f696-43cd-a709-be781e0a5d18\") " Oct 31 00:35:55.778126 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:55.778076 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a346f5b5-f696-43cd-a709-be781e0a5d18-kube-api-access-jndjh\" (UniqueName: \"kubernetes.io/projected/a346f5b5-f696-43cd-a709-be781e0a5d18-kube-api-access-jndjh\") pod \"a346f5b5-f696-43cd-a709-be781e0a5d18\" (UID: \"a346f5b5-f696-43cd-a709-be781e0a5d18\") " Oct 31 00:35:55.778364 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:55.778293 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a346f5b5-f696-43cd-a709-be781e0a5d18-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a346f5b5-f696-43cd-a709-be781e0a5d18-maas-api-xks-ca\") pod \"a346f5b5-f696-43cd-a709-be781e0a5d18\" (UID: \"a346f5b5-f696-43cd-a709-be781e0a5d18\") " Oct 31 00:35:55.778460 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:55.778388 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a346f5b5-f696-43cd-a709-be781e0a5d18-maas-api-openshift-ca" pod "a346f5b5-f696-43cd-a709-be781e0a5d18" (UID: "a346f5b5-f696-43cd-a709-be781e0a5d18"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:35:55.778779 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:55.778753 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a346f5b5-f696-43cd-a709-be781e0a5d18-maas-api-openshift-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:35:55.780457 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:55.780425 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a346f5b5-f696-43cd-a709-be781e0a5d18-maas-api-xks-ca" pod "a346f5b5-f696-43cd-a709-be781e0a5d18" (UID: "a346f5b5-f696-43cd-a709-be781e0a5d18"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:35:55.780563 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:55.780501 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a346f5b5-f696-43cd-a709-be781e0a5d18-kube-api-access-jndjh" pod "a346f5b5-f696-43cd-a709-be781e0a5d18" (UID: "a346f5b5-f696-43cd-a709-be781e0a5d18"). InnerVolumeSpecName "kube-api-access-jndjh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:35:55.879525 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:55.879483 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a346f5b5-f696-43cd-a709-be781e0a5d18-maas-api-xks-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:35:55.879525 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:55.879520 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jndjh\" (UniqueName: \"kubernetes.io/projected/a346f5b5-f696-43cd-a709-be781e0a5d18-kube-api-access-jndjh\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:35:56.530582 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:56.530552 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddabde0d-46kf9" Oct 31 00:35:56.530753 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:56.530583 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddabde0d-46kf9" event={"ID":"a346f5b5-f696-43cd-a709-be781e0a5d18","Type":"ContainerDied","Data":"5cf7651f38df58740e11e294870420a01743c462651600ac4b8a6d8940f4c2cf"} Oct 31 00:35:56.530753 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:56.530648 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5cf7651f38df58740e11e294870420a01743c462651600ac4b8a6d8940f4c2cf" Oct 31 00:35:59.474179 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:59.474135 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-412743/distinct-2-w5-412743-kserve-5b679b6b66-59prz" Oct 31 00:35:59.487488 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:59.487462 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-412743/distinct-2-w5-412743-kserve-5b679b6b66-59prz" Oct 31 00:35:59.563390 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:35:59.563351 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-412743/distinct-2-w5-412743-kserve-5b679b6b66-59prz" Oct 31 00:36:05.890355 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:36:05.890310 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 00:36:08.511498 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:08.511461 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f7d1a52-j7nfm"] Oct 31 00:36:08.512012 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:08.511999 5222 cpu_manager.go:414] "containerMap: removing container" podUID="a346f5b5-f696-43cd-a709-be781e0a5d18" containerName="revoke-keys" Oct 31 00:36:08.512063 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:08.512013 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a346f5b5-f696-43cd-a709-be781e0a5d18" containerName="revoke-keys" Oct 31 00:36:08.512100 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:08.512096 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="a346f5b5-f696-43cd-a709-be781e0a5d18" containerName="revoke-keys" Oct 31 00:36:08.516998 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:08.516978 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f7d1a52-j7nfm" Oct 31 00:36:08.569669 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:08.569636 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f7d1a52-j7nfm"] Oct 31 00:36:08.617256 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:08.617229 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:36:08.621743 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:08.621727 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:36:08.621867 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:08.621750 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:36:08.621867 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:08.621768 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfz9z\"" Oct 31 00:36:08.636583 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:08.636550 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c5fe715-e790-4c03-8cb9-7c3c726e75d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f7d1a52-j7nfm\" (UID: \"3c5fe715-e790-4c03-8cb9-7c3c726e75d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f7d1a52-j7nfm" Oct 31 00:36:08.636770 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:08.636605 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fh6xk\" (UniqueName: \"kubernetes.io/projected/3c5fe715-e790-4c03-8cb9-7c3c726e75d0-kube-api-access-fh6xk\") pod \"maas-api-revoke-sub-8f7d1a52-j7nfm\" (UID: \"3c5fe715-e790-4c03-8cb9-7c3c726e75d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f7d1a52-j7nfm" Oct 31 00:36:08.636841 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:08.636774 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c5fe715-e790-4c03-8cb9-7c3c726e75d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f7d1a52-j7nfm\" (UID: \"3c5fe715-e790-4c03-8cb9-7c3c726e75d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f7d1a52-j7nfm" Oct 31 00:36:08.737537 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:08.737510 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c5fe715-e790-4c03-8cb9-7c3c726e75d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f7d1a52-j7nfm\" (UID: \"3c5fe715-e790-4c03-8cb9-7c3c726e75d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f7d1a52-j7nfm" Oct 31 00:36:08.737696 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:08.737568 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fh6xk\" (UniqueName: \"kubernetes.io/projected/3c5fe715-e790-4c03-8cb9-7c3c726e75d0-kube-api-access-fh6xk\") pod \"maas-api-revoke-sub-8f7d1a52-j7nfm\" (UID: \"3c5fe715-e790-4c03-8cb9-7c3c726e75d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f7d1a52-j7nfm" Oct 31 00:36:08.737696 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:08.737628 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c5fe715-e790-4c03-8cb9-7c3c726e75d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f7d1a52-j7nfm\" (UID: \"3c5fe715-e790-4c03-8cb9-7c3c726e75d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f7d1a52-j7nfm" Oct 31 00:36:08.738402 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:08.738315 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c5fe715-e790-4c03-8cb9-7c3c726e75d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f7d1a52-j7nfm\" (UID: \"3c5fe715-e790-4c03-8cb9-7c3c726e75d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f7d1a52-j7nfm" Oct 31 00:36:08.740010 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:08.739989 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c5fe715-e790-4c03-8cb9-7c3c726e75d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f7d1a52-j7nfm\" (UID: \"3c5fe715-e790-4c03-8cb9-7c3c726e75d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f7d1a52-j7nfm" Oct 31 00:36:08.800589 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:08.800551 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fh6xk\" (UniqueName: \"kubernetes.io/projected/3c5fe715-e790-4c03-8cb9-7c3c726e75d0-kube-api-access-fh6xk\") pod \"maas-api-revoke-sub-8f7d1a52-j7nfm\" (UID: \"3c5fe715-e790-4c03-8cb9-7c3c726e75d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f7d1a52-j7nfm" Oct 31 00:36:08.830996 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:08.830962 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f7d1a52-j7nfm" Oct 31 00:36:09.208872 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:36:09.208838 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c5fe715_e790_4c03_8cb9_7c3c726e75d0.slice/crio-8873dc45ba2106700923c1f1a022b358e03e950f950853701ccb8dc35b1d3636 WatchSource:0}: Error finding container 8873dc45ba2106700923c1f1a022b358e03e950f950853701ccb8dc35b1d3636: Status 404 returned error can't find the container with id 8873dc45ba2106700923c1f1a022b358e03e950f950853701ccb8dc35b1d3636 Oct 31 00:36:09.212653 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:09.212631 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f7d1a52-j7nfm"] Oct 31 00:36:09.620103 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:09.620064 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f7d1a52-j7nfm" event={"ID":"3c5fe715-e790-4c03-8cb9-7c3c726e75d0","Type":"ContainerStarted","Data":"d674453d382ca74058341d3556b540aedf9bbc6d8fb279679411bb78b08ad3e0"} Oct 31 00:36:09.620549 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:09.620159 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f7d1a52-j7nfm" event={"ID":"3c5fe715-e790-4c03-8cb9-7c3c726e75d0","Type":"ContainerStarted","Data":"8873dc45ba2106700923c1f1a022b358e03e950f950853701ccb8dc35b1d3636"} Oct 31 00:36:09.739866 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:09.739794 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f7d1a52-j7nfm" podStartSLOduration=1.739776067 podStartE2EDuration="1.739776067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:36:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:36:09.738516308 +0000 UTC m=+1262.209686058" watchObservedRunningTime="2026-10-31 00:36:09.739776067 +0000 UTC m=+1262.210945819" Oct 31 00:36:10.628871 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:10.628839 5222 generic.go:356] "Generic (PLEG): container finished" podID="3c5fe715-e790-4c03-8cb9-7c3c726e75d0" containerID="d674453d382ca74058341d3556b540aedf9bbc6d8fb279679411bb78b08ad3e0" exitCode=0 Oct 31 00:36:10.629309 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:10.628926 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f7d1a52-j7nfm" event={"ID":"3c5fe715-e790-4c03-8cb9-7c3c726e75d0","Type":"ContainerDied","Data":"d674453d382ca74058341d3556b540aedf9bbc6d8fb279679411bb78b08ad3e0"} Oct 31 00:36:11.055157 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:11.055125 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c305a8e6-2v7dx"] Oct 31 00:36:11.058308 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:11.058283 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c305a8e6-2v7dx"] Oct 31 00:36:11.657977 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:11.657945 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b134273-ee2e-4348-8580-f5eba829c608" path="/var/lib/kubelet/pods/2b134273-ee2e-4348-8580-f5eba829c608/volumes" Oct 31 00:36:11.802509 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:11.802481 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f7d1a52-j7nfm" Oct 31 00:36:11.877022 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:11.876989 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c5fe715-e790-4c03-8cb9-7c3c726e75d0-kube-api-access-fh6xk\" (UniqueName: \"kubernetes.io/projected/3c5fe715-e790-4c03-8cb9-7c3c726e75d0-kube-api-access-fh6xk\") pod \"3c5fe715-e790-4c03-8cb9-7c3c726e75d0\" (UID: \"3c5fe715-e790-4c03-8cb9-7c3c726e75d0\") " Oct 31 00:36:11.877202 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:11.877081 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c5fe715-e790-4c03-8cb9-7c3c726e75d0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c5fe715-e790-4c03-8cb9-7c3c726e75d0-maas-api-xks-ca\") pod \"3c5fe715-e790-4c03-8cb9-7c3c726e75d0\" (UID: \"3c5fe715-e790-4c03-8cb9-7c3c726e75d0\") " Oct 31 00:36:11.877202 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:11.877121 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c5fe715-e790-4c03-8cb9-7c3c726e75d0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c5fe715-e790-4c03-8cb9-7c3c726e75d0-maas-api-openshift-ca\") pod \"3c5fe715-e790-4c03-8cb9-7c3c726e75d0\" (UID: \"3c5fe715-e790-4c03-8cb9-7c3c726e75d0\") " Oct 31 00:36:11.877597 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:11.877560 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c5fe715-e790-4c03-8cb9-7c3c726e75d0-maas-api-openshift-ca" pod "3c5fe715-e790-4c03-8cb9-7c3c726e75d0" (UID: "3c5fe715-e790-4c03-8cb9-7c3c726e75d0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:36:11.879238 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:11.879221 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c5fe715-e790-4c03-8cb9-7c3c726e75d0-kube-api-access-fh6xk" pod "3c5fe715-e790-4c03-8cb9-7c3c726e75d0" (UID: "3c5fe715-e790-4c03-8cb9-7c3c726e75d0"). InnerVolumeSpecName "kube-api-access-fh6xk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:36:11.879425 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:11.879403 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c5fe715-e790-4c03-8cb9-7c3c726e75d0-maas-api-xks-ca" pod "3c5fe715-e790-4c03-8cb9-7c3c726e75d0" (UID: "3c5fe715-e790-4c03-8cb9-7c3c726e75d0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:36:11.978270 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:11.978237 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fh6xk\" (UniqueName: \"kubernetes.io/projected/3c5fe715-e790-4c03-8cb9-7c3c726e75d0-kube-api-access-fh6xk\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:36:11.978270 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:11.978270 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c5fe715-e790-4c03-8cb9-7c3c726e75d0-maas-api-xks-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:36:11.978501 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:11.978283 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c5fe715-e790-4c03-8cb9-7c3c726e75d0-maas-api-openshift-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:36:12.261632 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:12.261546 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-56xs9"] Oct 31 00:36:12.262086 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:12.262071 5222 cpu_manager.go:414] "containerMap: removing container" podUID="3c5fe715-e790-4c03-8cb9-7c3c726e75d0" containerName="revoke-keys" Oct 31 00:36:12.262129 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:12.262088 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c5fe715-e790-4c03-8cb9-7c3c726e75d0" containerName="revoke-keys" Oct 31 00:36:12.262202 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:12.262193 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c5fe715-e790-4c03-8cb9-7c3c726e75d0" containerName="revoke-keys" Oct 31 00:36:12.266004 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:12.265984 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-56xs9" Oct 31 00:36:12.375914 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:12.375882 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-56xs9"] Oct 31 00:36:12.384232 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:12.384187 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15886257-5d46-4575-acb2-68a0da38b954-istiod-ca-cert\") pod \"e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-56xs9\" (UID: \"15886257-5d46-4575-acb2-68a0da38b954\") " pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-56xs9" Oct 31 00:36:12.384410 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:12.384267 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15886257-5d46-4575-acb2-68a0da38b954-istio-token\") pod \"e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-56xs9\" (UID: \"15886257-5d46-4575-acb2-68a0da38b954\") " pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-56xs9" Oct 31 00:36:12.384465 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:12.384440 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15886257-5d46-4575-acb2-68a0da38b954-workload-socket\") pod \"e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-56xs9\" (UID: \"15886257-5d46-4575-acb2-68a0da38b954\") " pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-56xs9" Oct 31 00:36:12.384506 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:12.384491 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15886257-5d46-4575-acb2-68a0da38b954-workload-certs\") pod \"e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-56xs9\" (UID: \"15886257-5d46-4575-acb2-68a0da38b954\") " pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-56xs9" Oct 31 00:36:12.384680 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:12.384658 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15886257-5d46-4575-acb2-68a0da38b954-credential-socket\") pod \"e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-56xs9\" (UID: \"15886257-5d46-4575-acb2-68a0da38b954\") " pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-56xs9" Oct 31 00:36:12.384735 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:12.384721 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15886257-5d46-4575-acb2-68a0da38b954-istio-envoy\") pod \"e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-56xs9\" (UID: \"15886257-5d46-4575-acb2-68a0da38b954\") " pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-56xs9" Oct 31 00:36:12.384817 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:12.384803 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zmpp\" (UniqueName: \"kubernetes.io/projected/15886257-5d46-4575-acb2-68a0da38b954-kube-api-access-8zmpp\") pod \"e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-56xs9\" (UID: \"15886257-5d46-4575-acb2-68a0da38b954\") " pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-56xs9" Oct 31 00:36:12.384858 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:12.384849 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15886257-5d46-4575-acb2-68a0da38b954-istio-podinfo\") pod \"e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-56xs9\" (UID: \"15886257-5d46-4575-acb2-68a0da38b954\") " pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-56xs9" Oct 31 00:36:12.384945 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:12.384921 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15886257-5d46-4575-acb2-68a0da38b954-istio-data\") pod \"e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-56xs9\" (UID: \"15886257-5d46-4575-acb2-68a0da38b954\") " pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-56xs9" Oct 31 00:36:12.485607 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:12.485558 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zmpp\" (UniqueName: \"kubernetes.io/projected/15886257-5d46-4575-acb2-68a0da38b954-kube-api-access-8zmpp\") pod \"e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-56xs9\" (UID: \"15886257-5d46-4575-acb2-68a0da38b954\") " pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-56xs9" Oct 31 00:36:12.485607 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:12.485619 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15886257-5d46-4575-acb2-68a0da38b954-istio-podinfo\") pod \"e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-56xs9\" (UID: \"15886257-5d46-4575-acb2-68a0da38b954\") " pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-56xs9" Oct 31 00:36:12.485879 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:12.485649 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15886257-5d46-4575-acb2-68a0da38b954-istio-data\") pod \"e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-56xs9\" (UID: \"15886257-5d46-4575-acb2-68a0da38b954\") " pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-56xs9" Oct 31 00:36:12.485879 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:12.485674 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15886257-5d46-4575-acb2-68a0da38b954-istiod-ca-cert\") pod \"e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-56xs9\" (UID: \"15886257-5d46-4575-acb2-68a0da38b954\") " pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-56xs9" Oct 31 00:36:12.485879 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:12.485708 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15886257-5d46-4575-acb2-68a0da38b954-istio-token\") pod \"e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-56xs9\" (UID: \"15886257-5d46-4575-acb2-68a0da38b954\") " pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-56xs9" Oct 31 00:36:12.485879 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:12.485750 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15886257-5d46-4575-acb2-68a0da38b954-workload-socket\") pod \"e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-56xs9\" (UID: \"15886257-5d46-4575-acb2-68a0da38b954\") " pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-56xs9" Oct 31 00:36:12.485879 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:12.485774 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15886257-5d46-4575-acb2-68a0da38b954-workload-certs\") pod \"e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-56xs9\" (UID: \"15886257-5d46-4575-acb2-68a0da38b954\") " pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-56xs9" Oct 31 00:36:12.485879 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:12.485862 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15886257-5d46-4575-acb2-68a0da38b954-credential-socket\") pod \"e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-56xs9\" (UID: \"15886257-5d46-4575-acb2-68a0da38b954\") " pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-56xs9" Oct 31 00:36:12.486185 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:12.485914 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15886257-5d46-4575-acb2-68a0da38b954-istio-envoy\") pod \"e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-56xs9\" (UID: \"15886257-5d46-4575-acb2-68a0da38b954\") " pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-56xs9" Oct 31 00:36:12.486185 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:12.486138 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15886257-5d46-4575-acb2-68a0da38b954-istio-data\") pod \"e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-56xs9\" (UID: \"15886257-5d46-4575-acb2-68a0da38b954\") " pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-56xs9" Oct 31 00:36:12.486295 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:12.486190 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15886257-5d46-4575-acb2-68a0da38b954-workload-socket\") pod \"e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-56xs9\" (UID: \"15886257-5d46-4575-acb2-68a0da38b954\") " pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-56xs9" Oct 31 00:36:12.486295 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:12.486244 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15886257-5d46-4575-acb2-68a0da38b954-workload-certs\") pod \"e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-56xs9\" (UID: \"15886257-5d46-4575-acb2-68a0da38b954\") " pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-56xs9" Oct 31 00:36:12.486416 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:12.486339 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15886257-5d46-4575-acb2-68a0da38b954-credential-socket\") pod \"e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-56xs9\" (UID: \"15886257-5d46-4575-acb2-68a0da38b954\") " pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-56xs9" Oct 31 00:36:12.486662 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:12.486643 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15886257-5d46-4575-acb2-68a0da38b954-istiod-ca-cert\") pod \"e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-56xs9\" (UID: \"15886257-5d46-4575-acb2-68a0da38b954\") " pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-56xs9" Oct 31 00:36:12.488388 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:12.488353 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15886257-5d46-4575-acb2-68a0da38b954-istio-envoy\") pod \"e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-56xs9\" (UID: \"15886257-5d46-4575-acb2-68a0da38b954\") " pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-56xs9" Oct 31 00:36:12.488498 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:12.488411 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15886257-5d46-4575-acb2-68a0da38b954-istio-podinfo\") pod \"e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-56xs9\" (UID: \"15886257-5d46-4575-acb2-68a0da38b954\") " pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-56xs9" Oct 31 00:36:12.548943 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:12.548906 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15886257-5d46-4575-acb2-68a0da38b954-istio-token\") pod \"e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-56xs9\" (UID: \"15886257-5d46-4575-acb2-68a0da38b954\") " pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-56xs9" Oct 31 00:36:12.550543 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:12.550509 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zmpp\" (UniqueName: \"kubernetes.io/projected/15886257-5d46-4575-acb2-68a0da38b954-kube-api-access-8zmpp\") pod \"e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-56xs9\" (UID: \"15886257-5d46-4575-acb2-68a0da38b954\") " pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-56xs9" Oct 31 00:36:12.580760 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:12.580722 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-56xs9" Oct 31 00:36:12.664790 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:12.664762 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f7d1a52-j7nfm" Oct 31 00:36:12.665290 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:12.664896 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f7d1a52-j7nfm" event={"ID":"3c5fe715-e790-4c03-8cb9-7c3c726e75d0","Type":"ContainerDied","Data":"8873dc45ba2106700923c1f1a022b358e03e950f950853701ccb8dc35b1d3636"} Oct 31 00:36:12.665434 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:12.665318 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8873dc45ba2106700923c1f1a022b358e03e950f950853701ccb8dc35b1d3636" Oct 31 00:36:13.018280 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:36:13.018252 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15886257_5d46_4575_acb2_68a0da38b954.slice/crio-a0a5bb7813d105fda295597b13b79440180adf34a3b290955955151fae6396a7 WatchSource:0}: Error finding container a0a5bb7813d105fda295597b13b79440180adf34a3b290955955151fae6396a7: Status 404 returned error can't find the container with id a0a5bb7813d105fda295597b13b79440180adf34a3b290955955151fae6396a7 Oct 31 00:36:13.020915 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:13.020873 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:36:13.021042 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:13.020969 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:36:13.021042 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:13.021016 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:36:13.023685 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:13.023662 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-56xs9"] Oct 31 00:36:13.674032 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:13.673994 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-56xs9" event={"ID":"15886257-5d46-4575-acb2-68a0da38b954","Type":"ContainerStarted","Data":"4a0a3ba14c7811082b0ca15f89859b3b5e0b541589b793cbd5a1dc8f628e61e6"} Oct 31 00:36:13.674458 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:13.674049 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-56xs9" event={"ID":"15886257-5d46-4575-acb2-68a0da38b954","Type":"ContainerStarted","Data":"a0a5bb7813d105fda295597b13b79440180adf34a3b290955955151fae6396a7"} Oct 31 00:36:13.801540 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:13.801482 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-56xs9" podStartSLOduration=1.801464289 podStartE2EDuration="1.801464289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:36:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:36:13.800813224 +0000 UTC m=+1266.271982973" watchObservedRunningTime="2026-10-31 00:36:13.801464289 +0000 UTC m=+1266.272634038" Oct 31 00:36:14.580989 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:14.580896 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-56xs9" Oct 31 00:36:14.582783 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:14.582755 5222 patch_prober.go:29] interesting pod/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-56xs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 31 00:36:14.582920 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:14.582802 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-56xs9" podUID="15886257-5d46-4575-acb2-68a0da38b954" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 31 00:36:15.581284 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:15.581251 5222 patch_prober.go:29] interesting pod/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-56xs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 31 00:36:15.581734 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:15.581305 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-56xs9" podUID="15886257-5d46-4575-acb2-68a0da38b954" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 31 00:36:16.582066 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:16.582030 5222 patch_prober.go:29] interesting pod/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-56xs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 31 00:36:16.582481 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:16.582081 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-56xs9" podUID="15886257-5d46-4575-acb2-68a0da38b954" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 31 00:36:17.581976 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:17.581940 5222 patch_prober.go:29] interesting pod/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-56xs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 31 00:36:17.582200 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:17.582002 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-56xs9" podUID="15886257-5d46-4575-acb2-68a0da38b954" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 31 00:36:18.581095 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:18.581057 5222 patch_prober.go:29] interesting pod/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-56xs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 31 00:36:18.581306 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:18.581122 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-56xs9" podUID="15886257-5d46-4575-acb2-68a0da38b954" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 31 00:36:19.581772 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:19.581739 5222 patch_prober.go:29] interesting pod/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-56xs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 31 00:36:19.582199 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:19.581797 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-56xs9" podUID="15886257-5d46-4575-acb2-68a0da38b954" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 31 00:36:21.040750 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:21.040711 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-56xs9" Oct 31 00:36:21.051098 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:21.044104 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-56xs9" Oct 31 00:36:21.051400 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:21.051353 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-56xs9" Oct 31 00:36:22.052534 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:22.052498 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-729c92a6-2vqlw"] Oct 31 00:36:22.103328 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:22.103292 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-729c92a6-2vqlw"] Oct 31 00:36:23.656715 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:23.656678 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="040ddba8-7817-4ff7-b301-229cc557f8d5" path="/var/lib/kubelet/pods/040ddba8-7817-4ff7-b301-229cc557f8d5/volumes" Oct 31 00:36:24.067739 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:24.067694 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66356b86-fbsl2"] Oct 31 00:36:24.071926 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:24.071902 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66356b86-fbsl2" Oct 31 00:36:24.126582 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:24.126552 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66356b86-fbsl2"] Oct 31 00:36:24.146774 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:24.146734 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/472f67dd-2129-4a33-8054-4d3bdb164f9c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-66356b86-fbsl2\" (UID: \"472f67dd-2129-4a33-8054-4d3bdb164f9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66356b86-fbsl2" Oct 31 00:36:24.146947 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:24.146790 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/472f67dd-2129-4a33-8054-4d3bdb164f9c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-66356b86-fbsl2\" (UID: \"472f67dd-2129-4a33-8054-4d3bdb164f9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66356b86-fbsl2" Oct 31 00:36:24.146947 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:24.146871 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6r67\" (UniqueName: \"kubernetes.io/projected/472f67dd-2129-4a33-8054-4d3bdb164f9c-kube-api-access-c6r67\") pod \"maas-api-revoke-sub-66356b86-fbsl2\" (UID: \"472f67dd-2129-4a33-8054-4d3bdb164f9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66356b86-fbsl2" Oct 31 00:36:24.172762 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:24.172731 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:36:24.172762 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:24.172761 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:36:24.172981 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:24.172761 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:36:24.176919 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:24.176899 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfz9z\"" Oct 31 00:36:24.248005 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:24.247972 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/472f67dd-2129-4a33-8054-4d3bdb164f9c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-66356b86-fbsl2\" (UID: \"472f67dd-2129-4a33-8054-4d3bdb164f9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66356b86-fbsl2" Oct 31 00:36:24.248182 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:24.248014 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/472f67dd-2129-4a33-8054-4d3bdb164f9c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-66356b86-fbsl2\" (UID: \"472f67dd-2129-4a33-8054-4d3bdb164f9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66356b86-fbsl2" Oct 31 00:36:24.248182 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:24.248035 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6r67\" (UniqueName: \"kubernetes.io/projected/472f67dd-2129-4a33-8054-4d3bdb164f9c-kube-api-access-c6r67\") pod \"maas-api-revoke-sub-66356b86-fbsl2\" (UID: \"472f67dd-2129-4a33-8054-4d3bdb164f9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66356b86-fbsl2" Oct 31 00:36:24.248784 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:24.248764 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/472f67dd-2129-4a33-8054-4d3bdb164f9c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-66356b86-fbsl2\" (UID: \"472f67dd-2129-4a33-8054-4d3bdb164f9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66356b86-fbsl2" Oct 31 00:36:24.250602 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:24.250569 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/472f67dd-2129-4a33-8054-4d3bdb164f9c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-66356b86-fbsl2\" (UID: \"472f67dd-2129-4a33-8054-4d3bdb164f9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66356b86-fbsl2" Oct 31 00:36:24.322138 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:24.322066 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6r67\" (UniqueName: \"kubernetes.io/projected/472f67dd-2129-4a33-8054-4d3bdb164f9c-kube-api-access-c6r67\") pod \"maas-api-revoke-sub-66356b86-fbsl2\" (UID: \"472f67dd-2129-4a33-8054-4d3bdb164f9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66356b86-fbsl2" Oct 31 00:36:24.385558 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:24.385525 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66356b86-fbsl2" Oct 31 00:36:24.587415 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:24.587303 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ea3a02b-6lg28"] Oct 31 00:36:24.592531 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:24.592511 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ea3a02b-6lg28" Oct 31 00:36:24.644627 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:24.644583 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ea3a02b-6lg28"] Oct 31 00:36:24.652624 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:24.652585 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhrzx\" (UniqueName: \"kubernetes.io/projected/66c3d766-2417-4ab1-87ae-4d678dfa3f7f-kube-api-access-jhrzx\") pod \"maas-api-revoke-sub-5ea3a02b-6lg28\" (UID: \"66c3d766-2417-4ab1-87ae-4d678dfa3f7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ea3a02b-6lg28" Oct 31 00:36:24.652832 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:24.652787 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66c3d766-2417-4ab1-87ae-4d678dfa3f7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ea3a02b-6lg28\" (UID: \"66c3d766-2417-4ab1-87ae-4d678dfa3f7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ea3a02b-6lg28" Oct 31 00:36:24.652832 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:24.652822 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66c3d766-2417-4ab1-87ae-4d678dfa3f7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ea3a02b-6lg28\" (UID: \"66c3d766-2417-4ab1-87ae-4d678dfa3f7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ea3a02b-6lg28" Oct 31 00:36:24.754356 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:24.754320 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66c3d766-2417-4ab1-87ae-4d678dfa3f7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ea3a02b-6lg28\" (UID: \"66c3d766-2417-4ab1-87ae-4d678dfa3f7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ea3a02b-6lg28" Oct 31 00:36:24.754951 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:24.754928 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66c3d766-2417-4ab1-87ae-4d678dfa3f7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ea3a02b-6lg28\" (UID: \"66c3d766-2417-4ab1-87ae-4d678dfa3f7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ea3a02b-6lg28" Oct 31 00:36:24.755095 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:24.755081 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhrzx\" (UniqueName: \"kubernetes.io/projected/66c3d766-2417-4ab1-87ae-4d678dfa3f7f-kube-api-access-jhrzx\") pod \"maas-api-revoke-sub-5ea3a02b-6lg28\" (UID: \"66c3d766-2417-4ab1-87ae-4d678dfa3f7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ea3a02b-6lg28" Oct 31 00:36:24.755238 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:24.755214 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66c3d766-2417-4ab1-87ae-4d678dfa3f7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ea3a02b-6lg28\" (UID: \"66c3d766-2417-4ab1-87ae-4d678dfa3f7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ea3a02b-6lg28" Oct 31 00:36:24.758615 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:24.758584 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66c3d766-2417-4ab1-87ae-4d678dfa3f7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ea3a02b-6lg28\" (UID: \"66c3d766-2417-4ab1-87ae-4d678dfa3f7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ea3a02b-6lg28" Oct 31 00:36:24.776626 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:24.776595 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66356b86-fbsl2"] Oct 31 00:36:24.817722 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:24.817689 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhrzx\" (UniqueName: \"kubernetes.io/projected/66c3d766-2417-4ab1-87ae-4d678dfa3f7f-kube-api-access-jhrzx\") pod \"maas-api-revoke-sub-5ea3a02b-6lg28\" (UID: \"66c3d766-2417-4ab1-87ae-4d678dfa3f7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ea3a02b-6lg28" Oct 31 00:36:24.907421 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:24.907306 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ea3a02b-6lg28" Oct 31 00:36:25.291212 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:36:25.291172 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66c3d766_2417_4ab1_87ae_4d678dfa3f7f.slice/crio-adc76db148d5f04c25255557f4aeb51fe4050ed3f7b78c7dca65723fd139a298 WatchSource:0}: Error finding container adc76db148d5f04c25255557f4aeb51fe4050ed3f7b78c7dca65723fd139a298: Status 404 returned error can't find the container with id adc76db148d5f04c25255557f4aeb51fe4050ed3f7b78c7dca65723fd139a298 Oct 31 00:36:25.295878 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:25.295849 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ea3a02b-6lg28"] Oct 31 00:36:25.779638 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:25.779601 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ea3a02b-6lg28" event={"ID":"66c3d766-2417-4ab1-87ae-4d678dfa3f7f","Type":"ContainerStarted","Data":"17e63d20b9c2f609e8c3b902a01e95ea6cae7da72d6d45a6b01b086df1637c8e"} Oct 31 00:36:25.780115 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:25.779660 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ea3a02b-6lg28" event={"ID":"66c3d766-2417-4ab1-87ae-4d678dfa3f7f","Type":"ContainerStarted","Data":"adc76db148d5f04c25255557f4aeb51fe4050ed3f7b78c7dca65723fd139a298"} Oct 31 00:36:25.781208 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:25.781184 5222 generic.go:356] "Generic (PLEG): container finished" podID="472f67dd-2129-4a33-8054-4d3bdb164f9c" containerID="2ccfd9e4c81fd99b874ae570a375da55ff324da7feab4cf8c32f9876729e91a8" exitCode=0 Oct 31 00:36:25.781316 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:25.781241 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66356b86-fbsl2" event={"ID":"472f67dd-2129-4a33-8054-4d3bdb164f9c","Type":"ContainerDied","Data":"2ccfd9e4c81fd99b874ae570a375da55ff324da7feab4cf8c32f9876729e91a8"} Oct 31 00:36:25.781316 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:25.781282 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66356b86-fbsl2" event={"ID":"472f67dd-2129-4a33-8054-4d3bdb164f9c","Type":"ContainerStarted","Data":"33f8ca156ae0ee7cb9a26b217169792b665afcd5b4799d6e876bf43abd8f4f6a"} Oct 31 00:36:25.906424 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:25.906281 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ea3a02b-6lg28" podStartSLOduration=1.9062669269999999 podStartE2EDuration="1.906266927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:36:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:36:25.904021051 +0000 UTC m=+1278.375190799" watchObservedRunningTime="2026-10-31 00:36:25.906266927 +0000 UTC m=+1278.377436676" Oct 31 00:36:26.790207 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:26.790173 5222 generic.go:356] "Generic (PLEG): container finished" podID="66c3d766-2417-4ab1-87ae-4d678dfa3f7f" containerID="17e63d20b9c2f609e8c3b902a01e95ea6cae7da72d6d45a6b01b086df1637c8e" exitCode=0 Oct 31 00:36:26.790685 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:26.790259 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ea3a02b-6lg28" event={"ID":"66c3d766-2417-4ab1-87ae-4d678dfa3f7f","Type":"ContainerDied","Data":"17e63d20b9c2f609e8c3b902a01e95ea6cae7da72d6d45a6b01b086df1637c8e"} Oct 31 00:36:26.953892 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:26.953860 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66356b86-fbsl2" Oct 31 00:36:26.982655 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:26.982625 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/472f67dd-2129-4a33-8054-4d3bdb164f9c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/472f67dd-2129-4a33-8054-4d3bdb164f9c-maas-api-openshift-ca\") pod \"472f67dd-2129-4a33-8054-4d3bdb164f9c\" (UID: \"472f67dd-2129-4a33-8054-4d3bdb164f9c\") " Oct 31 00:36:26.982830 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:26.982729 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/472f67dd-2129-4a33-8054-4d3bdb164f9c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/472f67dd-2129-4a33-8054-4d3bdb164f9c-maas-api-xks-ca\") pod \"472f67dd-2129-4a33-8054-4d3bdb164f9c\" (UID: \"472f67dd-2129-4a33-8054-4d3bdb164f9c\") " Oct 31 00:36:26.982830 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:26.982766 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/472f67dd-2129-4a33-8054-4d3bdb164f9c-kube-api-access-c6r67\" (UniqueName: \"kubernetes.io/projected/472f67dd-2129-4a33-8054-4d3bdb164f9c-kube-api-access-c6r67\") pod \"472f67dd-2129-4a33-8054-4d3bdb164f9c\" (UID: \"472f67dd-2129-4a33-8054-4d3bdb164f9c\") " Oct 31 00:36:26.983070 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:26.983043 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/472f67dd-2129-4a33-8054-4d3bdb164f9c-maas-api-openshift-ca" pod "472f67dd-2129-4a33-8054-4d3bdb164f9c" (UID: "472f67dd-2129-4a33-8054-4d3bdb164f9c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:36:26.983249 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:26.983229 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/472f67dd-2129-4a33-8054-4d3bdb164f9c-maas-api-openshift-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:36:26.985019 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:26.984990 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/472f67dd-2129-4a33-8054-4d3bdb164f9c-maas-api-xks-ca" pod "472f67dd-2129-4a33-8054-4d3bdb164f9c" (UID: "472f67dd-2129-4a33-8054-4d3bdb164f9c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:36:26.985125 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:26.985050 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/472f67dd-2129-4a33-8054-4d3bdb164f9c-kube-api-access-c6r67" pod "472f67dd-2129-4a33-8054-4d3bdb164f9c" (UID: "472f67dd-2129-4a33-8054-4d3bdb164f9c"). InnerVolumeSpecName "kube-api-access-c6r67". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:36:27.068114 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:27.068021 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25c7e9be-89hxv"] Oct 31 00:36:27.073794 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:27.073765 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25c7e9be-89hxv"] Oct 31 00:36:27.084204 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:27.084172 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/472f67dd-2129-4a33-8054-4d3bdb164f9c-maas-api-xks-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:36:27.084204 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:27.084203 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6r67\" (UniqueName: \"kubernetes.io/projected/472f67dd-2129-4a33-8054-4d3bdb164f9c-kube-api-access-c6r67\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:36:27.640244 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:27.640206 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-65wmw"] Oct 31 00:36:27.640900 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:27.640869 5222 cpu_manager.go:414] "containerMap: removing container" podUID="472f67dd-2129-4a33-8054-4d3bdb164f9c" containerName="revoke-keys" Oct 31 00:36:27.640900 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:27.640898 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="472f67dd-2129-4a33-8054-4d3bdb164f9c" containerName="revoke-keys" Oct 31 00:36:27.641089 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:27.641072 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="472f67dd-2129-4a33-8054-4d3bdb164f9c" containerName="revoke-keys" Oct 31 00:36:27.645403 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:27.645339 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-65wmw" Oct 31 00:36:27.658179 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:27.658151 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06d94ff5-8b6d-44c5-a617-1fc79d585fb8" path="/var/lib/kubelet/pods/06d94ff5-8b6d-44c5-a617-1fc79d585fb8/volumes" Oct 31 00:36:27.692008 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:27.691976 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0faaa14f-bbf7-4915-ba75-4248ae943904-workload-socket\") pod \"e2e-worker-w3-1e5e75-openshift-default-879c67bc5-65wmw\" (UID: \"0faaa14f-bbf7-4915-ba75-4248ae943904\") " pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-65wmw" Oct 31 00:36:27.692247 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:27.692214 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0faaa14f-bbf7-4915-ba75-4248ae943904-workload-certs\") pod \"e2e-worker-w3-1e5e75-openshift-default-879c67bc5-65wmw\" (UID: \"0faaa14f-bbf7-4915-ba75-4248ae943904\") " pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-65wmw" Oct 31 00:36:27.692465 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:27.692263 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0faaa14f-bbf7-4915-ba75-4248ae943904-istio-token\") pod \"e2e-worker-w3-1e5e75-openshift-default-879c67bc5-65wmw\" (UID: \"0faaa14f-bbf7-4915-ba75-4248ae943904\") " pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-65wmw" Oct 31 00:36:27.692465 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:27.692282 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0faaa14f-bbf7-4915-ba75-4248ae943904-istio-data\") pod \"e2e-worker-w3-1e5e75-openshift-default-879c67bc5-65wmw\" (UID: \"0faaa14f-bbf7-4915-ba75-4248ae943904\") " pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-65wmw" Oct 31 00:36:27.692465 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:27.692322 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0faaa14f-bbf7-4915-ba75-4248ae943904-istio-podinfo\") pod \"e2e-worker-w3-1e5e75-openshift-default-879c67bc5-65wmw\" (UID: \"0faaa14f-bbf7-4915-ba75-4248ae943904\") " pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-65wmw" Oct 31 00:36:27.692624 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:27.692593 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0faaa14f-bbf7-4915-ba75-4248ae943904-istio-envoy\") pod \"e2e-worker-w3-1e5e75-openshift-default-879c67bc5-65wmw\" (UID: \"0faaa14f-bbf7-4915-ba75-4248ae943904\") " pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-65wmw" Oct 31 00:36:27.692681 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:27.692643 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sf9rl\" (UniqueName: \"kubernetes.io/projected/0faaa14f-bbf7-4915-ba75-4248ae943904-kube-api-access-sf9rl\") pod \"e2e-worker-w3-1e5e75-openshift-default-879c67bc5-65wmw\" (UID: \"0faaa14f-bbf7-4915-ba75-4248ae943904\") " pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-65wmw" Oct 31 00:36:27.692733 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:27.692694 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0faaa14f-bbf7-4915-ba75-4248ae943904-credential-socket\") pod \"e2e-worker-w3-1e5e75-openshift-default-879c67bc5-65wmw\" (UID: \"0faaa14f-bbf7-4915-ba75-4248ae943904\") " pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-65wmw" Oct 31 00:36:27.692733 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:27.692719 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0faaa14f-bbf7-4915-ba75-4248ae943904-istiod-ca-cert\") pod \"e2e-worker-w3-1e5e75-openshift-default-879c67bc5-65wmw\" (UID: \"0faaa14f-bbf7-4915-ba75-4248ae943904\") " pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-65wmw" Oct 31 00:36:27.701133 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:27.701103 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-1e5e75-openshift-default-dockercfg-gw4mk\"" Oct 31 00:36:27.703721 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:27.703691 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-65wmw"] Oct 31 00:36:27.793561 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:27.793529 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0faaa14f-bbf7-4915-ba75-4248ae943904-workload-certs\") pod \"e2e-worker-w3-1e5e75-openshift-default-879c67bc5-65wmw\" (UID: \"0faaa14f-bbf7-4915-ba75-4248ae943904\") " pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-65wmw" Oct 31 00:36:27.793561 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:27.793570 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0faaa14f-bbf7-4915-ba75-4248ae943904-istio-token\") pod \"e2e-worker-w3-1e5e75-openshift-default-879c67bc5-65wmw\" (UID: \"0faaa14f-bbf7-4915-ba75-4248ae943904\") " pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-65wmw" Oct 31 00:36:27.794132 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:27.793591 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0faaa14f-bbf7-4915-ba75-4248ae943904-istio-data\") pod \"e2e-worker-w3-1e5e75-openshift-default-879c67bc5-65wmw\" (UID: \"0faaa14f-bbf7-4915-ba75-4248ae943904\") " pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-65wmw" Oct 31 00:36:27.794132 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:27.793631 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0faaa14f-bbf7-4915-ba75-4248ae943904-istio-podinfo\") pod \"e2e-worker-w3-1e5e75-openshift-default-879c67bc5-65wmw\" (UID: \"0faaa14f-bbf7-4915-ba75-4248ae943904\") " pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-65wmw" Oct 31 00:36:27.794132 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:27.793800 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0faaa14f-bbf7-4915-ba75-4248ae943904-istio-envoy\") pod \"e2e-worker-w3-1e5e75-openshift-default-879c67bc5-65wmw\" (UID: \"0faaa14f-bbf7-4915-ba75-4248ae943904\") " pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-65wmw" Oct 31 00:36:27.794132 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:27.793872 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sf9rl\" (UniqueName: \"kubernetes.io/projected/0faaa14f-bbf7-4915-ba75-4248ae943904-kube-api-access-sf9rl\") pod \"e2e-worker-w3-1e5e75-openshift-default-879c67bc5-65wmw\" (UID: \"0faaa14f-bbf7-4915-ba75-4248ae943904\") " pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-65wmw" Oct 31 00:36:27.794132 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:27.793943 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0faaa14f-bbf7-4915-ba75-4248ae943904-credential-socket\") pod \"e2e-worker-w3-1e5e75-openshift-default-879c67bc5-65wmw\" (UID: \"0faaa14f-bbf7-4915-ba75-4248ae943904\") " pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-65wmw" Oct 31 00:36:27.794132 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:27.793980 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0faaa14f-bbf7-4915-ba75-4248ae943904-istiod-ca-cert\") pod \"e2e-worker-w3-1e5e75-openshift-default-879c67bc5-65wmw\" (UID: \"0faaa14f-bbf7-4915-ba75-4248ae943904\") " pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-65wmw" Oct 31 00:36:27.794132 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:27.793991 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0faaa14f-bbf7-4915-ba75-4248ae943904-workload-certs\") pod \"e2e-worker-w3-1e5e75-openshift-default-879c67bc5-65wmw\" (UID: \"0faaa14f-bbf7-4915-ba75-4248ae943904\") " pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-65wmw" Oct 31 00:36:27.794132 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:27.794067 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0faaa14f-bbf7-4915-ba75-4248ae943904-istio-data\") pod \"e2e-worker-w3-1e5e75-openshift-default-879c67bc5-65wmw\" (UID: \"0faaa14f-bbf7-4915-ba75-4248ae943904\") " pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-65wmw" Oct 31 00:36:27.794132 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:27.794087 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0faaa14f-bbf7-4915-ba75-4248ae943904-workload-socket\") pod \"e2e-worker-w3-1e5e75-openshift-default-879c67bc5-65wmw\" (UID: \"0faaa14f-bbf7-4915-ba75-4248ae943904\") " pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-65wmw" Oct 31 00:36:27.794613 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:27.794272 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0faaa14f-bbf7-4915-ba75-4248ae943904-credential-socket\") pod \"e2e-worker-w3-1e5e75-openshift-default-879c67bc5-65wmw\" (UID: \"0faaa14f-bbf7-4915-ba75-4248ae943904\") " pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-65wmw" Oct 31 00:36:27.794613 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:27.794498 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0faaa14f-bbf7-4915-ba75-4248ae943904-workload-socket\") pod \"e2e-worker-w3-1e5e75-openshift-default-879c67bc5-65wmw\" (UID: \"0faaa14f-bbf7-4915-ba75-4248ae943904\") " pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-65wmw" Oct 31 00:36:27.794719 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:27.794694 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0faaa14f-bbf7-4915-ba75-4248ae943904-istiod-ca-cert\") pod \"e2e-worker-w3-1e5e75-openshift-default-879c67bc5-65wmw\" (UID: \"0faaa14f-bbf7-4915-ba75-4248ae943904\") " pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-65wmw" Oct 31 00:36:27.796908 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:27.796879 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0faaa14f-bbf7-4915-ba75-4248ae943904-istio-envoy\") pod \"e2e-worker-w3-1e5e75-openshift-default-879c67bc5-65wmw\" (UID: \"0faaa14f-bbf7-4915-ba75-4248ae943904\") " pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-65wmw" Oct 31 00:36:27.797095 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:27.797079 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0faaa14f-bbf7-4915-ba75-4248ae943904-istio-podinfo\") pod \"e2e-worker-w3-1e5e75-openshift-default-879c67bc5-65wmw\" (UID: \"0faaa14f-bbf7-4915-ba75-4248ae943904\") " pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-65wmw" Oct 31 00:36:27.800533 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:27.800507 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66356b86-fbsl2" Oct 31 00:36:27.800632 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:27.800552 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66356b86-fbsl2" event={"ID":"472f67dd-2129-4a33-8054-4d3bdb164f9c","Type":"ContainerDied","Data":"33f8ca156ae0ee7cb9a26b217169792b665afcd5b4799d6e876bf43abd8f4f6a"} Oct 31 00:36:27.800668 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:27.800642 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="33f8ca156ae0ee7cb9a26b217169792b665afcd5b4799d6e876bf43abd8f4f6a" Oct 31 00:36:27.856881 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:27.856846 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0faaa14f-bbf7-4915-ba75-4248ae943904-istio-token\") pod \"e2e-worker-w3-1e5e75-openshift-default-879c67bc5-65wmw\" (UID: \"0faaa14f-bbf7-4915-ba75-4248ae943904\") " pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-65wmw" Oct 31 00:36:27.857156 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:27.857137 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sf9rl\" (UniqueName: \"kubernetes.io/projected/0faaa14f-bbf7-4915-ba75-4248ae943904-kube-api-access-sf9rl\") pod \"e2e-worker-w3-1e5e75-openshift-default-879c67bc5-65wmw\" (UID: \"0faaa14f-bbf7-4915-ba75-4248ae943904\") " pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-65wmw" Oct 31 00:36:27.960754 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:27.960728 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-65wmw" Oct 31 00:36:27.962877 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:27.962843 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ea3a02b-6lg28" Oct 31 00:36:27.996522 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:27.996491 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66c3d766-2417-4ab1-87ae-4d678dfa3f7f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66c3d766-2417-4ab1-87ae-4d678dfa3f7f-maas-api-xks-ca\") pod \"66c3d766-2417-4ab1-87ae-4d678dfa3f7f\" (UID: \"66c3d766-2417-4ab1-87ae-4d678dfa3f7f\") " Oct 31 00:36:27.996697 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:27.996615 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66c3d766-2417-4ab1-87ae-4d678dfa3f7f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66c3d766-2417-4ab1-87ae-4d678dfa3f7f-maas-api-openshift-ca\") pod \"66c3d766-2417-4ab1-87ae-4d678dfa3f7f\" (UID: \"66c3d766-2417-4ab1-87ae-4d678dfa3f7f\") " Oct 31 00:36:27.996769 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:27.996710 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66c3d766-2417-4ab1-87ae-4d678dfa3f7f-kube-api-access-jhrzx\" (UniqueName: \"kubernetes.io/projected/66c3d766-2417-4ab1-87ae-4d678dfa3f7f-kube-api-access-jhrzx\") pod \"66c3d766-2417-4ab1-87ae-4d678dfa3f7f\" (UID: \"66c3d766-2417-4ab1-87ae-4d678dfa3f7f\") " Oct 31 00:36:27.997064 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:27.997015 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66c3d766-2417-4ab1-87ae-4d678dfa3f7f-maas-api-openshift-ca" pod "66c3d766-2417-4ab1-87ae-4d678dfa3f7f" (UID: "66c3d766-2417-4ab1-87ae-4d678dfa3f7f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:36:27.997242 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:27.997227 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66c3d766-2417-4ab1-87ae-4d678dfa3f7f-maas-api-openshift-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:36:27.998746 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:27.998719 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66c3d766-2417-4ab1-87ae-4d678dfa3f7f-maas-api-xks-ca" pod "66c3d766-2417-4ab1-87ae-4d678dfa3f7f" (UID: "66c3d766-2417-4ab1-87ae-4d678dfa3f7f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:36:27.999072 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:27.999041 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66c3d766-2417-4ab1-87ae-4d678dfa3f7f-kube-api-access-jhrzx" pod "66c3d766-2417-4ab1-87ae-4d678dfa3f7f" (UID: "66c3d766-2417-4ab1-87ae-4d678dfa3f7f"). InnerVolumeSpecName "kube-api-access-jhrzx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:36:28.098878 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:28.098845 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66c3d766-2417-4ab1-87ae-4d678dfa3f7f-maas-api-xks-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:36:28.098878 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:28.098879 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhrzx\" (UniqueName: \"kubernetes.io/projected/66c3d766-2417-4ab1-87ae-4d678dfa3f7f-kube-api-access-jhrzx\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:36:28.402428 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:28.402346 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-65wmw"] Oct 31 00:36:28.402656 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:28.402628 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:36:28.402703 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:28.402687 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:36:28.402737 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:28.402723 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:36:28.811237 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:28.811203 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-65wmw" event={"ID":"0faaa14f-bbf7-4915-ba75-4248ae943904","Type":"ContainerStarted","Data":"70cecf3323b0806d1e90ffdd4ed2c7290a2acf0caf9d10518818d4e10ab710b6"} Oct 31 00:36:28.811724 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:28.811266 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-65wmw" event={"ID":"0faaa14f-bbf7-4915-ba75-4248ae943904","Type":"ContainerStarted","Data":"aba18a424d06984a70a5c5d85585af7582d3e7d9b4c247626a3f0bd9215009f9"} Oct 31 00:36:28.813207 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:28.813187 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ea3a02b-6lg28" Oct 31 00:36:28.813342 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:28.813230 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ea3a02b-6lg28" event={"ID":"66c3d766-2417-4ab1-87ae-4d678dfa3f7f","Type":"ContainerDied","Data":"adc76db148d5f04c25255557f4aeb51fe4050ed3f7b78c7dca65723fd139a298"} Oct 31 00:36:28.813342 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:28.813293 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="adc76db148d5f04c25255557f4aeb51fe4050ed3f7b78c7dca65723fd139a298" Oct 31 00:36:28.939724 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:28.939663 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-65wmw" podStartSLOduration=1.939643588 podStartE2EDuration="1.939643588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:36:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:36:28.937957623 +0000 UTC m=+1281.409127397" watchObservedRunningTime="2026-10-31 00:36:28.939643588 +0000 UTC m=+1281.410813337" Oct 31 00:36:28.966600 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:28.966562 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-65wmw" Oct 31 00:36:29.963950 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:29.963904 5222 patch_prober.go:29] interesting pod/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-65wmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 31 00:36:29.964495 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:29.963964 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-65wmw" podUID="0faaa14f-bbf7-4915-ba75-4248ae943904" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 31 00:36:30.962901 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:30.962864 5222 patch_prober.go:29] interesting pod/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-65wmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 31 00:36:30.963113 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:30.962926 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-65wmw" podUID="0faaa14f-bbf7-4915-ba75-4248ae943904" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 31 00:36:31.962882 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:31.962838 5222 patch_prober.go:29] interesting pod/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-65wmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 31 00:36:31.963358 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:31.962903 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-65wmw" podUID="0faaa14f-bbf7-4915-ba75-4248ae943904" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 31 00:36:32.961787 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:32.961750 5222 patch_prober.go:29] interesting pod/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-65wmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 31 00:36:32.962074 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:32.962043 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-65wmw" podUID="0faaa14f-bbf7-4915-ba75-4248ae943904" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 31 00:36:33.962886 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:33.962851 5222 patch_prober.go:29] interesting pod/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-65wmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 31 00:36:33.963330 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:33.962915 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-65wmw" podUID="0faaa14f-bbf7-4915-ba75-4248ae943904" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 31 00:36:34.961066 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:34.961028 5222 patch_prober.go:29] interesting pod/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-65wmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 31 00:36:34.961262 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:34.961095 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-65wmw" podUID="0faaa14f-bbf7-4915-ba75-4248ae943904" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 31 00:36:35.894445 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:36:35.894404 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 00:36:35.965782 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:35.965751 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-65wmw" Oct 31 00:36:35.966172 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:35.966152 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-65wmw" Oct 31 00:36:35.966969 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:35.966949 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-65wmw" Oct 31 00:36:40.439455 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:40.439427 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-730b8259-d7mpx"] Oct 31 00:36:40.440039 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:40.440022 5222 cpu_manager.go:414] "containerMap: removing container" podUID="66c3d766-2417-4ab1-87ae-4d678dfa3f7f" containerName="revoke-keys" Oct 31 00:36:40.440039 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:40.440037 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66c3d766-2417-4ab1-87ae-4d678dfa3f7f" containerName="revoke-keys" Oct 31 00:36:40.440166 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:40.440154 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="66c3d766-2417-4ab1-87ae-4d678dfa3f7f" containerName="revoke-keys" Oct 31 00:36:40.444034 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:40.444014 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-730b8259-d7mpx" Oct 31 00:36:40.499066 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:40.499031 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-730b8259-d7mpx"] Oct 31 00:36:40.544147 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:40.544117 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:36:40.544329 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:40.544166 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfz9z\"" Oct 31 00:36:40.544329 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:40.544166 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:36:40.548814 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:40.548797 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:36:40.587862 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:40.587820 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qwtn\" (UniqueName: \"kubernetes.io/projected/5c366443-f988-4827-9d18-b985810050da-kube-api-access-2qwtn\") pod \"maas-api-revoke-sub-730b8259-d7mpx\" (UID: \"5c366443-f988-4827-9d18-b985810050da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-730b8259-d7mpx" Oct 31 00:36:40.588044 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:40.588002 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c366443-f988-4827-9d18-b985810050da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-730b8259-d7mpx\" (UID: \"5c366443-f988-4827-9d18-b985810050da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-730b8259-d7mpx" Oct 31 00:36:40.588120 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:40.588103 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c366443-f988-4827-9d18-b985810050da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-730b8259-d7mpx\" (UID: \"5c366443-f988-4827-9d18-b985810050da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-730b8259-d7mpx" Oct 31 00:36:40.688977 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:40.688942 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c366443-f988-4827-9d18-b985810050da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-730b8259-d7mpx\" (UID: \"5c366443-f988-4827-9d18-b985810050da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-730b8259-d7mpx" Oct 31 00:36:40.689151 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:40.689064 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qwtn\" (UniqueName: \"kubernetes.io/projected/5c366443-f988-4827-9d18-b985810050da-kube-api-access-2qwtn\") pod \"maas-api-revoke-sub-730b8259-d7mpx\" (UID: \"5c366443-f988-4827-9d18-b985810050da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-730b8259-d7mpx" Oct 31 00:36:40.689151 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:40.689144 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c366443-f988-4827-9d18-b985810050da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-730b8259-d7mpx\" (UID: \"5c366443-f988-4827-9d18-b985810050da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-730b8259-d7mpx" Oct 31 00:36:40.689913 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:40.689782 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c366443-f988-4827-9d18-b985810050da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-730b8259-d7mpx\" (UID: \"5c366443-f988-4827-9d18-b985810050da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-730b8259-d7mpx" Oct 31 00:36:40.691542 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:40.691525 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c366443-f988-4827-9d18-b985810050da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-730b8259-d7mpx\" (UID: \"5c366443-f988-4827-9d18-b985810050da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-730b8259-d7mpx" Oct 31 00:36:40.752081 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:40.752046 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qwtn\" (UniqueName: \"kubernetes.io/projected/5c366443-f988-4827-9d18-b985810050da-kube-api-access-2qwtn\") pod \"maas-api-revoke-sub-730b8259-d7mpx\" (UID: \"5c366443-f988-4827-9d18-b985810050da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-730b8259-d7mpx" Oct 31 00:36:40.758785 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:40.758761 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-730b8259-d7mpx" Oct 31 00:36:41.134833 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:36:41.134801 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c366443_f988_4827_9d18_b985810050da.slice/crio-4fe3329b3dbd6d41c87310d346144d8d1425b24f6580264a8b403038ce49ab78 WatchSource:0}: Error finding container 4fe3329b3dbd6d41c87310d346144d8d1425b24f6580264a8b403038ce49ab78: Status 404 returned error can't find the container with id 4fe3329b3dbd6d41c87310d346144d8d1425b24f6580264a8b403038ce49ab78 Oct 31 00:36:41.138835 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:41.138814 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-730b8259-d7mpx"] Oct 31 00:36:41.327138 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:41.327103 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ad093e0-nfqcb"] Oct 31 00:36:41.332633 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:41.332608 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ad093e0-nfqcb" Oct 31 00:36:41.400721 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:41.400621 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ad093e0-nfqcb"] Oct 31 00:36:41.498847 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:41.498765 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63148eb3-f1b9-49a9-be6f-13abac0aa8e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8ad093e0-nfqcb\" (UID: \"63148eb3-f1b9-49a9-be6f-13abac0aa8e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ad093e0-nfqcb" Oct 31 00:36:41.499238 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:41.498884 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wj6nt\" (UniqueName: \"kubernetes.io/projected/63148eb3-f1b9-49a9-be6f-13abac0aa8e9-kube-api-access-wj6nt\") pod \"maas-api-revoke-sub-8ad093e0-nfqcb\" (UID: \"63148eb3-f1b9-49a9-be6f-13abac0aa8e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ad093e0-nfqcb" Oct 31 00:36:41.499238 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:41.498934 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63148eb3-f1b9-49a9-be6f-13abac0aa8e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8ad093e0-nfqcb\" (UID: \"63148eb3-f1b9-49a9-be6f-13abac0aa8e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ad093e0-nfqcb" Oct 31 00:36:41.600051 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:41.600010 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63148eb3-f1b9-49a9-be6f-13abac0aa8e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8ad093e0-nfqcb\" (UID: \"63148eb3-f1b9-49a9-be6f-13abac0aa8e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ad093e0-nfqcb" Oct 31 00:36:41.600238 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:41.600179 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63148eb3-f1b9-49a9-be6f-13abac0aa8e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8ad093e0-nfqcb\" (UID: \"63148eb3-f1b9-49a9-be6f-13abac0aa8e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ad093e0-nfqcb" Oct 31 00:36:41.600238 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:41.600231 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wj6nt\" (UniqueName: \"kubernetes.io/projected/63148eb3-f1b9-49a9-be6f-13abac0aa8e9-kube-api-access-wj6nt\") pod \"maas-api-revoke-sub-8ad093e0-nfqcb\" (UID: \"63148eb3-f1b9-49a9-be6f-13abac0aa8e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ad093e0-nfqcb" Oct 31 00:36:41.600747 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:41.600724 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63148eb3-f1b9-49a9-be6f-13abac0aa8e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8ad093e0-nfqcb\" (UID: \"63148eb3-f1b9-49a9-be6f-13abac0aa8e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ad093e0-nfqcb" Oct 31 00:36:41.602683 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:41.602658 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63148eb3-f1b9-49a9-be6f-13abac0aa8e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8ad093e0-nfqcb\" (UID: \"63148eb3-f1b9-49a9-be6f-13abac0aa8e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ad093e0-nfqcb" Oct 31 00:36:41.663245 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:41.663172 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wj6nt\" (UniqueName: \"kubernetes.io/projected/63148eb3-f1b9-49a9-be6f-13abac0aa8e9-kube-api-access-wj6nt\") pod \"maas-api-revoke-sub-8ad093e0-nfqcb\" (UID: \"63148eb3-f1b9-49a9-be6f-13abac0aa8e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ad093e0-nfqcb" Oct 31 00:36:41.934310 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:41.934217 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-730b8259-d7mpx" event={"ID":"5c366443-f988-4827-9d18-b985810050da","Type":"ContainerStarted","Data":"412fef350af0aeab64a890b63d041929f3a02f0805bdf7d6bf9a19dbc68b6572"} Oct 31 00:36:41.934310 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:41.934272 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-730b8259-d7mpx" event={"ID":"5c366443-f988-4827-9d18-b985810050da","Type":"ContainerStarted","Data":"4fe3329b3dbd6d41c87310d346144d8d1425b24f6580264a8b403038ce49ab78"} Oct 31 00:36:41.962141 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:41.962106 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ad093e0-nfqcb" Oct 31 00:36:42.055949 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:42.055894 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-730b8259-d7mpx" podStartSLOduration=2.055876998 podStartE2EDuration="2.055876998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:36:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:36:42.052064044 +0000 UTC m=+1294.523233808" watchObservedRunningTime="2026-10-31 00:36:42.055876998 +0000 UTC m=+1294.527046760" Oct 31 00:36:42.337120 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:36:42.337075 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63148eb3_f1b9_49a9_be6f_13abac0aa8e9.slice/crio-0ad7764b6b231e959deee8b6dde87c777b5fc87e5f904fbb19dc388d5d5cc971 WatchSource:0}: Error finding container 0ad7764b6b231e959deee8b6dde87c777b5fc87e5f904fbb19dc388d5d5cc971: Status 404 returned error can't find the container with id 0ad7764b6b231e959deee8b6dde87c777b5fc87e5f904fbb19dc388d5d5cc971 Oct 31 00:36:42.340732 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:42.340706 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ad093e0-nfqcb"] Oct 31 00:36:42.944277 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:42.944195 5222 generic.go:356] "Generic (PLEG): container finished" podID="5c366443-f988-4827-9d18-b985810050da" containerID="412fef350af0aeab64a890b63d041929f3a02f0805bdf7d6bf9a19dbc68b6572" exitCode=0 Oct 31 00:36:42.944761 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:42.944290 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-730b8259-d7mpx" event={"ID":"5c366443-f988-4827-9d18-b985810050da","Type":"ContainerDied","Data":"412fef350af0aeab64a890b63d041929f3a02f0805bdf7d6bf9a19dbc68b6572"} Oct 31 00:36:42.946179 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:42.946149 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ad093e0-nfqcb" event={"ID":"63148eb3-f1b9-49a9-be6f-13abac0aa8e9","Type":"ContainerStarted","Data":"dbead3a9fee72047efdc001a4cbcc78958583c7fff7b9555efdf572653417966"} Oct 31 00:36:42.946293 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:42.946208 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ad093e0-nfqcb" event={"ID":"63148eb3-f1b9-49a9-be6f-13abac0aa8e9","Type":"ContainerStarted","Data":"0ad7764b6b231e959deee8b6dde87c777b5fc87e5f904fbb19dc388d5d5cc971"} Oct 31 00:36:43.192763 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:43.192698 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ad093e0-nfqcb" podStartSLOduration=2.192680098 podStartE2EDuration="2.192680098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:36:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:36:43.190441724 +0000 UTC m=+1295.661611474" watchObservedRunningTime="2026-10-31 00:36:43.192680098 +0000 UTC m=+1295.663849846" Oct 31 00:36:43.941154 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:43.941116 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sf5lc"] Oct 31 00:36:43.945341 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:43.945320 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sf5lc" Oct 31 00:36:43.959105 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:43.959072 5222 generic.go:356] "Generic (PLEG): container finished" podID="63148eb3-f1b9-49a9-be6f-13abac0aa8e9" containerID="dbead3a9fee72047efdc001a4cbcc78958583c7fff7b9555efdf572653417966" exitCode=0 Oct 31 00:36:43.959247 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:43.959218 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ad093e0-nfqcb" event={"ID":"63148eb3-f1b9-49a9-be6f-13abac0aa8e9","Type":"ContainerDied","Data":"dbead3a9fee72047efdc001a4cbcc78958583c7fff7b9555efdf572653417966"} Oct 31 00:36:43.999132 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:43.999094 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-101b9e-openshift-default-dockercfg-8l66m\"" Oct 31 00:36:44.035786 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:44.035734 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6039aff2-c967-413b-865a-d933b6c0c9ba-istio-envoy\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sf5lc\" (UID: \"6039aff2-c967-413b-865a-d933b6c0c9ba\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sf5lc" Oct 31 00:36:44.035786 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:44.035783 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6039aff2-c967-413b-865a-d933b6c0c9ba-istio-token\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sf5lc\" (UID: \"6039aff2-c967-413b-865a-d933b6c0c9ba\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sf5lc" Oct 31 00:36:44.036024 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:44.035870 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6039aff2-c967-413b-865a-d933b6c0c9ba-workload-certs\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sf5lc\" (UID: \"6039aff2-c967-413b-865a-d933b6c0c9ba\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sf5lc" Oct 31 00:36:44.036087 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:44.036016 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6039aff2-c967-413b-865a-d933b6c0c9ba-istio-data\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sf5lc\" (UID: \"6039aff2-c967-413b-865a-d933b6c0c9ba\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sf5lc" Oct 31 00:36:44.036087 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:44.036054 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdkfd\" (UniqueName: \"kubernetes.io/projected/6039aff2-c967-413b-865a-d933b6c0c9ba-kube-api-access-zdkfd\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sf5lc\" (UID: \"6039aff2-c967-413b-865a-d933b6c0c9ba\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sf5lc" Oct 31 00:36:44.036200 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:44.036093 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6039aff2-c967-413b-865a-d933b6c0c9ba-istiod-ca-cert\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sf5lc\" (UID: \"6039aff2-c967-413b-865a-d933b6c0c9ba\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sf5lc" Oct 31 00:36:44.036200 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:44.036173 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6039aff2-c967-413b-865a-d933b6c0c9ba-workload-socket\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sf5lc\" (UID: \"6039aff2-c967-413b-865a-d933b6c0c9ba\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sf5lc" Oct 31 00:36:44.036314 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:44.036263 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6039aff2-c967-413b-865a-d933b6c0c9ba-credential-socket\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sf5lc\" (UID: \"6039aff2-c967-413b-865a-d933b6c0c9ba\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sf5lc" Oct 31 00:36:44.036314 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:44.036306 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6039aff2-c967-413b-865a-d933b6c0c9ba-istio-podinfo\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sf5lc\" (UID: \"6039aff2-c967-413b-865a-d933b6c0c9ba\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sf5lc" Oct 31 00:36:44.056701 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:44.056667 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sf5lc"] Oct 31 00:36:44.106558 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:44.106535 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-730b8259-d7mpx" Oct 31 00:36:44.137301 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:44.137262 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6039aff2-c967-413b-865a-d933b6c0c9ba-istio-envoy\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sf5lc\" (UID: \"6039aff2-c967-413b-865a-d933b6c0c9ba\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sf5lc" Oct 31 00:36:44.137544 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:44.137310 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6039aff2-c967-413b-865a-d933b6c0c9ba-istio-token\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sf5lc\" (UID: \"6039aff2-c967-413b-865a-d933b6c0c9ba\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sf5lc" Oct 31 00:36:44.137544 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:44.137361 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6039aff2-c967-413b-865a-d933b6c0c9ba-workload-certs\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sf5lc\" (UID: \"6039aff2-c967-413b-865a-d933b6c0c9ba\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sf5lc" Oct 31 00:36:44.137544 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:44.137454 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6039aff2-c967-413b-865a-d933b6c0c9ba-istio-data\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sf5lc\" (UID: \"6039aff2-c967-413b-865a-d933b6c0c9ba\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sf5lc" Oct 31 00:36:44.137544 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:44.137485 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdkfd\" (UniqueName: \"kubernetes.io/projected/6039aff2-c967-413b-865a-d933b6c0c9ba-kube-api-access-zdkfd\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sf5lc\" (UID: \"6039aff2-c967-413b-865a-d933b6c0c9ba\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sf5lc" Oct 31 00:36:44.137544 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:44.137519 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6039aff2-c967-413b-865a-d933b6c0c9ba-istiod-ca-cert\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sf5lc\" (UID: \"6039aff2-c967-413b-865a-d933b6c0c9ba\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sf5lc" Oct 31 00:36:44.137871 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:44.137593 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6039aff2-c967-413b-865a-d933b6c0c9ba-workload-socket\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sf5lc\" (UID: \"6039aff2-c967-413b-865a-d933b6c0c9ba\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sf5lc" Oct 31 00:36:44.137871 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:44.137681 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6039aff2-c967-413b-865a-d933b6c0c9ba-credential-socket\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sf5lc\" (UID: \"6039aff2-c967-413b-865a-d933b6c0c9ba\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sf5lc" Oct 31 00:36:44.137871 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:44.137727 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6039aff2-c967-413b-865a-d933b6c0c9ba-istio-podinfo\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sf5lc\" (UID: \"6039aff2-c967-413b-865a-d933b6c0c9ba\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sf5lc" Oct 31 00:36:44.138039 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:44.137894 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6039aff2-c967-413b-865a-d933b6c0c9ba-istio-data\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sf5lc\" (UID: \"6039aff2-c967-413b-865a-d933b6c0c9ba\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sf5lc" Oct 31 00:36:44.138039 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:44.137943 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6039aff2-c967-413b-865a-d933b6c0c9ba-workload-certs\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sf5lc\" (UID: \"6039aff2-c967-413b-865a-d933b6c0c9ba\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sf5lc" Oct 31 00:36:44.138142 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:44.138074 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6039aff2-c967-413b-865a-d933b6c0c9ba-workload-socket\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sf5lc\" (UID: \"6039aff2-c967-413b-865a-d933b6c0c9ba\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sf5lc" Oct 31 00:36:44.138388 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:44.138344 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6039aff2-c967-413b-865a-d933b6c0c9ba-istiod-ca-cert\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sf5lc\" (UID: \"6039aff2-c967-413b-865a-d933b6c0c9ba\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sf5lc" Oct 31 00:36:44.138440 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:44.138351 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6039aff2-c967-413b-865a-d933b6c0c9ba-credential-socket\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sf5lc\" (UID: \"6039aff2-c967-413b-865a-d933b6c0c9ba\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sf5lc" Oct 31 00:36:44.139859 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:44.139837 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6039aff2-c967-413b-865a-d933b6c0c9ba-istio-envoy\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sf5lc\" (UID: \"6039aff2-c967-413b-865a-d933b6c0c9ba\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sf5lc" Oct 31 00:36:44.140168 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:44.140150 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6039aff2-c967-413b-865a-d933b6c0c9ba-istio-podinfo\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sf5lc\" (UID: \"6039aff2-c967-413b-865a-d933b6c0c9ba\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sf5lc" Oct 31 00:36:44.205251 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:44.205173 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6039aff2-c967-413b-865a-d933b6c0c9ba-istio-token\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sf5lc\" (UID: \"6039aff2-c967-413b-865a-d933b6c0c9ba\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sf5lc" Oct 31 00:36:44.205251 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:44.205213 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdkfd\" (UniqueName: \"kubernetes.io/projected/6039aff2-c967-413b-865a-d933b6c0c9ba-kube-api-access-zdkfd\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sf5lc\" (UID: \"6039aff2-c967-413b-865a-d933b6c0c9ba\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sf5lc" Oct 31 00:36:44.238576 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:44.238547 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c366443-f988-4827-9d18-b985810050da-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c366443-f988-4827-9d18-b985810050da-maas-api-openshift-ca\") pod \"5c366443-f988-4827-9d18-b985810050da\" (UID: \"5c366443-f988-4827-9d18-b985810050da\") " Oct 31 00:36:44.238576 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:44.238581 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c366443-f988-4827-9d18-b985810050da-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c366443-f988-4827-9d18-b985810050da-maas-api-xks-ca\") pod \"5c366443-f988-4827-9d18-b985810050da\" (UID: \"5c366443-f988-4827-9d18-b985810050da\") " Oct 31 00:36:44.238811 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:44.238672 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c366443-f988-4827-9d18-b985810050da-kube-api-access-2qwtn\" (UniqueName: \"kubernetes.io/projected/5c366443-f988-4827-9d18-b985810050da-kube-api-access-2qwtn\") pod \"5c366443-f988-4827-9d18-b985810050da\" (UID: \"5c366443-f988-4827-9d18-b985810050da\") " Oct 31 00:36:44.238983 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:44.238961 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c366443-f988-4827-9d18-b985810050da-maas-api-openshift-ca" pod "5c366443-f988-4827-9d18-b985810050da" (UID: "5c366443-f988-4827-9d18-b985810050da"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:36:44.239218 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:44.239198 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c366443-f988-4827-9d18-b985810050da-maas-api-openshift-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:36:44.240859 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:44.240831 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c366443-f988-4827-9d18-b985810050da-maas-api-xks-ca" pod "5c366443-f988-4827-9d18-b985810050da" (UID: "5c366443-f988-4827-9d18-b985810050da"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:36:44.240976 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:44.240882 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c366443-f988-4827-9d18-b985810050da-kube-api-access-2qwtn" pod "5c366443-f988-4827-9d18-b985810050da" (UID: "5c366443-f988-4827-9d18-b985810050da"). InnerVolumeSpecName "kube-api-access-2qwtn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:36:44.260141 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:44.260108 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sf5lc" Oct 31 00:36:44.340790 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:44.340761 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c366443-f988-4827-9d18-b985810050da-maas-api-xks-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:36:44.340790 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:44.340789 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qwtn\" (UniqueName: \"kubernetes.io/projected/5c366443-f988-4827-9d18-b985810050da-kube-api-access-2qwtn\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:36:44.700327 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:36:44.700290 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6039aff2_c967_413b_865a_d933b6c0c9ba.slice/crio-1d3484ed1ef5ced6e5ec9da187ca5252b57767fc95c4295553de2040ab2b2e6c WatchSource:0}: Error finding container 1d3484ed1ef5ced6e5ec9da187ca5252b57767fc95c4295553de2040ab2b2e6c: Status 404 returned error can't find the container with id 1d3484ed1ef5ced6e5ec9da187ca5252b57767fc95c4295553de2040ab2b2e6c Oct 31 00:36:44.703127 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:44.703081 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:36:44.703210 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:44.703174 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:36:44.703244 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:44.703218 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:36:44.704524 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:44.704503 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sf5lc"] Oct 31 00:36:44.970124 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:44.970019 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sf5lc" event={"ID":"6039aff2-c967-413b-865a-d933b6c0c9ba","Type":"ContainerStarted","Data":"e80d62d0507fb6c42657188e5402d5a0f774b71d46f47fa2b20847b935c462ce"} Oct 31 00:36:44.970124 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:44.970096 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sf5lc" event={"ID":"6039aff2-c967-413b-865a-d933b6c0c9ba","Type":"ContainerStarted","Data":"1d3484ed1ef5ced6e5ec9da187ca5252b57767fc95c4295553de2040ab2b2e6c"} Oct 31 00:36:44.971914 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:44.971890 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-730b8259-d7mpx" Oct 31 00:36:44.972053 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:44.971919 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-730b8259-d7mpx" event={"ID":"5c366443-f988-4827-9d18-b985810050da","Type":"ContainerDied","Data":"4fe3329b3dbd6d41c87310d346144d8d1425b24f6580264a8b403038ce49ab78"} Oct 31 00:36:44.972053 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:44.971967 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4fe3329b3dbd6d41c87310d346144d8d1425b24f6580264a8b403038ce49ab78" Oct 31 00:36:45.058790 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:45.058759 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f04bac1c-nlzq5"] Oct 31 00:36:45.062323 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:45.062298 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f04bac1c-nlzq5"] Oct 31 00:36:45.096145 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:45.096099 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sf5lc" podStartSLOduration=2.096083213 podStartE2EDuration="2.096083213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:36:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:36:45.095751344 +0000 UTC m=+1297.566921093" watchObservedRunningTime="2026-10-31 00:36:45.096083213 +0000 UTC m=+1297.567252962" Oct 31 00:36:45.114586 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:45.114552 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-056c3b16-dbshj"] Oct 31 00:36:45.115331 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:45.115311 5222 cpu_manager.go:414] "containerMap: removing container" podUID="5c366443-f988-4827-9d18-b985810050da" containerName="revoke-keys" Oct 31 00:36:45.115422 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:45.115336 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c366443-f988-4827-9d18-b985810050da" containerName="revoke-keys" Oct 31 00:36:45.115616 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:45.115599 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c366443-f988-4827-9d18-b985810050da" containerName="revoke-keys" Oct 31 00:36:45.120320 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:45.120301 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-056c3b16-dbshj" Oct 31 00:36:45.144862 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:45.144835 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ad093e0-nfqcb" Oct 31 00:36:45.169334 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:45.169303 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-056c3b16-dbshj"] Oct 31 00:36:45.251957 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:45.251879 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63148eb3-f1b9-49a9-be6f-13abac0aa8e9-kube-api-access-wj6nt\" (UniqueName: \"kubernetes.io/projected/63148eb3-f1b9-49a9-be6f-13abac0aa8e9-kube-api-access-wj6nt\") pod \"63148eb3-f1b9-49a9-be6f-13abac0aa8e9\" (UID: \"63148eb3-f1b9-49a9-be6f-13abac0aa8e9\") " Oct 31 00:36:45.252105 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:45.251995 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63148eb3-f1b9-49a9-be6f-13abac0aa8e9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63148eb3-f1b9-49a9-be6f-13abac0aa8e9-maas-api-xks-ca\") pod \"63148eb3-f1b9-49a9-be6f-13abac0aa8e9\" (UID: \"63148eb3-f1b9-49a9-be6f-13abac0aa8e9\") " Oct 31 00:36:45.252105 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:45.252042 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63148eb3-f1b9-49a9-be6f-13abac0aa8e9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63148eb3-f1b9-49a9-be6f-13abac0aa8e9-maas-api-openshift-ca\") pod \"63148eb3-f1b9-49a9-be6f-13abac0aa8e9\" (UID: \"63148eb3-f1b9-49a9-be6f-13abac0aa8e9\") " Oct 31 00:36:45.252461 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:45.252430 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bf2l\" (UniqueName: \"kubernetes.io/projected/4ca0cfaa-8fd4-473b-9d14-776c25ffdb05-kube-api-access-7bf2l\") pod \"maas-api-revoke-sub-056c3b16-dbshj\" (UID: \"4ca0cfaa-8fd4-473b-9d14-776c25ffdb05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-056c3b16-dbshj" Oct 31 00:36:45.252602 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:45.252448 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63148eb3-f1b9-49a9-be6f-13abac0aa8e9-maas-api-openshift-ca" pod "63148eb3-f1b9-49a9-be6f-13abac0aa8e9" (UID: "63148eb3-f1b9-49a9-be6f-13abac0aa8e9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:36:45.252602 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:45.252477 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ca0cfaa-8fd4-473b-9d14-776c25ffdb05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-056c3b16-dbshj\" (UID: \"4ca0cfaa-8fd4-473b-9d14-776c25ffdb05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-056c3b16-dbshj" Oct 31 00:36:45.252713 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:45.252600 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ca0cfaa-8fd4-473b-9d14-776c25ffdb05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-056c3b16-dbshj\" (UID: \"4ca0cfaa-8fd4-473b-9d14-776c25ffdb05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-056c3b16-dbshj" Oct 31 00:36:45.252713 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:45.252687 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63148eb3-f1b9-49a9-be6f-13abac0aa8e9-maas-api-openshift-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:36:45.254197 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:45.254170 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63148eb3-f1b9-49a9-be6f-13abac0aa8e9-kube-api-access-wj6nt" pod "63148eb3-f1b9-49a9-be6f-13abac0aa8e9" (UID: "63148eb3-f1b9-49a9-be6f-13abac0aa8e9"). InnerVolumeSpecName "kube-api-access-wj6nt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:36:45.254312 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:45.254292 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63148eb3-f1b9-49a9-be6f-13abac0aa8e9-maas-api-xks-ca" pod "63148eb3-f1b9-49a9-be6f-13abac0aa8e9" (UID: "63148eb3-f1b9-49a9-be6f-13abac0aa8e9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:36:45.261194 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:45.261162 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sf5lc" Oct 31 00:36:45.262841 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:45.262813 5222 patch_prober.go:29] interesting pod/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sf5lc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 31 00:36:45.262984 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:45.262873 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sf5lc" podUID="6039aff2-c967-413b-865a-d933b6c0c9ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 31 00:36:45.354379 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:45.354321 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ca0cfaa-8fd4-473b-9d14-776c25ffdb05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-056c3b16-dbshj\" (UID: \"4ca0cfaa-8fd4-473b-9d14-776c25ffdb05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-056c3b16-dbshj" Oct 31 00:36:45.354582 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:45.354522 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bf2l\" (UniqueName: \"kubernetes.io/projected/4ca0cfaa-8fd4-473b-9d14-776c25ffdb05-kube-api-access-7bf2l\") pod \"maas-api-revoke-sub-056c3b16-dbshj\" (UID: \"4ca0cfaa-8fd4-473b-9d14-776c25ffdb05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-056c3b16-dbshj" Oct 31 00:36:45.354582 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:45.354547 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ca0cfaa-8fd4-473b-9d14-776c25ffdb05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-056c3b16-dbshj\" (UID: \"4ca0cfaa-8fd4-473b-9d14-776c25ffdb05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-056c3b16-dbshj" Oct 31 00:36:45.354694 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:45.354604 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wj6nt\" (UniqueName: \"kubernetes.io/projected/63148eb3-f1b9-49a9-be6f-13abac0aa8e9-kube-api-access-wj6nt\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:36:45.354694 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:45.354615 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63148eb3-f1b9-49a9-be6f-13abac0aa8e9-maas-api-xks-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:36:45.355191 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:45.355170 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ca0cfaa-8fd4-473b-9d14-776c25ffdb05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-056c3b16-dbshj\" (UID: \"4ca0cfaa-8fd4-473b-9d14-776c25ffdb05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-056c3b16-dbshj" Oct 31 00:36:45.356984 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:45.356966 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ca0cfaa-8fd4-473b-9d14-776c25ffdb05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-056c3b16-dbshj\" (UID: \"4ca0cfaa-8fd4-473b-9d14-776c25ffdb05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-056c3b16-dbshj" Oct 31 00:36:45.417161 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:45.417128 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bf2l\" (UniqueName: \"kubernetes.io/projected/4ca0cfaa-8fd4-473b-9d14-776c25ffdb05-kube-api-access-7bf2l\") pod \"maas-api-revoke-sub-056c3b16-dbshj\" (UID: \"4ca0cfaa-8fd4-473b-9d14-776c25ffdb05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-056c3b16-dbshj" Oct 31 00:36:45.455383 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:45.455339 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-056c3b16-dbshj" Oct 31 00:36:45.656666 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:45.656635 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5e26610-abd9-476d-b1d5-151a53d88520" path="/var/lib/kubelet/pods/a5e26610-abd9-476d-b1d5-151a53d88520/volumes" Oct 31 00:36:45.837776 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:36:45.837745 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ca0cfaa_8fd4_473b_9d14_776c25ffdb05.slice/crio-ecce6d4073b91824653bcc31ed22a97ca6bb677312fab1485291f6256b717726 WatchSource:0}: Error finding container ecce6d4073b91824653bcc31ed22a97ca6bb677312fab1485291f6256b717726: Status 404 returned error can't find the container with id ecce6d4073b91824653bcc31ed22a97ca6bb677312fab1485291f6256b717726 Oct 31 00:36:45.841974 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:45.841858 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-056c3b16-dbshj"] Oct 31 00:36:45.980572 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:45.980544 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ad093e0-nfqcb" Oct 31 00:36:45.981020 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:45.980572 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ad093e0-nfqcb" event={"ID":"63148eb3-f1b9-49a9-be6f-13abac0aa8e9","Type":"ContainerDied","Data":"0ad7764b6b231e959deee8b6dde87c777b5fc87e5f904fbb19dc388d5d5cc971"} Oct 31 00:36:45.981020 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:45.980627 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ad7764b6b231e959deee8b6dde87c777b5fc87e5f904fbb19dc388d5d5cc971" Oct 31 00:36:45.982088 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:45.982062 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-056c3b16-dbshj" event={"ID":"4ca0cfaa-8fd4-473b-9d14-776c25ffdb05","Type":"ContainerStarted","Data":"ecce6d4073b91824653bcc31ed22a97ca6bb677312fab1485291f6256b717726"} Oct 31 00:36:46.083926 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:46.083855 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-beee7c3c-568ms"] Oct 31 00:36:46.087895 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:46.087823 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-beee7c3c-568ms"] Oct 31 00:36:46.107873 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:46.107843 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6bf713ef-4p9bn"] Oct 31 00:36:46.108460 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:46.108426 5222 cpu_manager.go:414] "containerMap: removing container" podUID="63148eb3-f1b9-49a9-be6f-13abac0aa8e9" containerName="revoke-keys" Oct 31 00:36:46.108460 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:46.108453 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63148eb3-f1b9-49a9-be6f-13abac0aa8e9" containerName="revoke-keys" Oct 31 00:36:46.108630 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:46.108570 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="63148eb3-f1b9-49a9-be6f-13abac0aa8e9" containerName="revoke-keys" Oct 31 00:36:46.113506 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:46.113477 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bf713ef-4p9bn" Oct 31 00:36:46.168548 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:46.168511 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6bf713ef-4p9bn"] Oct 31 00:36:46.262973 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:46.262934 5222 patch_prober.go:29] interesting pod/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sf5lc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 31 00:36:46.263169 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:46.263004 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sf5lc" podUID="6039aff2-c967-413b-865a-d933b6c0c9ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 31 00:36:46.274883 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:46.274849 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcsc7\" (UniqueName: \"kubernetes.io/projected/90e2a297-20cc-42a5-bf04-5df44c6bfecb-kube-api-access-gcsc7\") pod \"maas-api-revoke-sub-6bf713ef-4p9bn\" (UID: \"90e2a297-20cc-42a5-bf04-5df44c6bfecb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bf713ef-4p9bn" Oct 31 00:36:46.275063 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:46.275042 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90e2a297-20cc-42a5-bf04-5df44c6bfecb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6bf713ef-4p9bn\" (UID: \"90e2a297-20cc-42a5-bf04-5df44c6bfecb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bf713ef-4p9bn" Oct 31 00:36:46.275182 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:46.275163 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90e2a297-20cc-42a5-bf04-5df44c6bfecb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6bf713ef-4p9bn\" (UID: \"90e2a297-20cc-42a5-bf04-5df44c6bfecb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bf713ef-4p9bn" Oct 31 00:36:46.377094 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:46.377007 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90e2a297-20cc-42a5-bf04-5df44c6bfecb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6bf713ef-4p9bn\" (UID: \"90e2a297-20cc-42a5-bf04-5df44c6bfecb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bf713ef-4p9bn" Oct 31 00:36:46.377251 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:46.377193 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcsc7\" (UniqueName: \"kubernetes.io/projected/90e2a297-20cc-42a5-bf04-5df44c6bfecb-kube-api-access-gcsc7\") pod \"maas-api-revoke-sub-6bf713ef-4p9bn\" (UID: \"90e2a297-20cc-42a5-bf04-5df44c6bfecb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bf713ef-4p9bn" Oct 31 00:36:46.377292 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:46.377275 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90e2a297-20cc-42a5-bf04-5df44c6bfecb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6bf713ef-4p9bn\" (UID: \"90e2a297-20cc-42a5-bf04-5df44c6bfecb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bf713ef-4p9bn" Oct 31 00:36:46.378011 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:46.377971 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90e2a297-20cc-42a5-bf04-5df44c6bfecb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6bf713ef-4p9bn\" (UID: \"90e2a297-20cc-42a5-bf04-5df44c6bfecb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bf713ef-4p9bn" Oct 31 00:36:46.379777 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:46.379755 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90e2a297-20cc-42a5-bf04-5df44c6bfecb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6bf713ef-4p9bn\" (UID: \"90e2a297-20cc-42a5-bf04-5df44c6bfecb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bf713ef-4p9bn" Oct 31 00:36:46.457396 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:46.452211 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcsc7\" (UniqueName: \"kubernetes.io/projected/90e2a297-20cc-42a5-bf04-5df44c6bfecb-kube-api-access-gcsc7\") pod \"maas-api-revoke-sub-6bf713ef-4p9bn\" (UID: \"90e2a297-20cc-42a5-bf04-5df44c6bfecb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bf713ef-4p9bn" Oct 31 00:36:46.730447 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:46.730356 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bf713ef-4p9bn" Oct 31 00:36:46.990747 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:46.990654 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-056c3b16-dbshj" event={"ID":"4ca0cfaa-8fd4-473b-9d14-776c25ffdb05","Type":"ContainerStarted","Data":"85779401936cb28c830631bdcccaf0766b51b789218f8bad7a2a2bbb77b1f2aa"} Oct 31 00:36:47.065425 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:47.065390 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8010cb27-mhmdq"] Oct 31 00:36:47.113745 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:47.113709 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8010cb27-mhmdq"] Oct 31 00:36:47.187995 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:47.187863 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-056c3b16-dbshj" podStartSLOduration=2.187848738 podStartE2EDuration="2.187848738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:36:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:36:47.133051491 +0000 UTC m=+1299.604221240" watchObservedRunningTime="2026-10-31 00:36:47.187848738 +0000 UTC m=+1299.659018488" Oct 31 00:36:47.191090 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:47.191060 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6bf713ef-4p9bn"] Oct 31 00:36:47.191326 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:36:47.191305 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90e2a297_20cc_42a5_bf04_5df44c6bfecb.slice/crio-5a62d7ccb486c7ef8395692ffaef92909a73dabedd0c331dd51577d7acc735f4 WatchSource:0}: Error finding container 5a62d7ccb486c7ef8395692ffaef92909a73dabedd0c331dd51577d7acc735f4: Status 404 returned error can't find the container with id 5a62d7ccb486c7ef8395692ffaef92909a73dabedd0c331dd51577d7acc735f4 Oct 31 00:36:47.262449 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:47.261540 5222 patch_prober.go:29] interesting pod/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sf5lc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 31 00:36:47.262449 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:47.261605 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sf5lc" podUID="6039aff2-c967-413b-865a-d933b6c0c9ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 31 00:36:47.671976 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:47.671945 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9dd3e16f-40ab-458e-801c-fffa6b8b000b" path="/var/lib/kubelet/pods/9dd3e16f-40ab-458e-801c-fffa6b8b000b/volumes" Oct 31 00:36:47.672501 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:47.672486 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b533be37-fc63-4eb5-9e7e-5f2b6d6cd815" path="/var/lib/kubelet/pods/b533be37-fc63-4eb5-9e7e-5f2b6d6cd815/volumes" Oct 31 00:36:47.999702 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:47.999610 5222 generic.go:356] "Generic (PLEG): container finished" podID="4ca0cfaa-8fd4-473b-9d14-776c25ffdb05" containerID="85779401936cb28c830631bdcccaf0766b51b789218f8bad7a2a2bbb77b1f2aa" exitCode=0 Oct 31 00:36:47.999702 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:47.999687 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-056c3b16-dbshj" event={"ID":"4ca0cfaa-8fd4-473b-9d14-776c25ffdb05","Type":"ContainerDied","Data":"85779401936cb28c830631bdcccaf0766b51b789218f8bad7a2a2bbb77b1f2aa"} Oct 31 00:36:48.001621 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:48.001593 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bf713ef-4p9bn" event={"ID":"90e2a297-20cc-42a5-bf04-5df44c6bfecb","Type":"ContainerStarted","Data":"73a6ea12b22cb3de90da79d70911a1f415fdad53ecdd18d201fcea1017921ea4"} Oct 31 00:36:48.001764 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:48.001644 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bf713ef-4p9bn" event={"ID":"90e2a297-20cc-42a5-bf04-5df44c6bfecb","Type":"ContainerStarted","Data":"5a62d7ccb486c7ef8395692ffaef92909a73dabedd0c331dd51577d7acc735f4"} Oct 31 00:36:48.264462 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:48.263859 5222 patch_prober.go:29] interesting pod/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sf5lc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 31 00:36:48.264462 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:48.263924 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sf5lc" podUID="6039aff2-c967-413b-865a-d933b6c0c9ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 31 00:36:48.276783 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:48.276729 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bf713ef-4p9bn" podStartSLOduration=2.276713642 podStartE2EDuration="2.276713642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:36:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:36:48.272291051 +0000 UTC m=+1300.743460801" watchObservedRunningTime="2026-10-31 00:36:48.276713642 +0000 UTC m=+1300.747883391" Oct 31 00:36:49.019754 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:49.019721 5222 generic.go:356] "Generic (PLEG): container finished" podID="90e2a297-20cc-42a5-bf04-5df44c6bfecb" containerID="73a6ea12b22cb3de90da79d70911a1f415fdad53ecdd18d201fcea1017921ea4" exitCode=0 Oct 31 00:36:49.020167 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:49.019797 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bf713ef-4p9bn" event={"ID":"90e2a297-20cc-42a5-bf04-5df44c6bfecb","Type":"ContainerDied","Data":"73a6ea12b22cb3de90da79d70911a1f415fdad53ecdd18d201fcea1017921ea4"} Oct 31 00:36:49.216743 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:49.216711 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-056c3b16-dbshj" Oct 31 00:36:49.262398 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:49.260876 5222 patch_prober.go:29] interesting pod/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sf5lc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 31 00:36:49.262398 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:49.260938 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sf5lc" podUID="6039aff2-c967-413b-865a-d933b6c0c9ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 31 00:36:49.314134 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:49.314045 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ca0cfaa-8fd4-473b-9d14-776c25ffdb05-kube-api-access-7bf2l\" (UniqueName: \"kubernetes.io/projected/4ca0cfaa-8fd4-473b-9d14-776c25ffdb05-kube-api-access-7bf2l\") pod \"4ca0cfaa-8fd4-473b-9d14-776c25ffdb05\" (UID: \"4ca0cfaa-8fd4-473b-9d14-776c25ffdb05\") " Oct 31 00:36:49.314290 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:49.314258 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ca0cfaa-8fd4-473b-9d14-776c25ffdb05-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ca0cfaa-8fd4-473b-9d14-776c25ffdb05-maas-api-openshift-ca\") pod \"4ca0cfaa-8fd4-473b-9d14-776c25ffdb05\" (UID: \"4ca0cfaa-8fd4-473b-9d14-776c25ffdb05\") " Oct 31 00:36:49.314332 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:49.314317 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ca0cfaa-8fd4-473b-9d14-776c25ffdb05-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ca0cfaa-8fd4-473b-9d14-776c25ffdb05-maas-api-xks-ca\") pod \"4ca0cfaa-8fd4-473b-9d14-776c25ffdb05\" (UID: \"4ca0cfaa-8fd4-473b-9d14-776c25ffdb05\") " Oct 31 00:36:49.314651 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:49.314614 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ca0cfaa-8fd4-473b-9d14-776c25ffdb05-maas-api-openshift-ca" pod "4ca0cfaa-8fd4-473b-9d14-776c25ffdb05" (UID: "4ca0cfaa-8fd4-473b-9d14-776c25ffdb05"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:36:49.314801 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:49.314758 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ca0cfaa-8fd4-473b-9d14-776c25ffdb05-maas-api-openshift-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:36:49.316472 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:49.316446 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ca0cfaa-8fd4-473b-9d14-776c25ffdb05-kube-api-access-7bf2l" pod "4ca0cfaa-8fd4-473b-9d14-776c25ffdb05" (UID: "4ca0cfaa-8fd4-473b-9d14-776c25ffdb05"). InnerVolumeSpecName "kube-api-access-7bf2l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:36:49.316580 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:49.316511 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ca0cfaa-8fd4-473b-9d14-776c25ffdb05-maas-api-xks-ca" pod "4ca0cfaa-8fd4-473b-9d14-776c25ffdb05" (UID: "4ca0cfaa-8fd4-473b-9d14-776c25ffdb05"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:36:49.416334 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:49.416285 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ca0cfaa-8fd4-473b-9d14-776c25ffdb05-maas-api-xks-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:36:49.416334 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:49.416332 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bf2l\" (UniqueName: \"kubernetes.io/projected/4ca0cfaa-8fd4-473b-9d14-776c25ffdb05-kube-api-access-7bf2l\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:36:50.028492 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:50.028459 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-056c3b16-dbshj" Oct 31 00:36:50.028940 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:50.028464 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-056c3b16-dbshj" event={"ID":"4ca0cfaa-8fd4-473b-9d14-776c25ffdb05","Type":"ContainerDied","Data":"ecce6d4073b91824653bcc31ed22a97ca6bb677312fab1485291f6256b717726"} Oct 31 00:36:50.028940 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:50.028579 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ecce6d4073b91824653bcc31ed22a97ca6bb677312fab1485291f6256b717726" Oct 31 00:36:50.229339 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:50.229316 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bf713ef-4p9bn" Oct 31 00:36:50.262557 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:50.262503 5222 patch_prober.go:29] interesting pod/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sf5lc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 31 00:36:50.262765 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:50.262572 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sf5lc" podUID="6039aff2-c967-413b-865a-d933b6c0c9ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 31 00:36:50.327069 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:50.326988 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90e2a297-20cc-42a5-bf04-5df44c6bfecb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90e2a297-20cc-42a5-bf04-5df44c6bfecb-maas-api-openshift-ca\") pod \"90e2a297-20cc-42a5-bf04-5df44c6bfecb\" (UID: \"90e2a297-20cc-42a5-bf04-5df44c6bfecb\") " Oct 31 00:36:50.327069 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:50.327045 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90e2a297-20cc-42a5-bf04-5df44c6bfecb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90e2a297-20cc-42a5-bf04-5df44c6bfecb-maas-api-xks-ca\") pod \"90e2a297-20cc-42a5-bf04-5df44c6bfecb\" (UID: \"90e2a297-20cc-42a5-bf04-5df44c6bfecb\") " Oct 31 00:36:50.327254 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:50.327186 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90e2a297-20cc-42a5-bf04-5df44c6bfecb-kube-api-access-gcsc7\" (UniqueName: \"kubernetes.io/projected/90e2a297-20cc-42a5-bf04-5df44c6bfecb-kube-api-access-gcsc7\") pod \"90e2a297-20cc-42a5-bf04-5df44c6bfecb\" (UID: \"90e2a297-20cc-42a5-bf04-5df44c6bfecb\") " Oct 31 00:36:50.327402 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:50.327359 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90e2a297-20cc-42a5-bf04-5df44c6bfecb-maas-api-openshift-ca" pod "90e2a297-20cc-42a5-bf04-5df44c6bfecb" (UID: "90e2a297-20cc-42a5-bf04-5df44c6bfecb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:36:50.327635 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:50.327609 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90e2a297-20cc-42a5-bf04-5df44c6bfecb-maas-api-openshift-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:36:50.329341 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:50.329320 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90e2a297-20cc-42a5-bf04-5df44c6bfecb-kube-api-access-gcsc7" pod "90e2a297-20cc-42a5-bf04-5df44c6bfecb" (UID: "90e2a297-20cc-42a5-bf04-5df44c6bfecb"). InnerVolumeSpecName "kube-api-access-gcsc7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:36:50.329457 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:50.329435 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90e2a297-20cc-42a5-bf04-5df44c6bfecb-maas-api-xks-ca" pod "90e2a297-20cc-42a5-bf04-5df44c6bfecb" (UID: "90e2a297-20cc-42a5-bf04-5df44c6bfecb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:36:50.428899 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:50.428868 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90e2a297-20cc-42a5-bf04-5df44c6bfecb-maas-api-xks-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:36:50.428899 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:50.428895 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcsc7\" (UniqueName: \"kubernetes.io/projected/90e2a297-20cc-42a5-bf04-5df44c6bfecb-kube-api-access-gcsc7\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:36:51.036984 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:51.036952 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bf713ef-4p9bn" Oct 31 00:36:51.037442 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:51.036980 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bf713ef-4p9bn" event={"ID":"90e2a297-20cc-42a5-bf04-5df44c6bfecb","Type":"ContainerDied","Data":"5a62d7ccb486c7ef8395692ffaef92909a73dabedd0c331dd51577d7acc735f4"} Oct 31 00:36:51.037442 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:51.037036 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5a62d7ccb486c7ef8395692ffaef92909a73dabedd0c331dd51577d7acc735f4" Oct 31 00:36:51.262426 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:51.260872 5222 patch_prober.go:29] interesting pod/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sf5lc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 31 00:36:51.262426 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:51.260937 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sf5lc" podUID="6039aff2-c967-413b-865a-d933b6c0c9ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 31 00:36:51.505361 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:51.505266 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-79tk8"] Oct 31 00:36:52.261394 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:52.260816 5222 patch_prober.go:29] interesting pod/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sf5lc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 31 00:36:52.261394 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:52.260877 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sf5lc" podUID="6039aff2-c967-413b-865a-d933b6c0c9ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 31 00:36:54.326556 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:54.326464 5222 patch_prober.go:29] interesting pod/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sf5lc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:36:54.326556 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:54.326529 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sf5lc" podUID="6039aff2-c967-413b-865a-d933b6c0c9ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:36:54.680447 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:54.680344 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sf5lc" Oct 31 00:36:54.681266 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:54.681241 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sf5lc" Oct 31 00:36:54.682087 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:54.682066 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sf5lc" Oct 31 00:36:56.881196 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:56.881151 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b182c63-kgc4k"] Oct 31 00:36:56.881741 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:56.881726 5222 cpu_manager.go:414] "containerMap: removing container" podUID="4ca0cfaa-8fd4-473b-9d14-776c25ffdb05" containerName="revoke-keys" Oct 31 00:36:56.881782 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:56.881743 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ca0cfaa-8fd4-473b-9d14-776c25ffdb05" containerName="revoke-keys" Oct 31 00:36:56.881782 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:56.881756 5222 cpu_manager.go:414] "containerMap: removing container" podUID="90e2a297-20cc-42a5-bf04-5df44c6bfecb" containerName="revoke-keys" Oct 31 00:36:56.881782 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:56.881761 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90e2a297-20cc-42a5-bf04-5df44c6bfecb" containerName="revoke-keys" Oct 31 00:36:56.881868 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:56.881862 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ca0cfaa-8fd4-473b-9d14-776c25ffdb05" containerName="revoke-keys" Oct 31 00:36:56.881903 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:56.881870 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="90e2a297-20cc-42a5-bf04-5df44c6bfecb" containerName="revoke-keys" Oct 31 00:36:56.885650 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:56.885633 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b182c63-kgc4k" Oct 31 00:36:56.940784 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:56.940747 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b182c63-kgc4k"] Oct 31 00:36:56.986580 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:56.986547 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:36:56.986580 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:56.986576 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:36:56.986804 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:56.986612 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfz9z\"" Oct 31 00:36:56.986804 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:56.986635 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:36:57.016823 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:57.016784 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qq9q9\" (UniqueName: \"kubernetes.io/projected/e22b4bd7-7950-40e6-88c5-5bfba6415954-kube-api-access-qq9q9\") pod \"maas-api-revoke-sub-9b182c63-kgc4k\" (UID: \"e22b4bd7-7950-40e6-88c5-5bfba6415954\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b182c63-kgc4k" Oct 31 00:36:57.017040 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:57.016850 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e22b4bd7-7950-40e6-88c5-5bfba6415954-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b182c63-kgc4k\" (UID: \"e22b4bd7-7950-40e6-88c5-5bfba6415954\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b182c63-kgc4k" Oct 31 00:36:57.017040 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:57.016926 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e22b4bd7-7950-40e6-88c5-5bfba6415954-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b182c63-kgc4k\" (UID: \"e22b4bd7-7950-40e6-88c5-5bfba6415954\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b182c63-kgc4k" Oct 31 00:36:57.117891 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:57.117857 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qq9q9\" (UniqueName: \"kubernetes.io/projected/e22b4bd7-7950-40e6-88c5-5bfba6415954-kube-api-access-qq9q9\") pod \"maas-api-revoke-sub-9b182c63-kgc4k\" (UID: \"e22b4bd7-7950-40e6-88c5-5bfba6415954\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b182c63-kgc4k" Oct 31 00:36:57.118093 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:57.117906 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e22b4bd7-7950-40e6-88c5-5bfba6415954-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b182c63-kgc4k\" (UID: \"e22b4bd7-7950-40e6-88c5-5bfba6415954\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b182c63-kgc4k" Oct 31 00:36:57.118093 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:57.117940 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e22b4bd7-7950-40e6-88c5-5bfba6415954-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b182c63-kgc4k\" (UID: \"e22b4bd7-7950-40e6-88c5-5bfba6415954\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b182c63-kgc4k" Oct 31 00:36:57.118666 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:57.118640 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e22b4bd7-7950-40e6-88c5-5bfba6415954-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b182c63-kgc4k\" (UID: \"e22b4bd7-7950-40e6-88c5-5bfba6415954\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b182c63-kgc4k" Oct 31 00:36:57.120667 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:57.120646 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e22b4bd7-7950-40e6-88c5-5bfba6415954-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b182c63-kgc4k\" (UID: \"e22b4bd7-7950-40e6-88c5-5bfba6415954\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b182c63-kgc4k" Oct 31 00:36:57.181338 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:57.181259 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qq9q9\" (UniqueName: \"kubernetes.io/projected/e22b4bd7-7950-40e6-88c5-5bfba6415954-kube-api-access-qq9q9\") pod \"maas-api-revoke-sub-9b182c63-kgc4k\" (UID: \"e22b4bd7-7950-40e6-88c5-5bfba6415954\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b182c63-kgc4k" Oct 31 00:36:57.204122 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:57.204091 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b182c63-kgc4k" Oct 31 00:36:57.581750 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:36:57.581716 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode22b4bd7_7950_40e6_88c5_5bfba6415954.slice/crio-5c4743ba8d422053221aa1684361d1ac3cfe7b76d89b41499db678262ee80ab4 WatchSource:0}: Error finding container 5c4743ba8d422053221aa1684361d1ac3cfe7b76d89b41499db678262ee80ab4: Status 404 returned error can't find the container with id 5c4743ba8d422053221aa1684361d1ac3cfe7b76d89b41499db678262ee80ab4 Oct 31 00:36:57.585822 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:57.585798 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b182c63-kgc4k"] Oct 31 00:36:58.143306 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:58.143221 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b182c63-kgc4k" event={"ID":"e22b4bd7-7950-40e6-88c5-5bfba6415954","Type":"ContainerStarted","Data":"db4a16dd001f7ce0260b1e9305eead3b0cc915cd26f013b4b420f8821388ea42"} Oct 31 00:36:58.143306 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:58.143279 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b182c63-kgc4k" event={"ID":"e22b4bd7-7950-40e6-88c5-5bfba6415954","Type":"ContainerStarted","Data":"5c4743ba8d422053221aa1684361d1ac3cfe7b76d89b41499db678262ee80ab4"} Oct 31 00:36:58.281302 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:58.275442 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b182c63-kgc4k" podStartSLOduration=2.2754187679999998 podStartE2EDuration="2.275418768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:36:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:36:58.268748093 +0000 UTC m=+1310.739917845" watchObservedRunningTime="2026-10-31 00:36:58.275418768 +0000 UTC m=+1310.746588520" Oct 31 00:36:58.286156 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:58.286124 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f65ac1f3-wtqvf"] Oct 31 00:36:58.291792 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:58.291762 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f65ac1f3-wtqvf" Oct 31 00:36:58.372051 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:58.372014 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f65ac1f3-wtqvf"] Oct 31 00:36:58.436510 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:58.436479 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/701064e2-35dc-4b2d-9901-84c186af8f47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f65ac1f3-wtqvf\" (UID: \"701064e2-35dc-4b2d-9901-84c186af8f47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f65ac1f3-wtqvf" Oct 31 00:36:58.436663 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:58.436524 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8vrt\" (UniqueName: \"kubernetes.io/projected/701064e2-35dc-4b2d-9901-84c186af8f47-kube-api-access-l8vrt\") pod \"maas-api-revoke-sub-f65ac1f3-wtqvf\" (UID: \"701064e2-35dc-4b2d-9901-84c186af8f47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f65ac1f3-wtqvf" Oct 31 00:36:58.436843 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:58.436819 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/701064e2-35dc-4b2d-9901-84c186af8f47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f65ac1f3-wtqvf\" (UID: \"701064e2-35dc-4b2d-9901-84c186af8f47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f65ac1f3-wtqvf" Oct 31 00:36:58.538694 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:58.538658 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/701064e2-35dc-4b2d-9901-84c186af8f47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f65ac1f3-wtqvf\" (UID: \"701064e2-35dc-4b2d-9901-84c186af8f47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f65ac1f3-wtqvf" Oct 31 00:36:58.538922 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:58.538720 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/701064e2-35dc-4b2d-9901-84c186af8f47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f65ac1f3-wtqvf\" (UID: \"701064e2-35dc-4b2d-9901-84c186af8f47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f65ac1f3-wtqvf" Oct 31 00:36:58.538922 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:58.538825 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8vrt\" (UniqueName: \"kubernetes.io/projected/701064e2-35dc-4b2d-9901-84c186af8f47-kube-api-access-l8vrt\") pod \"maas-api-revoke-sub-f65ac1f3-wtqvf\" (UID: \"701064e2-35dc-4b2d-9901-84c186af8f47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f65ac1f3-wtqvf" Oct 31 00:36:58.539437 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:58.539419 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/701064e2-35dc-4b2d-9901-84c186af8f47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f65ac1f3-wtqvf\" (UID: \"701064e2-35dc-4b2d-9901-84c186af8f47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f65ac1f3-wtqvf" Oct 31 00:36:58.541238 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:58.541214 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/701064e2-35dc-4b2d-9901-84c186af8f47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f65ac1f3-wtqvf\" (UID: \"701064e2-35dc-4b2d-9901-84c186af8f47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f65ac1f3-wtqvf" Oct 31 00:36:58.604418 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:58.604354 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8vrt\" (UniqueName: \"kubernetes.io/projected/701064e2-35dc-4b2d-9901-84c186af8f47-kube-api-access-l8vrt\") pod \"maas-api-revoke-sub-f65ac1f3-wtqvf\" (UID: \"701064e2-35dc-4b2d-9901-84c186af8f47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f65ac1f3-wtqvf" Oct 31 00:36:58.638739 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:58.638705 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f65ac1f3-wtqvf" Oct 31 00:36:59.012793 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:36:59.012761 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod701064e2_35dc_4b2d_9901_84c186af8f47.slice/crio-d990a37d0e240a9d8baae30947bfe9a322525aef025da8e5fb95510c69f71086 WatchSource:0}: Error finding container d990a37d0e240a9d8baae30947bfe9a322525aef025da8e5fb95510c69f71086: Status 404 returned error can't find the container with id d990a37d0e240a9d8baae30947bfe9a322525aef025da8e5fb95510c69f71086 Oct 31 00:36:59.017337 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:59.017318 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f65ac1f3-wtqvf"] Oct 31 00:36:59.153518 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:59.153487 5222 generic.go:356] "Generic (PLEG): container finished" podID="e22b4bd7-7950-40e6-88c5-5bfba6415954" containerID="db4a16dd001f7ce0260b1e9305eead3b0cc915cd26f013b4b420f8821388ea42" exitCode=0 Oct 31 00:36:59.153958 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:59.153540 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b182c63-kgc4k" event={"ID":"e22b4bd7-7950-40e6-88c5-5bfba6415954","Type":"ContainerDied","Data":"db4a16dd001f7ce0260b1e9305eead3b0cc915cd26f013b4b420f8821388ea42"} Oct 31 00:36:59.155128 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:36:59.155107 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f65ac1f3-wtqvf" event={"ID":"701064e2-35dc-4b2d-9901-84c186af8f47","Type":"ContainerStarted","Data":"d990a37d0e240a9d8baae30947bfe9a322525aef025da8e5fb95510c69f71086"} Oct 31 00:37:00.163489 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:00.163401 5222 generic.go:356] "Generic (PLEG): container finished" podID="701064e2-35dc-4b2d-9901-84c186af8f47" containerID="3ec14f207b23ab9014d528d3fa8d1ed205aea6c27989fe47789adb7191dcf7b0" exitCode=0 Oct 31 00:37:00.163489 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:00.163462 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f65ac1f3-wtqvf" event={"ID":"701064e2-35dc-4b2d-9901-84c186af8f47","Type":"ContainerDied","Data":"3ec14f207b23ab9014d528d3fa8d1ed205aea6c27989fe47789adb7191dcf7b0"} Oct 31 00:37:00.316402 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:00.316347 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b182c63-kgc4k" Oct 31 00:37:00.463165 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:00.463063 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e22b4bd7-7950-40e6-88c5-5bfba6415954-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e22b4bd7-7950-40e6-88c5-5bfba6415954-maas-api-xks-ca\") pod \"e22b4bd7-7950-40e6-88c5-5bfba6415954\" (UID: \"e22b4bd7-7950-40e6-88c5-5bfba6415954\") " Oct 31 00:37:00.463342 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:00.463233 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e22b4bd7-7950-40e6-88c5-5bfba6415954-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e22b4bd7-7950-40e6-88c5-5bfba6415954-maas-api-openshift-ca\") pod \"e22b4bd7-7950-40e6-88c5-5bfba6415954\" (UID: \"e22b4bd7-7950-40e6-88c5-5bfba6415954\") " Oct 31 00:37:00.463342 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:00.463276 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e22b4bd7-7950-40e6-88c5-5bfba6415954-kube-api-access-qq9q9\" (UniqueName: \"kubernetes.io/projected/e22b4bd7-7950-40e6-88c5-5bfba6415954-kube-api-access-qq9q9\") pod \"e22b4bd7-7950-40e6-88c5-5bfba6415954\" (UID: \"e22b4bd7-7950-40e6-88c5-5bfba6415954\") " Oct 31 00:37:00.463699 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:00.463669 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e22b4bd7-7950-40e6-88c5-5bfba6415954-maas-api-openshift-ca" pod "e22b4bd7-7950-40e6-88c5-5bfba6415954" (UID: "e22b4bd7-7950-40e6-88c5-5bfba6415954"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:37:00.465420 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:00.465344 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e22b4bd7-7950-40e6-88c5-5bfba6415954-maas-api-xks-ca" pod "e22b4bd7-7950-40e6-88c5-5bfba6415954" (UID: "e22b4bd7-7950-40e6-88c5-5bfba6415954"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:37:00.465545 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:00.465522 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e22b4bd7-7950-40e6-88c5-5bfba6415954-kube-api-access-qq9q9" pod "e22b4bd7-7950-40e6-88c5-5bfba6415954" (UID: "e22b4bd7-7950-40e6-88c5-5bfba6415954"). InnerVolumeSpecName "kube-api-access-qq9q9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:37:00.564342 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:00.564309 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e22b4bd7-7950-40e6-88c5-5bfba6415954-maas-api-openshift-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:00.564342 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:00.564338 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qq9q9\" (UniqueName: \"kubernetes.io/projected/e22b4bd7-7950-40e6-88c5-5bfba6415954-kube-api-access-qq9q9\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:00.564342 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:00.564348 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e22b4bd7-7950-40e6-88c5-5bfba6415954-maas-api-xks-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:01.171809 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:01.171776 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b182c63-kgc4k" Oct 31 00:37:01.172204 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:01.171817 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b182c63-kgc4k" event={"ID":"e22b4bd7-7950-40e6-88c5-5bfba6415954","Type":"ContainerDied","Data":"5c4743ba8d422053221aa1684361d1ac3cfe7b76d89b41499db678262ee80ab4"} Oct 31 00:37:01.172204 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:01.171868 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5c4743ba8d422053221aa1684361d1ac3cfe7b76d89b41499db678262ee80ab4" Oct 31 00:37:01.327680 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:01.327653 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f65ac1f3-wtqvf" Oct 31 00:37:01.476123 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:01.476027 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/701064e2-35dc-4b2d-9901-84c186af8f47-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/701064e2-35dc-4b2d-9901-84c186af8f47-maas-api-xks-ca\") pod \"701064e2-35dc-4b2d-9901-84c186af8f47\" (UID: \"701064e2-35dc-4b2d-9901-84c186af8f47\") " Oct 31 00:37:01.476123 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:01.476087 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/701064e2-35dc-4b2d-9901-84c186af8f47-kube-api-access-l8vrt\" (UniqueName: \"kubernetes.io/projected/701064e2-35dc-4b2d-9901-84c186af8f47-kube-api-access-l8vrt\") pod \"701064e2-35dc-4b2d-9901-84c186af8f47\" (UID: \"701064e2-35dc-4b2d-9901-84c186af8f47\") " Oct 31 00:37:01.476400 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:01.476166 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/701064e2-35dc-4b2d-9901-84c186af8f47-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/701064e2-35dc-4b2d-9901-84c186af8f47-maas-api-openshift-ca\") pod \"701064e2-35dc-4b2d-9901-84c186af8f47\" (UID: \"701064e2-35dc-4b2d-9901-84c186af8f47\") " Oct 31 00:37:01.476620 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:01.476582 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/701064e2-35dc-4b2d-9901-84c186af8f47-maas-api-openshift-ca" pod "701064e2-35dc-4b2d-9901-84c186af8f47" (UID: "701064e2-35dc-4b2d-9901-84c186af8f47"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:37:01.476822 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:01.476803 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/701064e2-35dc-4b2d-9901-84c186af8f47-maas-api-openshift-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:01.478556 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:01.478525 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/701064e2-35dc-4b2d-9901-84c186af8f47-maas-api-xks-ca" pod "701064e2-35dc-4b2d-9901-84c186af8f47" (UID: "701064e2-35dc-4b2d-9901-84c186af8f47"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:37:01.478944 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:01.478925 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/701064e2-35dc-4b2d-9901-84c186af8f47-kube-api-access-l8vrt" pod "701064e2-35dc-4b2d-9901-84c186af8f47" (UID: "701064e2-35dc-4b2d-9901-84c186af8f47"). InnerVolumeSpecName "kube-api-access-l8vrt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:37:01.578153 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:01.578116 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/701064e2-35dc-4b2d-9901-84c186af8f47-maas-api-xks-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:01.578153 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:01.578153 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8vrt\" (UniqueName: \"kubernetes.io/projected/701064e2-35dc-4b2d-9901-84c186af8f47-kube-api-access-l8vrt\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:02.051564 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:02.051529 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-701a0bb8-l4ld8"] Oct 31 00:37:02.103626 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:02.103577 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-701a0bb8-l4ld8"] Oct 31 00:37:02.149795 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:02.149762 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef147e9e-tzz8g"] Oct 31 00:37:02.150354 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:02.150341 5222 cpu_manager.go:414] "containerMap: removing container" podUID="701064e2-35dc-4b2d-9901-84c186af8f47" containerName="revoke-keys" Oct 31 00:37:02.150428 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:02.150355 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="701064e2-35dc-4b2d-9901-84c186af8f47" containerName="revoke-keys" Oct 31 00:37:02.150428 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:02.150399 5222 cpu_manager.go:414] "containerMap: removing container" podUID="e22b4bd7-7950-40e6-88c5-5bfba6415954" containerName="revoke-keys" Oct 31 00:37:02.150428 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:02.150407 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e22b4bd7-7950-40e6-88c5-5bfba6415954" containerName="revoke-keys" Oct 31 00:37:02.150518 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:02.150501 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="701064e2-35dc-4b2d-9901-84c186af8f47" containerName="revoke-keys" Oct 31 00:37:02.150518 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:02.150514 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="e22b4bd7-7950-40e6-88c5-5bfba6415954" containerName="revoke-keys" Oct 31 00:37:02.154317 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:02.154298 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef147e9e-tzz8g" Oct 31 00:37:02.181362 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:02.181324 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f65ac1f3-wtqvf" event={"ID":"701064e2-35dc-4b2d-9901-84c186af8f47","Type":"ContainerDied","Data":"d990a37d0e240a9d8baae30947bfe9a322525aef025da8e5fb95510c69f71086"} Oct 31 00:37:02.181763 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:02.181338 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f65ac1f3-wtqvf" Oct 31 00:37:02.181763 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:02.181402 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d990a37d0e240a9d8baae30947bfe9a322525aef025da8e5fb95510c69f71086" Oct 31 00:37:02.208446 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:02.208402 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef147e9e-tzz8g"] Oct 31 00:37:02.286740 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:02.286704 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a7a7936-2833-4bf8-80d6-dbb269954605-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef147e9e-tzz8g\" (UID: \"2a7a7936-2833-4bf8-80d6-dbb269954605\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef147e9e-tzz8g" Oct 31 00:37:02.286914 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:02.286797 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a7a7936-2833-4bf8-80d6-dbb269954605-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef147e9e-tzz8g\" (UID: \"2a7a7936-2833-4bf8-80d6-dbb269954605\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef147e9e-tzz8g" Oct 31 00:37:02.286914 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:02.286903 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlf8p\" (UniqueName: \"kubernetes.io/projected/2a7a7936-2833-4bf8-80d6-dbb269954605-kube-api-access-vlf8p\") pod \"maas-api-revoke-sub-ef147e9e-tzz8g\" (UID: \"2a7a7936-2833-4bf8-80d6-dbb269954605\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef147e9e-tzz8g" Oct 31 00:37:02.388170 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:02.388079 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlf8p\" (UniqueName: \"kubernetes.io/projected/2a7a7936-2833-4bf8-80d6-dbb269954605-kube-api-access-vlf8p\") pod \"maas-api-revoke-sub-ef147e9e-tzz8g\" (UID: \"2a7a7936-2833-4bf8-80d6-dbb269954605\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef147e9e-tzz8g" Oct 31 00:37:02.388319 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:02.388232 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a7a7936-2833-4bf8-80d6-dbb269954605-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef147e9e-tzz8g\" (UID: \"2a7a7936-2833-4bf8-80d6-dbb269954605\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef147e9e-tzz8g" Oct 31 00:37:02.388319 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:02.388268 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a7a7936-2833-4bf8-80d6-dbb269954605-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef147e9e-tzz8g\" (UID: \"2a7a7936-2833-4bf8-80d6-dbb269954605\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef147e9e-tzz8g" Oct 31 00:37:02.389074 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:02.389001 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a7a7936-2833-4bf8-80d6-dbb269954605-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef147e9e-tzz8g\" (UID: \"2a7a7936-2833-4bf8-80d6-dbb269954605\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef147e9e-tzz8g" Oct 31 00:37:02.390955 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:02.390925 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a7a7936-2833-4bf8-80d6-dbb269954605-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef147e9e-tzz8g\" (UID: \"2a7a7936-2833-4bf8-80d6-dbb269954605\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef147e9e-tzz8g" Oct 31 00:37:02.451334 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:02.451298 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlf8p\" (UniqueName: \"kubernetes.io/projected/2a7a7936-2833-4bf8-80d6-dbb269954605-kube-api-access-vlf8p\") pod \"maas-api-revoke-sub-ef147e9e-tzz8g\" (UID: \"2a7a7936-2833-4bf8-80d6-dbb269954605\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef147e9e-tzz8g" Oct 31 00:37:02.468021 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:02.467991 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef147e9e-tzz8g" Oct 31 00:37:02.845097 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:37:02.845068 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a7a7936_2833_4bf8_80d6_dbb269954605.slice/crio-a0e131b6f7877c18867b194659708df83a257f1d2fa813fb4c69a84e02df8c1c WatchSource:0}: Error finding container a0e131b6f7877c18867b194659708df83a257f1d2fa813fb4c69a84e02df8c1c: Status 404 returned error can't find the container with id a0e131b6f7877c18867b194659708df83a257f1d2fa813fb4c69a84e02df8c1c Oct 31 00:37:02.849141 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:02.849118 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef147e9e-tzz8g"] Oct 31 00:37:03.051460 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:03.051416 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37870f8f-9mrhh"] Oct 31 00:37:03.055063 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:03.055037 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37870f8f-9mrhh"] Oct 31 00:37:03.190853 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:03.190815 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef147e9e-tzz8g" event={"ID":"2a7a7936-2833-4bf8-80d6-dbb269954605","Type":"ContainerStarted","Data":"7bea0d31f78cfe6d4e92d4d5cd5d90b7cc161c7f477bc2cbc5dc18b172b1cf28"} Oct 31 00:37:03.191364 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:03.190911 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef147e9e-tzz8g" event={"ID":"2a7a7936-2833-4bf8-80d6-dbb269954605","Type":"ContainerStarted","Data":"a0e131b6f7877c18867b194659708df83a257f1d2fa813fb4c69a84e02df8c1c"} Oct 31 00:37:03.312826 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:03.312757 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef147e9e-tzz8g" podStartSLOduration=1.312740915 podStartE2EDuration="1.312740915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:37:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:37:03.308921023 +0000 UTC m=+1315.780090772" watchObservedRunningTime="2026-10-31 00:37:03.312740915 +0000 UTC m=+1315.783910675" Oct 31 00:37:03.657765 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:03.657730 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b8ba874-e544-43e0-a229-3fb9c78f57e8" path="/var/lib/kubelet/pods/2b8ba874-e544-43e0-a229-3fb9c78f57e8/volumes" Oct 31 00:37:03.658182 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:03.658168 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5f79f3d-1edc-4406-94bc-0785f51d9cb8" path="/var/lib/kubelet/pods/d5f79f3d-1edc-4406-94bc-0785f51d9cb8/volumes" Oct 31 00:37:04.055951 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:04.055912 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b5d5936-5dgtj"] Oct 31 00:37:04.106341 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:04.106313 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b5d5936-5dgtj"] Oct 31 00:37:04.201119 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:04.201081 5222 generic.go:356] "Generic (PLEG): container finished" podID="2a7a7936-2833-4bf8-80d6-dbb269954605" containerID="7bea0d31f78cfe6d4e92d4d5cd5d90b7cc161c7f477bc2cbc5dc18b172b1cf28" exitCode=0 Oct 31 00:37:04.201710 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:04.201175 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef147e9e-tzz8g" event={"ID":"2a7a7936-2833-4bf8-80d6-dbb269954605","Type":"ContainerDied","Data":"7bea0d31f78cfe6d4e92d4d5cd5d90b7cc161c7f477bc2cbc5dc18b172b1cf28"} Oct 31 00:37:05.373537 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:05.373510 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef147e9e-tzz8g" Oct 31 00:37:05.425060 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:05.425026 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a7a7936-2833-4bf8-80d6-dbb269954605-kube-api-access-vlf8p\" (UniqueName: \"kubernetes.io/projected/2a7a7936-2833-4bf8-80d6-dbb269954605-kube-api-access-vlf8p\") pod \"2a7a7936-2833-4bf8-80d6-dbb269954605\" (UID: \"2a7a7936-2833-4bf8-80d6-dbb269954605\") " Oct 31 00:37:05.425249 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:05.425152 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a7a7936-2833-4bf8-80d6-dbb269954605-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a7a7936-2833-4bf8-80d6-dbb269954605-maas-api-xks-ca\") pod \"2a7a7936-2833-4bf8-80d6-dbb269954605\" (UID: \"2a7a7936-2833-4bf8-80d6-dbb269954605\") " Oct 31 00:37:05.425249 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:05.425184 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a7a7936-2833-4bf8-80d6-dbb269954605-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a7a7936-2833-4bf8-80d6-dbb269954605-maas-api-openshift-ca\") pod \"2a7a7936-2833-4bf8-80d6-dbb269954605\" (UID: \"2a7a7936-2833-4bf8-80d6-dbb269954605\") " Oct 31 00:37:05.425659 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:05.425627 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a7a7936-2833-4bf8-80d6-dbb269954605-maas-api-openshift-ca" pod "2a7a7936-2833-4bf8-80d6-dbb269954605" (UID: "2a7a7936-2833-4bf8-80d6-dbb269954605"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:37:05.427538 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:05.427508 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a7a7936-2833-4bf8-80d6-dbb269954605-maas-api-xks-ca" pod "2a7a7936-2833-4bf8-80d6-dbb269954605" (UID: "2a7a7936-2833-4bf8-80d6-dbb269954605"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:37:05.427642 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:05.427600 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a7a7936-2833-4bf8-80d6-dbb269954605-kube-api-access-vlf8p" pod "2a7a7936-2833-4bf8-80d6-dbb269954605" (UID: "2a7a7936-2833-4bf8-80d6-dbb269954605"). InnerVolumeSpecName "kube-api-access-vlf8p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:37:05.526446 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:05.526411 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a7a7936-2833-4bf8-80d6-dbb269954605-maas-api-openshift-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:05.526446 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:05.526443 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlf8p\" (UniqueName: \"kubernetes.io/projected/2a7a7936-2833-4bf8-80d6-dbb269954605-kube-api-access-vlf8p\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:05.526446 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:05.526453 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a7a7936-2833-4bf8-80d6-dbb269954605-maas-api-xks-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:05.656656 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:05.656620 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37a9c8d6-0dc8-485a-82e5-ef26bb3785a0" path="/var/lib/kubelet/pods/37a9c8d6-0dc8-485a-82e5-ef26bb3785a0/volumes" Oct 31 00:37:05.891136 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:37:05.891038 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 00:37:06.218427 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:06.218323 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef147e9e-tzz8g" Oct 31 00:37:06.218427 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:06.218345 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef147e9e-tzz8g" event={"ID":"2a7a7936-2833-4bf8-80d6-dbb269954605","Type":"ContainerDied","Data":"a0e131b6f7877c18867b194659708df83a257f1d2fa813fb4c69a84e02df8c1c"} Oct 31 00:37:06.218611 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:06.218438 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a0e131b6f7877c18867b194659708df83a257f1d2fa813fb4c69a84e02df8c1c" Oct 31 00:37:07.057495 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:07.057461 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4b13d6e-5mvsk"] Oct 31 00:37:07.108499 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:07.108456 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4b13d6e-5mvsk"] Oct 31 00:37:07.529427 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:07.529390 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c09e604-cxspn"] Oct 31 00:37:07.530000 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:07.529982 5222 cpu_manager.go:414] "containerMap: removing container" podUID="2a7a7936-2833-4bf8-80d6-dbb269954605" containerName="revoke-keys" Oct 31 00:37:07.530087 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:07.530003 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a7a7936-2833-4bf8-80d6-dbb269954605" containerName="revoke-keys" Oct 31 00:37:07.530172 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:07.530143 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a7a7936-2833-4bf8-80d6-dbb269954605" containerName="revoke-keys" Oct 31 00:37:07.533962 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:07.533940 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c09e604-cxspn" Oct 31 00:37:07.550456 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:07.550431 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96e468eb-c292-4249-b748-b36cb86a4538-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7c09e604-cxspn\" (UID: \"96e468eb-c292-4249-b748-b36cb86a4538\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c09e604-cxspn" Oct 31 00:37:07.550569 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:07.550463 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w59vf\" (UniqueName: \"kubernetes.io/projected/96e468eb-c292-4249-b748-b36cb86a4538-kube-api-access-w59vf\") pod \"maas-api-revoke-sub-7c09e604-cxspn\" (UID: \"96e468eb-c292-4249-b748-b36cb86a4538\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c09e604-cxspn" Oct 31 00:37:07.550662 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:07.550642 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96e468eb-c292-4249-b748-b36cb86a4538-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7c09e604-cxspn\" (UID: \"96e468eb-c292-4249-b748-b36cb86a4538\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c09e604-cxspn" Oct 31 00:37:07.587437 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:07.587326 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c09e604-cxspn"] Oct 31 00:37:07.634286 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:07.634251 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:37:07.634286 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:07.634274 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:37:07.634286 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:07.634274 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfz9z\"" Oct 31 00:37:07.634286 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:07.634274 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:37:07.651673 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:07.651632 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96e468eb-c292-4249-b748-b36cb86a4538-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7c09e604-cxspn\" (UID: \"96e468eb-c292-4249-b748-b36cb86a4538\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c09e604-cxspn" Oct 31 00:37:07.651673 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:07.651674 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w59vf\" (UniqueName: \"kubernetes.io/projected/96e468eb-c292-4249-b748-b36cb86a4538-kube-api-access-w59vf\") pod \"maas-api-revoke-sub-7c09e604-cxspn\" (UID: \"96e468eb-c292-4249-b748-b36cb86a4538\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c09e604-cxspn" Oct 31 00:37:07.651918 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:07.651825 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96e468eb-c292-4249-b748-b36cb86a4538-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7c09e604-cxspn\" (UID: \"96e468eb-c292-4249-b748-b36cb86a4538\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c09e604-cxspn" Oct 31 00:37:07.652786 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:07.652750 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96e468eb-c292-4249-b748-b36cb86a4538-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7c09e604-cxspn\" (UID: \"96e468eb-c292-4249-b748-b36cb86a4538\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c09e604-cxspn" Oct 31 00:37:07.654528 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:07.654499 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96e468eb-c292-4249-b748-b36cb86a4538-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7c09e604-cxspn\" (UID: \"96e468eb-c292-4249-b748-b36cb86a4538\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c09e604-cxspn" Oct 31 00:37:07.659496 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:07.659466 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d25ee3cb-5017-4649-8f50-c9527f4c9dd9" path="/var/lib/kubelet/pods/d25ee3cb-5017-4649-8f50-c9527f4c9dd9/volumes" Oct 31 00:37:07.717309 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:07.717270 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w59vf\" (UniqueName: \"kubernetes.io/projected/96e468eb-c292-4249-b748-b36cb86a4538-kube-api-access-w59vf\") pod \"maas-api-revoke-sub-7c09e604-cxspn\" (UID: \"96e468eb-c292-4249-b748-b36cb86a4538\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c09e604-cxspn" Oct 31 00:37:07.847733 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:07.847633 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c09e604-cxspn" Oct 31 00:37:07.852283 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:07.851869 5222 scope.go:122] "RemoveContainer" containerID="eedbfc09916da2e38df1cf93b0fca2c7ac63cdb446011c9fc7b6c9be82702f10" Oct 31 00:37:07.866452 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:07.866427 5222 scope.go:122] "RemoveContainer" containerID="035db8436c66d133a6be99d922a2cbc3ced94d4ee9e87fa0c821f7d1586de576" Oct 31 00:37:08.225809 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:37:08.225767 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96e468eb_c292_4249_b748_b36cb86a4538.slice/crio-b0599b7829bb92d7731d3c0248ecdbe5da90e935386b9f291bea20b95582dd52 WatchSource:0}: Error finding container b0599b7829bb92d7731d3c0248ecdbe5da90e935386b9f291bea20b95582dd52: Status 404 returned error can't find the container with id b0599b7829bb92d7731d3c0248ecdbe5da90e935386b9f291bea20b95582dd52 Oct 31 00:37:08.229840 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:08.229814 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c09e604-cxspn"] Oct 31 00:37:09.241840 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:09.241808 5222 generic.go:356] "Generic (PLEG): container finished" podID="96e468eb-c292-4249-b748-b36cb86a4538" containerID="4f1952dd4e40840f355fdfc55a22b546e7aa7d917066619051778f385e2a87cb" exitCode=0 Oct 31 00:37:09.242217 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:09.241853 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c09e604-cxspn" event={"ID":"96e468eb-c292-4249-b748-b36cb86a4538","Type":"ContainerDied","Data":"4f1952dd4e40840f355fdfc55a22b546e7aa7d917066619051778f385e2a87cb"} Oct 31 00:37:09.242217 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:09.241918 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c09e604-cxspn" event={"ID":"96e468eb-c292-4249-b748-b36cb86a4538","Type":"ContainerStarted","Data":"b0599b7829bb92d7731d3c0248ecdbe5da90e935386b9f291bea20b95582dd52"} Oct 31 00:37:10.404198 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:10.404169 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c09e604-cxspn" Oct 31 00:37:10.496464 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:10.496421 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96e468eb-c292-4249-b748-b36cb86a4538-kube-api-access-w59vf\" (UniqueName: \"kubernetes.io/projected/96e468eb-c292-4249-b748-b36cb86a4538-kube-api-access-w59vf\") pod \"96e468eb-c292-4249-b748-b36cb86a4538\" (UID: \"96e468eb-c292-4249-b748-b36cb86a4538\") " Oct 31 00:37:10.496619 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:10.496595 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96e468eb-c292-4249-b748-b36cb86a4538-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96e468eb-c292-4249-b748-b36cb86a4538-maas-api-openshift-ca\") pod \"96e468eb-c292-4249-b748-b36cb86a4538\" (UID: \"96e468eb-c292-4249-b748-b36cb86a4538\") " Oct 31 00:37:10.496683 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:10.496655 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96e468eb-c292-4249-b748-b36cb86a4538-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96e468eb-c292-4249-b748-b36cb86a4538-maas-api-xks-ca\") pod \"96e468eb-c292-4249-b748-b36cb86a4538\" (UID: \"96e468eb-c292-4249-b748-b36cb86a4538\") " Oct 31 00:37:10.497029 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:10.496987 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96e468eb-c292-4249-b748-b36cb86a4538-maas-api-openshift-ca" pod "96e468eb-c292-4249-b748-b36cb86a4538" (UID: "96e468eb-c292-4249-b748-b36cb86a4538"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:37:10.498733 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:10.498700 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96e468eb-c292-4249-b748-b36cb86a4538-kube-api-access-w59vf" pod "96e468eb-c292-4249-b748-b36cb86a4538" (UID: "96e468eb-c292-4249-b748-b36cb86a4538"). InnerVolumeSpecName "kube-api-access-w59vf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:37:10.498903 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:10.498883 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96e468eb-c292-4249-b748-b36cb86a4538-maas-api-xks-ca" pod "96e468eb-c292-4249-b748-b36cb86a4538" (UID: "96e468eb-c292-4249-b748-b36cb86a4538"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:37:10.598119 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:10.598084 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96e468eb-c292-4249-b748-b36cb86a4538-maas-api-openshift-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:10.598119 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:10.598117 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96e468eb-c292-4249-b748-b36cb86a4538-maas-api-xks-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:10.598305 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:10.598129 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w59vf\" (UniqueName: \"kubernetes.io/projected/96e468eb-c292-4249-b748-b36cb86a4538-kube-api-access-w59vf\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:11.213071 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:11.213034 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2fe61f62-xkstq"] Oct 31 00:37:11.213582 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:11.213568 5222 cpu_manager.go:414] "containerMap: removing container" podUID="96e468eb-c292-4249-b748-b36cb86a4538" containerName="revoke-keys" Oct 31 00:37:11.213635 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:11.213583 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96e468eb-c292-4249-b748-b36cb86a4538" containerName="revoke-keys" Oct 31 00:37:11.213709 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:11.213700 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="96e468eb-c292-4249-b748-b36cb86a4538" containerName="revoke-keys" Oct 31 00:37:11.217670 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:11.217650 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fe61f62-xkstq" Oct 31 00:37:11.258335 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:11.258297 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c09e604-cxspn" Oct 31 00:37:11.258335 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:11.258320 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c09e604-cxspn" event={"ID":"96e468eb-c292-4249-b748-b36cb86a4538","Type":"ContainerDied","Data":"b0599b7829bb92d7731d3c0248ecdbe5da90e935386b9f291bea20b95582dd52"} Oct 31 00:37:11.258584 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:11.258415 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b0599b7829bb92d7731d3c0248ecdbe5da90e935386b9f291bea20b95582dd52" Oct 31 00:37:11.269271 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:11.269240 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2fe61f62-xkstq"] Oct 31 00:37:11.307287 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:11.307252 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e08c9285-911f-4816-a7e4-b0c0f48269ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2fe61f62-xkstq\" (UID: \"e08c9285-911f-4816-a7e4-b0c0f48269ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fe61f62-xkstq" Oct 31 00:37:11.307514 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:11.307315 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tf5gk\" (UniqueName: \"kubernetes.io/projected/e08c9285-911f-4816-a7e4-b0c0f48269ac-kube-api-access-tf5gk\") pod \"maas-api-revoke-sub-2fe61f62-xkstq\" (UID: \"e08c9285-911f-4816-a7e4-b0c0f48269ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fe61f62-xkstq" Oct 31 00:37:11.307514 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:11.307499 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e08c9285-911f-4816-a7e4-b0c0f48269ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2fe61f62-xkstq\" (UID: \"e08c9285-911f-4816-a7e4-b0c0f48269ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fe61f62-xkstq" Oct 31 00:37:11.408419 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:11.408386 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e08c9285-911f-4816-a7e4-b0c0f48269ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2fe61f62-xkstq\" (UID: \"e08c9285-911f-4816-a7e4-b0c0f48269ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fe61f62-xkstq" Oct 31 00:37:11.408951 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:11.408447 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tf5gk\" (UniqueName: \"kubernetes.io/projected/e08c9285-911f-4816-a7e4-b0c0f48269ac-kube-api-access-tf5gk\") pod \"maas-api-revoke-sub-2fe61f62-xkstq\" (UID: \"e08c9285-911f-4816-a7e4-b0c0f48269ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fe61f62-xkstq" Oct 31 00:37:11.408951 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:11.408539 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e08c9285-911f-4816-a7e4-b0c0f48269ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2fe61f62-xkstq\" (UID: \"e08c9285-911f-4816-a7e4-b0c0f48269ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fe61f62-xkstq" Oct 31 00:37:11.409073 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:11.409012 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e08c9285-911f-4816-a7e4-b0c0f48269ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2fe61f62-xkstq\" (UID: \"e08c9285-911f-4816-a7e4-b0c0f48269ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fe61f62-xkstq" Oct 31 00:37:11.411146 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:11.411114 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e08c9285-911f-4816-a7e4-b0c0f48269ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2fe61f62-xkstq\" (UID: \"e08c9285-911f-4816-a7e4-b0c0f48269ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fe61f62-xkstq" Oct 31 00:37:11.471332 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:11.471247 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tf5gk\" (UniqueName: \"kubernetes.io/projected/e08c9285-911f-4816-a7e4-b0c0f48269ac-kube-api-access-tf5gk\") pod \"maas-api-revoke-sub-2fe61f62-xkstq\" (UID: \"e08c9285-911f-4816-a7e4-b0c0f48269ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fe61f62-xkstq" Oct 31 00:37:11.535083 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:11.535052 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fe61f62-xkstq" Oct 31 00:37:11.922273 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:11.922250 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2fe61f62-xkstq"] Oct 31 00:37:12.275130 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:12.275091 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fe61f62-xkstq" event={"ID":"e08c9285-911f-4816-a7e4-b0c0f48269ac","Type":"ContainerStarted","Data":"8e14a7ea384665198e5f727a580c5487d5c196db81faa6666cb0b4d1456f379d"} Oct 31 00:37:12.275332 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:12.275181 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fe61f62-xkstq" event={"ID":"e08c9285-911f-4816-a7e4-b0c0f48269ac","Type":"ContainerStarted","Data":"084ae5622d2ac6f57216ecac9ed8bdbe8f186bf4985f505319bebb24ae89f221"} Oct 31 00:37:12.394165 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:12.394098 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fe61f62-xkstq" podStartSLOduration=1.3940852160000001 podStartE2EDuration="1.394085216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:37:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:37:12.393281498 +0000 UTC m=+1324.864451248" watchObservedRunningTime="2026-10-31 00:37:12.394085216 +0000 UTC m=+1324.865254965" Oct 31 00:37:13.285254 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:13.285216 5222 generic.go:356] "Generic (PLEG): container finished" podID="e08c9285-911f-4816-a7e4-b0c0f48269ac" containerID="8e14a7ea384665198e5f727a580c5487d5c196db81faa6666cb0b4d1456f379d" exitCode=0 Oct 31 00:37:13.285723 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:13.285301 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fe61f62-xkstq" event={"ID":"e08c9285-911f-4816-a7e4-b0c0f48269ac","Type":"ContainerDied","Data":"8e14a7ea384665198e5f727a580c5487d5c196db81faa6666cb0b4d1456f379d"} Oct 31 00:37:13.881779 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:13.881742 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dcba2d2b-ddk75"] Oct 31 00:37:13.886054 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:13.886026 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcba2d2b-ddk75" Oct 31 00:37:13.943089 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:13.943055 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dcba2d2b-ddk75"] Oct 31 00:37:13.962843 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:13.962812 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5189d24-61c9-4669-a66c-4856d33e3584-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dcba2d2b-ddk75\" (UID: \"c5189d24-61c9-4669-a66c-4856d33e3584\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcba2d2b-ddk75" Oct 31 00:37:13.963010 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:13.962874 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8kc8\" (UniqueName: \"kubernetes.io/projected/c5189d24-61c9-4669-a66c-4856d33e3584-kube-api-access-d8kc8\") pod \"maas-api-revoke-sub-dcba2d2b-ddk75\" (UID: \"c5189d24-61c9-4669-a66c-4856d33e3584\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcba2d2b-ddk75" Oct 31 00:37:13.963224 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:13.963200 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5189d24-61c9-4669-a66c-4856d33e3584-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dcba2d2b-ddk75\" (UID: \"c5189d24-61c9-4669-a66c-4856d33e3584\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcba2d2b-ddk75" Oct 31 00:37:14.064814 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:14.064774 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5189d24-61c9-4669-a66c-4856d33e3584-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dcba2d2b-ddk75\" (UID: \"c5189d24-61c9-4669-a66c-4856d33e3584\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcba2d2b-ddk75" Oct 31 00:37:14.065024 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:14.064838 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5189d24-61c9-4669-a66c-4856d33e3584-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dcba2d2b-ddk75\" (UID: \"c5189d24-61c9-4669-a66c-4856d33e3584\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcba2d2b-ddk75" Oct 31 00:37:14.065024 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:14.064864 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8kc8\" (UniqueName: \"kubernetes.io/projected/c5189d24-61c9-4669-a66c-4856d33e3584-kube-api-access-d8kc8\") pod \"maas-api-revoke-sub-dcba2d2b-ddk75\" (UID: \"c5189d24-61c9-4669-a66c-4856d33e3584\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcba2d2b-ddk75" Oct 31 00:37:14.065588 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:14.065567 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5189d24-61c9-4669-a66c-4856d33e3584-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dcba2d2b-ddk75\" (UID: \"c5189d24-61c9-4669-a66c-4856d33e3584\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcba2d2b-ddk75" Oct 31 00:37:14.067546 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:14.067527 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5189d24-61c9-4669-a66c-4856d33e3584-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dcba2d2b-ddk75\" (UID: \"c5189d24-61c9-4669-a66c-4856d33e3584\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcba2d2b-ddk75" Oct 31 00:37:14.128880 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:14.128846 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8kc8\" (UniqueName: \"kubernetes.io/projected/c5189d24-61c9-4669-a66c-4856d33e3584-kube-api-access-d8kc8\") pod \"maas-api-revoke-sub-dcba2d2b-ddk75\" (UID: \"c5189d24-61c9-4669-a66c-4856d33e3584\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcba2d2b-ddk75" Oct 31 00:37:14.199636 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:14.199551 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcba2d2b-ddk75" Oct 31 00:37:14.447902 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:14.447870 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fe61f62-xkstq" Oct 31 00:37:14.573871 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:14.573837 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e08c9285-911f-4816-a7e4-b0c0f48269ac-kube-api-access-tf5gk\" (UniqueName: \"kubernetes.io/projected/e08c9285-911f-4816-a7e4-b0c0f48269ac-kube-api-access-tf5gk\") pod \"e08c9285-911f-4816-a7e4-b0c0f48269ac\" (UID: \"e08c9285-911f-4816-a7e4-b0c0f48269ac\") " Oct 31 00:37:14.574040 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:14.573997 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e08c9285-911f-4816-a7e4-b0c0f48269ac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e08c9285-911f-4816-a7e4-b0c0f48269ac-maas-api-openshift-ca\") pod \"e08c9285-911f-4816-a7e4-b0c0f48269ac\" (UID: \"e08c9285-911f-4816-a7e4-b0c0f48269ac\") " Oct 31 00:37:14.574040 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:14.574034 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e08c9285-911f-4816-a7e4-b0c0f48269ac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e08c9285-911f-4816-a7e4-b0c0f48269ac-maas-api-xks-ca\") pod \"e08c9285-911f-4816-a7e4-b0c0f48269ac\" (UID: \"e08c9285-911f-4816-a7e4-b0c0f48269ac\") " Oct 31 00:37:14.574856 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:14.574309 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e08c9285-911f-4816-a7e4-b0c0f48269ac-maas-api-openshift-ca" pod "e08c9285-911f-4816-a7e4-b0c0f48269ac" (UID: "e08c9285-911f-4816-a7e4-b0c0f48269ac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:37:14.574856 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:14.574819 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e08c9285-911f-4816-a7e4-b0c0f48269ac-maas-api-openshift-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:14.576691 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:14.576663 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e08c9285-911f-4816-a7e4-b0c0f48269ac-kube-api-access-tf5gk" pod "e08c9285-911f-4816-a7e4-b0c0f48269ac" (UID: "e08c9285-911f-4816-a7e4-b0c0f48269ac"). InnerVolumeSpecName "kube-api-access-tf5gk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:37:14.576972 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:14.576945 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e08c9285-911f-4816-a7e4-b0c0f48269ac-maas-api-xks-ca" pod "e08c9285-911f-4816-a7e4-b0c0f48269ac" (UID: "e08c9285-911f-4816-a7e4-b0c0f48269ac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:37:14.581362 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:37:14.581332 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5189d24_61c9_4669_a66c_4856d33e3584.slice/crio-9e38941a111c8a282cd4f7f116b3d3683ebbcf96979f4aee6d89b98c0fb441c4 WatchSource:0}: Error finding container 9e38941a111c8a282cd4f7f116b3d3683ebbcf96979f4aee6d89b98c0fb441c4: Status 404 returned error can't find the container with id 9e38941a111c8a282cd4f7f116b3d3683ebbcf96979f4aee6d89b98c0fb441c4 Oct 31 00:37:14.583857 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:14.583834 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dcba2d2b-ddk75"] Oct 31 00:37:14.675777 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:14.675738 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e08c9285-911f-4816-a7e4-b0c0f48269ac-maas-api-xks-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:14.675777 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:14.675769 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tf5gk\" (UniqueName: \"kubernetes.io/projected/e08c9285-911f-4816-a7e4-b0c0f48269ac-kube-api-access-tf5gk\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:15.057096 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:15.057060 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52695c5b-btcq4"] Oct 31 00:37:15.061389 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:15.061343 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52695c5b-btcq4"] Oct 31 00:37:15.306490 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:15.306458 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fe61f62-xkstq" Oct 31 00:37:15.306490 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:15.306467 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fe61f62-xkstq" event={"ID":"e08c9285-911f-4816-a7e4-b0c0f48269ac","Type":"ContainerDied","Data":"084ae5622d2ac6f57216ecac9ed8bdbe8f186bf4985f505319bebb24ae89f221"} Oct 31 00:37:15.306752 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:15.306555 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="084ae5622d2ac6f57216ecac9ed8bdbe8f186bf4985f505319bebb24ae89f221" Oct 31 00:37:15.308476 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:15.308446 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcba2d2b-ddk75" event={"ID":"c5189d24-61c9-4669-a66c-4856d33e3584","Type":"ContainerStarted","Data":"a06f6c61b4701cf00861c598f717ea535f5d6a779adb3c8dffee138fc1ab61de"} Oct 31 00:37:15.308673 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:15.308643 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcba2d2b-ddk75" event={"ID":"c5189d24-61c9-4669-a66c-4856d33e3584","Type":"ContainerStarted","Data":"9e38941a111c8a282cd4f7f116b3d3683ebbcf96979f4aee6d89b98c0fb441c4"} Oct 31 00:37:15.429289 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:15.429235 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcba2d2b-ddk75" podStartSLOduration=2.42922019 podStartE2EDuration="2.42922019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:37:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:37:15.426164442 +0000 UTC m=+1327.897334196" watchObservedRunningTime="2026-10-31 00:37:15.42922019 +0000 UTC m=+1327.900389964" Oct 31 00:37:15.657034 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:15.656953 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ab68e7d-5046-4ed2-9ec7-f8270d3f478b" path="/var/lib/kubelet/pods/8ab68e7d-5046-4ed2-9ec7-f8270d3f478b/volumes" Oct 31 00:37:16.317986 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:16.317951 5222 generic.go:356] "Generic (PLEG): container finished" podID="c5189d24-61c9-4669-a66c-4856d33e3584" containerID="a06f6c61b4701cf00861c598f717ea535f5d6a779adb3c8dffee138fc1ab61de" exitCode=0 Oct 31 00:37:16.318172 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:16.318037 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcba2d2b-ddk75" event={"ID":"c5189d24-61c9-4669-a66c-4856d33e3584","Type":"ContainerDied","Data":"a06f6c61b4701cf00861c598f717ea535f5d6a779adb3c8dffee138fc1ab61de"} Oct 31 00:37:17.484579 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:17.484546 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcba2d2b-ddk75" Oct 31 00:37:17.616068 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:17.615969 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5189d24-61c9-4669-a66c-4856d33e3584-kube-api-access-d8kc8\" (UniqueName: \"kubernetes.io/projected/c5189d24-61c9-4669-a66c-4856d33e3584-kube-api-access-d8kc8\") pod \"c5189d24-61c9-4669-a66c-4856d33e3584\" (UID: \"c5189d24-61c9-4669-a66c-4856d33e3584\") " Oct 31 00:37:17.616068 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:17.616064 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5189d24-61c9-4669-a66c-4856d33e3584-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5189d24-61c9-4669-a66c-4856d33e3584-maas-api-openshift-ca\") pod \"c5189d24-61c9-4669-a66c-4856d33e3584\" (UID: \"c5189d24-61c9-4669-a66c-4856d33e3584\") " Oct 31 00:37:17.616300 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:17.616202 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5189d24-61c9-4669-a66c-4856d33e3584-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5189d24-61c9-4669-a66c-4856d33e3584-maas-api-xks-ca\") pod \"c5189d24-61c9-4669-a66c-4856d33e3584\" (UID: \"c5189d24-61c9-4669-a66c-4856d33e3584\") " Oct 31 00:37:17.616475 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:17.616448 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5189d24-61c9-4669-a66c-4856d33e3584-maas-api-openshift-ca" pod "c5189d24-61c9-4669-a66c-4856d33e3584" (UID: "c5189d24-61c9-4669-a66c-4856d33e3584"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:37:17.616858 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:17.616832 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5189d24-61c9-4669-a66c-4856d33e3584-maas-api-openshift-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:17.618467 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:17.618399 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5189d24-61c9-4669-a66c-4856d33e3584-kube-api-access-d8kc8" pod "c5189d24-61c9-4669-a66c-4856d33e3584" (UID: "c5189d24-61c9-4669-a66c-4856d33e3584"). InnerVolumeSpecName "kube-api-access-d8kc8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:37:17.618467 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:17.618434 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5189d24-61c9-4669-a66c-4856d33e3584-maas-api-xks-ca" pod "c5189d24-61c9-4669-a66c-4856d33e3584" (UID: "c5189d24-61c9-4669-a66c-4856d33e3584"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:37:17.718533 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:17.718499 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8kc8\" (UniqueName: \"kubernetes.io/projected/c5189d24-61c9-4669-a66c-4856d33e3584-kube-api-access-d8kc8\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:17.718533 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:17.718528 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5189d24-61c9-4669-a66c-4856d33e3584-maas-api-xks-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:18.339061 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:18.338265 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcba2d2b-ddk75" event={"ID":"c5189d24-61c9-4669-a66c-4856d33e3584","Type":"ContainerDied","Data":"9e38941a111c8a282cd4f7f116b3d3683ebbcf96979f4aee6d89b98c0fb441c4"} Oct 31 00:37:18.339061 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:18.338328 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9e38941a111c8a282cd4f7f116b3d3683ebbcf96979f4aee6d89b98c0fb441c4" Oct 31 00:37:18.339061 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:18.338419 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcba2d2b-ddk75" Oct 31 00:37:19.568976 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:19.568943 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18e8cedf-txqlt"] Oct 31 00:37:19.569503 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:19.569489 5222 cpu_manager.go:414] "containerMap: removing container" podUID="c5189d24-61c9-4669-a66c-4856d33e3584" containerName="revoke-keys" Oct 31 00:37:19.569548 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:19.569504 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5189d24-61c9-4669-a66c-4856d33e3584" containerName="revoke-keys" Oct 31 00:37:19.569548 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:19.569531 5222 cpu_manager.go:414] "containerMap: removing container" podUID="e08c9285-911f-4816-a7e4-b0c0f48269ac" containerName="revoke-keys" Oct 31 00:37:19.569548 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:19.569536 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e08c9285-911f-4816-a7e4-b0c0f48269ac" containerName="revoke-keys" Oct 31 00:37:19.569648 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:19.569617 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5189d24-61c9-4669-a66c-4856d33e3584" containerName="revoke-keys" Oct 31 00:37:19.569648 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:19.569628 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="e08c9285-911f-4816-a7e4-b0c0f48269ac" containerName="revoke-keys" Oct 31 00:37:19.573415 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:19.573398 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18e8cedf-txqlt" Oct 31 00:37:19.627219 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:19.627179 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18e8cedf-txqlt"] Oct 31 00:37:19.627493 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:19.627471 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:37:19.673606 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:19.673573 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:37:19.673606 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:19.673594 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:37:19.673830 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:19.673636 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfz9z\"" Oct 31 00:37:19.745713 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:19.745680 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrw5z\" (UniqueName: \"kubernetes.io/projected/47f9711a-f60e-4fec-accd-a065ab4eb48a-kube-api-access-jrw5z\") pod \"maas-api-revoke-sub-18e8cedf-txqlt\" (UID: \"47f9711a-f60e-4fec-accd-a065ab4eb48a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18e8cedf-txqlt" Oct 31 00:37:19.745911 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:19.745749 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47f9711a-f60e-4fec-accd-a065ab4eb48a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18e8cedf-txqlt\" (UID: \"47f9711a-f60e-4fec-accd-a065ab4eb48a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18e8cedf-txqlt" Oct 31 00:37:19.745911 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:19.745868 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47f9711a-f60e-4fec-accd-a065ab4eb48a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18e8cedf-txqlt\" (UID: \"47f9711a-f60e-4fec-accd-a065ab4eb48a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18e8cedf-txqlt" Oct 31 00:37:19.847679 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:19.847585 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47f9711a-f60e-4fec-accd-a065ab4eb48a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18e8cedf-txqlt\" (UID: \"47f9711a-f60e-4fec-accd-a065ab4eb48a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18e8cedf-txqlt" Oct 31 00:37:19.847679 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:19.847632 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47f9711a-f60e-4fec-accd-a065ab4eb48a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18e8cedf-txqlt\" (UID: \"47f9711a-f60e-4fec-accd-a065ab4eb48a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18e8cedf-txqlt" Oct 31 00:37:19.847975 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:19.847946 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrw5z\" (UniqueName: \"kubernetes.io/projected/47f9711a-f60e-4fec-accd-a065ab4eb48a-kube-api-access-jrw5z\") pod \"maas-api-revoke-sub-18e8cedf-txqlt\" (UID: \"47f9711a-f60e-4fec-accd-a065ab4eb48a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18e8cedf-txqlt" Oct 31 00:37:19.848424 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:19.848360 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47f9711a-f60e-4fec-accd-a065ab4eb48a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18e8cedf-txqlt\" (UID: \"47f9711a-f60e-4fec-accd-a065ab4eb48a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18e8cedf-txqlt" Oct 31 00:37:19.850294 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:19.850266 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47f9711a-f60e-4fec-accd-a065ab4eb48a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18e8cedf-txqlt\" (UID: \"47f9711a-f60e-4fec-accd-a065ab4eb48a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18e8cedf-txqlt" Oct 31 00:37:19.911066 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:19.911027 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrw5z\" (UniqueName: \"kubernetes.io/projected/47f9711a-f60e-4fec-accd-a065ab4eb48a-kube-api-access-jrw5z\") pod \"maas-api-revoke-sub-18e8cedf-txqlt\" (UID: \"47f9711a-f60e-4fec-accd-a065ab4eb48a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18e8cedf-txqlt" Oct 31 00:37:20.054143 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:20.054108 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1019bddd-hdvj8"] Oct 31 00:37:20.057923 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:20.057901 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da56edec-wglq9"] Oct 31 00:37:20.061284 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:20.061267 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1019bddd-hdvj8"] Oct 31 00:37:20.064454 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:20.064427 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da56edec-wglq9"] Oct 31 00:37:20.186894 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:20.186807 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18e8cedf-txqlt" Oct 31 00:37:20.565244 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:37:20.565213 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47f9711a_f60e_4fec_accd_a065ab4eb48a.slice/crio-7297d6bb268e55d3d9e775d3c02e9bab5427aa05b8015c85862f71867a3c6b62 WatchSource:0}: Error finding container 7297d6bb268e55d3d9e775d3c02e9bab5427aa05b8015c85862f71867a3c6b62: Status 404 returned error can't find the container with id 7297d6bb268e55d3d9e775d3c02e9bab5427aa05b8015c85862f71867a3c6b62 Oct 31 00:37:20.568901 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:20.568878 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18e8cedf-txqlt"] Oct 31 00:37:21.363215 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:21.363168 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18e8cedf-txqlt" event={"ID":"47f9711a-f60e-4fec-accd-a065ab4eb48a","Type":"ContainerStarted","Data":"fb65a47e597744ff8b421cacceca6f8019eb67ae8a363e752994e6b2cd3c7785"} Oct 31 00:37:21.363626 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:21.363230 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18e8cedf-txqlt" event={"ID":"47f9711a-f60e-4fec-accd-a065ab4eb48a","Type":"ContainerStarted","Data":"7297d6bb268e55d3d9e775d3c02e9bab5427aa05b8015c85862f71867a3c6b62"} Oct 31 00:37:21.490359 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:21.490297 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18e8cedf-txqlt" podStartSLOduration=2.4902807080000002 podStartE2EDuration="2.490280708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:37:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:37:21.487730417 +0000 UTC m=+1333.958900165" watchObservedRunningTime="2026-10-31 00:37:21.490280708 +0000 UTC m=+1333.961450498" Oct 31 00:37:21.656968 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:21.656884 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6dfec126-5c72-4be9-ad1f-c605c63bea90" path="/var/lib/kubelet/pods/6dfec126-5c72-4be9-ad1f-c605c63bea90/volumes" Oct 31 00:37:21.657337 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:21.657324 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98010426-9a98-407e-ad55-7eeb3313b555" path="/var/lib/kubelet/pods/98010426-9a98-407e-ad55-7eeb3313b555/volumes" Oct 31 00:37:22.109086 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:22.109048 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55867ce2-z8zk8"] Oct 31 00:37:22.113048 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:22.113021 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55867ce2-z8zk8" Oct 31 00:37:22.183855 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:22.183820 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zk67v\" (UniqueName: \"kubernetes.io/projected/e8a5550a-5d20-4ec9-8272-dc539a00e96d-kube-api-access-zk67v\") pod \"maas-api-revoke-sub-55867ce2-z8zk8\" (UID: \"e8a5550a-5d20-4ec9-8272-dc539a00e96d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55867ce2-z8zk8" Oct 31 00:37:22.184042 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:22.183881 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8a5550a-5d20-4ec9-8272-dc539a00e96d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-55867ce2-z8zk8\" (UID: \"e8a5550a-5d20-4ec9-8272-dc539a00e96d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55867ce2-z8zk8" Oct 31 00:37:22.184042 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:22.184004 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8a5550a-5d20-4ec9-8272-dc539a00e96d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-55867ce2-z8zk8\" (UID: \"e8a5550a-5d20-4ec9-8272-dc539a00e96d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55867ce2-z8zk8" Oct 31 00:37:22.200969 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:22.200932 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55867ce2-z8zk8"] Oct 31 00:37:22.293425 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:22.287294 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zk67v\" (UniqueName: \"kubernetes.io/projected/e8a5550a-5d20-4ec9-8272-dc539a00e96d-kube-api-access-zk67v\") pod \"maas-api-revoke-sub-55867ce2-z8zk8\" (UID: \"e8a5550a-5d20-4ec9-8272-dc539a00e96d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55867ce2-z8zk8" Oct 31 00:37:22.293425 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:22.287473 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8a5550a-5d20-4ec9-8272-dc539a00e96d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-55867ce2-z8zk8\" (UID: \"e8a5550a-5d20-4ec9-8272-dc539a00e96d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55867ce2-z8zk8" Oct 31 00:37:22.293425 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:22.287550 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8a5550a-5d20-4ec9-8272-dc539a00e96d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-55867ce2-z8zk8\" (UID: \"e8a5550a-5d20-4ec9-8272-dc539a00e96d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55867ce2-z8zk8" Oct 31 00:37:22.293425 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:22.288870 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8a5550a-5d20-4ec9-8272-dc539a00e96d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-55867ce2-z8zk8\" (UID: \"e8a5550a-5d20-4ec9-8272-dc539a00e96d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55867ce2-z8zk8" Oct 31 00:37:22.293425 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:22.292059 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8a5550a-5d20-4ec9-8272-dc539a00e96d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-55867ce2-z8zk8\" (UID: \"e8a5550a-5d20-4ec9-8272-dc539a00e96d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55867ce2-z8zk8" Oct 31 00:37:22.355402 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:22.355338 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zk67v\" (UniqueName: \"kubernetes.io/projected/e8a5550a-5d20-4ec9-8272-dc539a00e96d-kube-api-access-zk67v\") pod \"maas-api-revoke-sub-55867ce2-z8zk8\" (UID: \"e8a5550a-5d20-4ec9-8272-dc539a00e96d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55867ce2-z8zk8" Oct 31 00:37:22.372695 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:22.372606 5222 generic.go:356] "Generic (PLEG): container finished" podID="47f9711a-f60e-4fec-accd-a065ab4eb48a" containerID="fb65a47e597744ff8b421cacceca6f8019eb67ae8a363e752994e6b2cd3c7785" exitCode=0 Oct 31 00:37:22.372695 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:22.372646 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18e8cedf-txqlt" event={"ID":"47f9711a-f60e-4fec-accd-a065ab4eb48a","Type":"ContainerDied","Data":"fb65a47e597744ff8b421cacceca6f8019eb67ae8a363e752994e6b2cd3c7785"} Oct 31 00:37:22.426628 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:22.426588 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55867ce2-z8zk8" Oct 31 00:37:22.805948 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:37:22.805904 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8a5550a_5d20_4ec9_8272_dc539a00e96d.slice/crio-b73c88b04b1f7dc77fbfaaf888d747ce41948f0d9d66e08cf2577ac2e5c74bc0 WatchSource:0}: Error finding container b73c88b04b1f7dc77fbfaaf888d747ce41948f0d9d66e08cf2577ac2e5c74bc0: Status 404 returned error can't find the container with id b73c88b04b1f7dc77fbfaaf888d747ce41948f0d9d66e08cf2577ac2e5c74bc0 Oct 31 00:37:22.808958 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:22.808927 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55867ce2-z8zk8"] Oct 31 00:37:23.381967 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:23.381924 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55867ce2-z8zk8" event={"ID":"e8a5550a-5d20-4ec9-8272-dc539a00e96d","Type":"ContainerStarted","Data":"d6b07d7835a4fcba7670fc1965a39e3b60bdc3f5f3e3a1aaee9e20f044c002ae"} Oct 31 00:37:23.382408 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:23.381993 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55867ce2-z8zk8" event={"ID":"e8a5550a-5d20-4ec9-8272-dc539a00e96d","Type":"ContainerStarted","Data":"b73c88b04b1f7dc77fbfaaf888d747ce41948f0d9d66e08cf2577ac2e5c74bc0"} Oct 31 00:37:23.521949 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:23.521914 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a92e31b-bnxkp"] Oct 31 00:37:23.526690 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:23.526661 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a92e31b-bnxkp" Oct 31 00:37:23.577996 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:23.577946 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55867ce2-z8zk8" podStartSLOduration=1.577929302 podStartE2EDuration="1.577929302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:37:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:37:23.567051455 +0000 UTC m=+1336.038221203" watchObservedRunningTime="2026-10-31 00:37:23.577929302 +0000 UTC m=+1336.049099128" Oct 31 00:37:23.580320 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:23.580293 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a92e31b-bnxkp"] Oct 31 00:37:23.594990 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:23.594959 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18e8cedf-txqlt" Oct 31 00:37:23.602936 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:23.602902 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sk58f\" (UniqueName: \"kubernetes.io/projected/852ea7c9-0231-499f-9fb2-1a3256ad77c3-kube-api-access-sk58f\") pod \"maas-api-revoke-sub-7a92e31b-bnxkp\" (UID: \"852ea7c9-0231-499f-9fb2-1a3256ad77c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a92e31b-bnxkp" Oct 31 00:37:23.603136 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:23.603064 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/852ea7c9-0231-499f-9fb2-1a3256ad77c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a92e31b-bnxkp\" (UID: \"852ea7c9-0231-499f-9fb2-1a3256ad77c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a92e31b-bnxkp" Oct 31 00:37:23.603238 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:23.603219 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/852ea7c9-0231-499f-9fb2-1a3256ad77c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a92e31b-bnxkp\" (UID: \"852ea7c9-0231-499f-9fb2-1a3256ad77c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a92e31b-bnxkp" Oct 31 00:37:23.704343 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:23.704311 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47f9711a-f60e-4fec-accd-a065ab4eb48a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47f9711a-f60e-4fec-accd-a065ab4eb48a-maas-api-openshift-ca\") pod \"47f9711a-f60e-4fec-accd-a065ab4eb48a\" (UID: \"47f9711a-f60e-4fec-accd-a065ab4eb48a\") " Oct 31 00:37:23.704591 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:23.704454 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47f9711a-f60e-4fec-accd-a065ab4eb48a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47f9711a-f60e-4fec-accd-a065ab4eb48a-maas-api-xks-ca\") pod \"47f9711a-f60e-4fec-accd-a065ab4eb48a\" (UID: \"47f9711a-f60e-4fec-accd-a065ab4eb48a\") " Oct 31 00:37:23.704591 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:23.704487 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47f9711a-f60e-4fec-accd-a065ab4eb48a-kube-api-access-jrw5z\" (UniqueName: \"kubernetes.io/projected/47f9711a-f60e-4fec-accd-a065ab4eb48a-kube-api-access-jrw5z\") pod \"47f9711a-f60e-4fec-accd-a065ab4eb48a\" (UID: \"47f9711a-f60e-4fec-accd-a065ab4eb48a\") " Oct 31 00:37:23.704805 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:23.704776 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47f9711a-f60e-4fec-accd-a065ab4eb48a-maas-api-openshift-ca" pod "47f9711a-f60e-4fec-accd-a065ab4eb48a" (UID: "47f9711a-f60e-4fec-accd-a065ab4eb48a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:37:23.705054 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:23.705024 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sk58f\" (UniqueName: \"kubernetes.io/projected/852ea7c9-0231-499f-9fb2-1a3256ad77c3-kube-api-access-sk58f\") pod \"maas-api-revoke-sub-7a92e31b-bnxkp\" (UID: \"852ea7c9-0231-499f-9fb2-1a3256ad77c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a92e31b-bnxkp" Oct 31 00:37:23.705329 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:23.705182 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/852ea7c9-0231-499f-9fb2-1a3256ad77c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a92e31b-bnxkp\" (UID: \"852ea7c9-0231-499f-9fb2-1a3256ad77c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a92e31b-bnxkp" Oct 31 00:37:23.705964 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:23.705857 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/852ea7c9-0231-499f-9fb2-1a3256ad77c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a92e31b-bnxkp\" (UID: \"852ea7c9-0231-499f-9fb2-1a3256ad77c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a92e31b-bnxkp" Oct 31 00:37:23.706119 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:23.706097 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47f9711a-f60e-4fec-accd-a065ab4eb48a-maas-api-openshift-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:23.706516 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:23.706496 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/852ea7c9-0231-499f-9fb2-1a3256ad77c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a92e31b-bnxkp\" (UID: \"852ea7c9-0231-499f-9fb2-1a3256ad77c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a92e31b-bnxkp" Oct 31 00:37:23.706999 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:23.706968 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47f9711a-f60e-4fec-accd-a065ab4eb48a-maas-api-xks-ca" pod "47f9711a-f60e-4fec-accd-a065ab4eb48a" (UID: "47f9711a-f60e-4fec-accd-a065ab4eb48a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:37:23.707065 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:23.707023 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47f9711a-f60e-4fec-accd-a065ab4eb48a-kube-api-access-jrw5z" pod "47f9711a-f60e-4fec-accd-a065ab4eb48a" (UID: "47f9711a-f60e-4fec-accd-a065ab4eb48a"). InnerVolumeSpecName "kube-api-access-jrw5z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:37:23.707606 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:23.707589 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/852ea7c9-0231-499f-9fb2-1a3256ad77c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a92e31b-bnxkp\" (UID: \"852ea7c9-0231-499f-9fb2-1a3256ad77c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a92e31b-bnxkp" Oct 31 00:37:23.768256 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:23.768225 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sk58f\" (UniqueName: \"kubernetes.io/projected/852ea7c9-0231-499f-9fb2-1a3256ad77c3-kube-api-access-sk58f\") pod \"maas-api-revoke-sub-7a92e31b-bnxkp\" (UID: \"852ea7c9-0231-499f-9fb2-1a3256ad77c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a92e31b-bnxkp" Oct 31 00:37:23.807849 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:23.807819 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47f9711a-f60e-4fec-accd-a065ab4eb48a-maas-api-xks-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:23.807849 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:23.807845 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrw5z\" (UniqueName: \"kubernetes.io/projected/47f9711a-f60e-4fec-accd-a065ab4eb48a-kube-api-access-jrw5z\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:23.844224 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:23.844191 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a92e31b-bnxkp" Oct 31 00:37:24.053199 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:24.053115 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0a93465-jv7vb"] Oct 31 00:37:24.056940 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:24.056912 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0a93465-jv7vb"] Oct 31 00:37:24.220746 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:37:24.220713 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod852ea7c9_0231_499f_9fb2_1a3256ad77c3.slice/crio-676a99267ad31bd0a833ec2693d30b57873df4d8052f199f4bda623daf9396ac WatchSource:0}: Error finding container 676a99267ad31bd0a833ec2693d30b57873df4d8052f199f4bda623daf9396ac: Status 404 returned error can't find the container with id 676a99267ad31bd0a833ec2693d30b57873df4d8052f199f4bda623daf9396ac Oct 31 00:37:24.224315 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:24.224294 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a92e31b-bnxkp"] Oct 31 00:37:24.392180 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:24.392142 5222 generic.go:356] "Generic (PLEG): container finished" podID="e8a5550a-5d20-4ec9-8272-dc539a00e96d" containerID="d6b07d7835a4fcba7670fc1965a39e3b60bdc3f5f3e3a1aaee9e20f044c002ae" exitCode=0 Oct 31 00:37:24.392674 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:24.392238 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55867ce2-z8zk8" event={"ID":"e8a5550a-5d20-4ec9-8272-dc539a00e96d","Type":"ContainerDied","Data":"d6b07d7835a4fcba7670fc1965a39e3b60bdc3f5f3e3a1aaee9e20f044c002ae"} Oct 31 00:37:24.393801 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:24.393784 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18e8cedf-txqlt" Oct 31 00:37:24.393928 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:24.393785 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18e8cedf-txqlt" event={"ID":"47f9711a-f60e-4fec-accd-a065ab4eb48a","Type":"ContainerDied","Data":"7297d6bb268e55d3d9e775d3c02e9bab5427aa05b8015c85862f71867a3c6b62"} Oct 31 00:37:24.393928 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:24.393899 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7297d6bb268e55d3d9e775d3c02e9bab5427aa05b8015c85862f71867a3c6b62" Oct 31 00:37:24.395144 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:24.395115 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a92e31b-bnxkp" event={"ID":"852ea7c9-0231-499f-9fb2-1a3256ad77c3","Type":"ContainerStarted","Data":"676a99267ad31bd0a833ec2693d30b57873df4d8052f199f4bda623daf9396ac"} Oct 31 00:37:25.404913 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:25.404821 5222 generic.go:356] "Generic (PLEG): container finished" podID="852ea7c9-0231-499f-9fb2-1a3256ad77c3" containerID="cec70e4cc7b562990b22c53ee9670cd2f56c5b2796c0d696cfa0edd0b2bf5143" exitCode=0 Oct 31 00:37:25.405351 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:25.404906 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a92e31b-bnxkp" event={"ID":"852ea7c9-0231-499f-9fb2-1a3256ad77c3","Type":"ContainerDied","Data":"cec70e4cc7b562990b22c53ee9670cd2f56c5b2796c0d696cfa0edd0b2bf5143"} Oct 31 00:37:25.421638 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:25.421606 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5558bdda-f54k5"] Oct 31 00:37:25.422170 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:25.422157 5222 cpu_manager.go:414] "containerMap: removing container" podUID="47f9711a-f60e-4fec-accd-a065ab4eb48a" containerName="revoke-keys" Oct 31 00:37:25.422223 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:25.422172 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47f9711a-f60e-4fec-accd-a065ab4eb48a" containerName="revoke-keys" Oct 31 00:37:25.422280 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:25.422271 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="47f9711a-f60e-4fec-accd-a065ab4eb48a" containerName="revoke-keys" Oct 31 00:37:25.425990 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:25.425966 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5558bdda-f54k5" Oct 31 00:37:25.481706 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:25.481667 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5558bdda-f54k5"] Oct 31 00:37:25.531823 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:25.531789 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba482f5f-fe5e-4204-b5ec-99ad3003acb5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5558bdda-f54k5\" (UID: \"ba482f5f-fe5e-4204-b5ec-99ad3003acb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5558bdda-f54k5" Oct 31 00:37:25.532000 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:25.531846 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba482f5f-fe5e-4204-b5ec-99ad3003acb5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5558bdda-f54k5\" (UID: \"ba482f5f-fe5e-4204-b5ec-99ad3003acb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5558bdda-f54k5" Oct 31 00:37:25.532000 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:25.531935 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h92t2\" (UniqueName: \"kubernetes.io/projected/ba482f5f-fe5e-4204-b5ec-99ad3003acb5-kube-api-access-h92t2\") pod \"maas-api-revoke-sub-5558bdda-f54k5\" (UID: \"ba482f5f-fe5e-4204-b5ec-99ad3003acb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5558bdda-f54k5" Oct 31 00:37:25.564993 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:25.564967 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55867ce2-z8zk8" Oct 31 00:37:25.632886 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:25.632857 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8a5550a-5d20-4ec9-8272-dc539a00e96d-kube-api-access-zk67v\" (UniqueName: \"kubernetes.io/projected/e8a5550a-5d20-4ec9-8272-dc539a00e96d-kube-api-access-zk67v\") pod \"e8a5550a-5d20-4ec9-8272-dc539a00e96d\" (UID: \"e8a5550a-5d20-4ec9-8272-dc539a00e96d\") " Oct 31 00:37:25.633059 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:25.632917 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8a5550a-5d20-4ec9-8272-dc539a00e96d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8a5550a-5d20-4ec9-8272-dc539a00e96d-maas-api-openshift-ca\") pod \"e8a5550a-5d20-4ec9-8272-dc539a00e96d\" (UID: \"e8a5550a-5d20-4ec9-8272-dc539a00e96d\") " Oct 31 00:37:25.633141 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:25.633121 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8a5550a-5d20-4ec9-8272-dc539a00e96d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8a5550a-5d20-4ec9-8272-dc539a00e96d-maas-api-xks-ca\") pod \"e8a5550a-5d20-4ec9-8272-dc539a00e96d\" (UID: \"e8a5550a-5d20-4ec9-8272-dc539a00e96d\") " Oct 31 00:37:25.633420 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:25.633387 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8a5550a-5d20-4ec9-8272-dc539a00e96d-maas-api-openshift-ca" pod "e8a5550a-5d20-4ec9-8272-dc539a00e96d" (UID: "e8a5550a-5d20-4ec9-8272-dc539a00e96d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:37:25.633570 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:25.633548 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba482f5f-fe5e-4204-b5ec-99ad3003acb5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5558bdda-f54k5\" (UID: \"ba482f5f-fe5e-4204-b5ec-99ad3003acb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5558bdda-f54k5" Oct 31 00:37:25.633651 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:25.633622 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba482f5f-fe5e-4204-b5ec-99ad3003acb5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5558bdda-f54k5\" (UID: \"ba482f5f-fe5e-4204-b5ec-99ad3003acb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5558bdda-f54k5" Oct 31 00:37:25.633708 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:25.633667 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h92t2\" (UniqueName: \"kubernetes.io/projected/ba482f5f-fe5e-4204-b5ec-99ad3003acb5-kube-api-access-h92t2\") pod \"maas-api-revoke-sub-5558bdda-f54k5\" (UID: \"ba482f5f-fe5e-4204-b5ec-99ad3003acb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5558bdda-f54k5" Oct 31 00:37:25.633956 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:25.633937 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8a5550a-5d20-4ec9-8272-dc539a00e96d-maas-api-openshift-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:25.634876 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:25.634838 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba482f5f-fe5e-4204-b5ec-99ad3003acb5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5558bdda-f54k5\" (UID: \"ba482f5f-fe5e-4204-b5ec-99ad3003acb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5558bdda-f54k5" Oct 31 00:37:25.637559 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:25.637534 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba482f5f-fe5e-4204-b5ec-99ad3003acb5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5558bdda-f54k5\" (UID: \"ba482f5f-fe5e-4204-b5ec-99ad3003acb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5558bdda-f54k5" Oct 31 00:37:25.638852 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:25.638825 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8a5550a-5d20-4ec9-8272-dc539a00e96d-maas-api-xks-ca" pod "e8a5550a-5d20-4ec9-8272-dc539a00e96d" (UID: "e8a5550a-5d20-4ec9-8272-dc539a00e96d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:37:25.638852 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:25.638835 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8a5550a-5d20-4ec9-8272-dc539a00e96d-kube-api-access-zk67v" pod "e8a5550a-5d20-4ec9-8272-dc539a00e96d" (UID: "e8a5550a-5d20-4ec9-8272-dc539a00e96d"). InnerVolumeSpecName "kube-api-access-zk67v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:37:25.659932 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:25.659827 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b801f228-4d3a-4233-a4ea-a263bf53a34f" path="/var/lib/kubelet/pods/b801f228-4d3a-4233-a4ea-a263bf53a34f/volumes" Oct 31 00:37:25.696585 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:25.696551 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h92t2\" (UniqueName: \"kubernetes.io/projected/ba482f5f-fe5e-4204-b5ec-99ad3003acb5-kube-api-access-h92t2\") pod \"maas-api-revoke-sub-5558bdda-f54k5\" (UID: \"ba482f5f-fe5e-4204-b5ec-99ad3003acb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5558bdda-f54k5" Oct 31 00:37:25.735314 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:25.735272 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8a5550a-5d20-4ec9-8272-dc539a00e96d-maas-api-xks-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:25.735314 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:25.735308 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zk67v\" (UniqueName: \"kubernetes.io/projected/e8a5550a-5d20-4ec9-8272-dc539a00e96d-kube-api-access-zk67v\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:25.744666 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:25.744632 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5558bdda-f54k5" Oct 31 00:37:26.139789 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:37:26.139756 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba482f5f_fe5e_4204_b5ec_99ad3003acb5.slice/crio-d93ea324dd5f66aae18b0daf9353527c74585ce38705d0c0f4e967199481e50a WatchSource:0}: Error finding container d93ea324dd5f66aae18b0daf9353527c74585ce38705d0c0f4e967199481e50a: Status 404 returned error can't find the container with id d93ea324dd5f66aae18b0daf9353527c74585ce38705d0c0f4e967199481e50a Oct 31 00:37:26.143653 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:26.143626 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5558bdda-f54k5"] Oct 31 00:37:26.416438 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:26.416322 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55867ce2-z8zk8" Oct 31 00:37:26.416438 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:26.416338 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55867ce2-z8zk8" event={"ID":"e8a5550a-5d20-4ec9-8272-dc539a00e96d","Type":"ContainerDied","Data":"b73c88b04b1f7dc77fbfaaf888d747ce41948f0d9d66e08cf2577ac2e5c74bc0"} Oct 31 00:37:26.416438 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:26.416423 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b73c88b04b1f7dc77fbfaaf888d747ce41948f0d9d66e08cf2577ac2e5c74bc0" Oct 31 00:37:26.418124 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:26.418094 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5558bdda-f54k5" event={"ID":"ba482f5f-fe5e-4204-b5ec-99ad3003acb5","Type":"ContainerStarted","Data":"d93ea324dd5f66aae18b0daf9353527c74585ce38705d0c0f4e967199481e50a"} Oct 31 00:37:26.725659 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:26.725269 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a92e31b-bnxkp" Oct 31 00:37:26.769170 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:26.769122 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/852ea7c9-0231-499f-9fb2-1a3256ad77c3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/852ea7c9-0231-499f-9fb2-1a3256ad77c3-maas-api-xks-ca\") pod \"852ea7c9-0231-499f-9fb2-1a3256ad77c3\" (UID: \"852ea7c9-0231-499f-9fb2-1a3256ad77c3\") " Oct 31 00:37:26.769404 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:26.769205 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/852ea7c9-0231-499f-9fb2-1a3256ad77c3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/852ea7c9-0231-499f-9fb2-1a3256ad77c3-maas-api-openshift-ca\") pod \"852ea7c9-0231-499f-9fb2-1a3256ad77c3\" (UID: \"852ea7c9-0231-499f-9fb2-1a3256ad77c3\") " Oct 31 00:37:26.769404 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:26.769353 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/852ea7c9-0231-499f-9fb2-1a3256ad77c3-kube-api-access-sk58f\" (UniqueName: \"kubernetes.io/projected/852ea7c9-0231-499f-9fb2-1a3256ad77c3-kube-api-access-sk58f\") pod \"852ea7c9-0231-499f-9fb2-1a3256ad77c3\" (UID: \"852ea7c9-0231-499f-9fb2-1a3256ad77c3\") " Oct 31 00:37:26.770743 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:26.770704 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/852ea7c9-0231-499f-9fb2-1a3256ad77c3-maas-api-openshift-ca" pod "852ea7c9-0231-499f-9fb2-1a3256ad77c3" (UID: "852ea7c9-0231-499f-9fb2-1a3256ad77c3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:37:26.808657 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:26.790948 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/852ea7c9-0231-499f-9fb2-1a3256ad77c3-kube-api-access-sk58f" pod "852ea7c9-0231-499f-9fb2-1a3256ad77c3" (UID: "852ea7c9-0231-499f-9fb2-1a3256ad77c3"). InnerVolumeSpecName "kube-api-access-sk58f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:37:26.821444 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:26.814560 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/852ea7c9-0231-499f-9fb2-1a3256ad77c3-maas-api-xks-ca" pod "852ea7c9-0231-499f-9fb2-1a3256ad77c3" (UID: "852ea7c9-0231-499f-9fb2-1a3256ad77c3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:37:26.899970 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:26.898673 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/852ea7c9-0231-499f-9fb2-1a3256ad77c3-maas-api-openshift-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:26.899970 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:26.898706 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sk58f\" (UniqueName: \"kubernetes.io/projected/852ea7c9-0231-499f-9fb2-1a3256ad77c3-kube-api-access-sk58f\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:26.899970 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:26.898720 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/852ea7c9-0231-499f-9fb2-1a3256ad77c3-maas-api-xks-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:27.460396 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:27.459679 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a92e31b-bnxkp" event={"ID":"852ea7c9-0231-499f-9fb2-1a3256ad77c3","Type":"ContainerDied","Data":"676a99267ad31bd0a833ec2693d30b57873df4d8052f199f4bda623daf9396ac"} Oct 31 00:37:27.460396 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:27.459749 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="676a99267ad31bd0a833ec2693d30b57873df4d8052f199f4bda623daf9396ac" Oct 31 00:37:27.460396 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:27.459844 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a92e31b-bnxkp" Oct 31 00:37:27.505395 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:27.501816 5222 generic.go:356] "Generic (PLEG): container finished" podID="ba482f5f-fe5e-4204-b5ec-99ad3003acb5" containerID="4a5c4c517c5fa77706975c311eeabacceca5ad190656a68e0e72594dad7cb990" exitCode=0 Oct 31 00:37:27.505395 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:27.502010 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5558bdda-f54k5" event={"ID":"ba482f5f-fe5e-4204-b5ec-99ad3003acb5","Type":"ContainerDied","Data":"4a5c4c517c5fa77706975c311eeabacceca5ad190656a68e0e72594dad7cb990"} Oct 31 00:37:28.712558 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:28.712532 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5558bdda-f54k5" Oct 31 00:37:28.790605 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:28.790499 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba482f5f-fe5e-4204-b5ec-99ad3003acb5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba482f5f-fe5e-4204-b5ec-99ad3003acb5-maas-api-openshift-ca\") pod \"ba482f5f-fe5e-4204-b5ec-99ad3003acb5\" (UID: \"ba482f5f-fe5e-4204-b5ec-99ad3003acb5\") " Oct 31 00:37:28.790605 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:28.790548 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba482f5f-fe5e-4204-b5ec-99ad3003acb5-kube-api-access-h92t2\" (UniqueName: \"kubernetes.io/projected/ba482f5f-fe5e-4204-b5ec-99ad3003acb5-kube-api-access-h92t2\") pod \"ba482f5f-fe5e-4204-b5ec-99ad3003acb5\" (UID: \"ba482f5f-fe5e-4204-b5ec-99ad3003acb5\") " Oct 31 00:37:28.790844 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:28.790736 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba482f5f-fe5e-4204-b5ec-99ad3003acb5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba482f5f-fe5e-4204-b5ec-99ad3003acb5-maas-api-xks-ca\") pod \"ba482f5f-fe5e-4204-b5ec-99ad3003acb5\" (UID: \"ba482f5f-fe5e-4204-b5ec-99ad3003acb5\") " Oct 31 00:37:28.790929 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:28.790899 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba482f5f-fe5e-4204-b5ec-99ad3003acb5-maas-api-openshift-ca" pod "ba482f5f-fe5e-4204-b5ec-99ad3003acb5" (UID: "ba482f5f-fe5e-4204-b5ec-99ad3003acb5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:37:28.791314 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:28.791290 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba482f5f-fe5e-4204-b5ec-99ad3003acb5-maas-api-openshift-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:28.793143 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:28.793116 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba482f5f-fe5e-4204-b5ec-99ad3003acb5-maas-api-xks-ca" pod "ba482f5f-fe5e-4204-b5ec-99ad3003acb5" (UID: "ba482f5f-fe5e-4204-b5ec-99ad3003acb5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:37:28.793143 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:28.793127 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba482f5f-fe5e-4204-b5ec-99ad3003acb5-kube-api-access-h92t2" pod "ba482f5f-fe5e-4204-b5ec-99ad3003acb5" (UID: "ba482f5f-fe5e-4204-b5ec-99ad3003acb5"). InnerVolumeSpecName "kube-api-access-h92t2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:37:28.892567 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:28.892527 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h92t2\" (UniqueName: \"kubernetes.io/projected/ba482f5f-fe5e-4204-b5ec-99ad3003acb5-kube-api-access-h92t2\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:28.892567 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:28.892558 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba482f5f-fe5e-4204-b5ec-99ad3003acb5-maas-api-xks-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:29.530163 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:29.530131 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5558bdda-f54k5" event={"ID":"ba482f5f-fe5e-4204-b5ec-99ad3003acb5","Type":"ContainerDied","Data":"d93ea324dd5f66aae18b0daf9353527c74585ce38705d0c0f4e967199481e50a"} Oct 31 00:37:29.530345 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:29.530217 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d93ea324dd5f66aae18b0daf9353527c74585ce38705d0c0f4e967199481e50a" Oct 31 00:37:29.530345 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:29.530144 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5558bdda-f54k5" Oct 31 00:37:30.054105 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:30.054072 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f83ba8e-sjjz4"] Oct 31 00:37:30.061803 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:30.061772 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f83ba8e-sjjz4"] Oct 31 00:37:30.416965 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:30.416883 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a81ead5-2ctzc"] Oct 31 00:37:30.417440 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:30.417424 5222 cpu_manager.go:414] "containerMap: removing container" podUID="852ea7c9-0231-499f-9fb2-1a3256ad77c3" containerName="revoke-keys" Oct 31 00:37:30.417505 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:30.417441 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="852ea7c9-0231-499f-9fb2-1a3256ad77c3" containerName="revoke-keys" Oct 31 00:37:30.417505 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:30.417466 5222 cpu_manager.go:414] "containerMap: removing container" podUID="ba482f5f-fe5e-4204-b5ec-99ad3003acb5" containerName="revoke-keys" Oct 31 00:37:30.417505 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:30.417472 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba482f5f-fe5e-4204-b5ec-99ad3003acb5" containerName="revoke-keys" Oct 31 00:37:30.417505 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:30.417498 5222 cpu_manager.go:414] "containerMap: removing container" podUID="e8a5550a-5d20-4ec9-8272-dc539a00e96d" containerName="revoke-keys" Oct 31 00:37:30.417505 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:30.417505 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8a5550a-5d20-4ec9-8272-dc539a00e96d" containerName="revoke-keys" Oct 31 00:37:30.417645 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:30.417603 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8a5550a-5d20-4ec9-8272-dc539a00e96d" containerName="revoke-keys" Oct 31 00:37:30.417645 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:30.417615 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="852ea7c9-0231-499f-9fb2-1a3256ad77c3" containerName="revoke-keys" Oct 31 00:37:30.417645 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:30.417621 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba482f5f-fe5e-4204-b5ec-99ad3003acb5" containerName="revoke-keys" Oct 31 00:37:30.421424 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:30.421405 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a81ead5-2ctzc" Oct 31 00:37:30.476404 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:30.476345 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a81ead5-2ctzc"] Oct 31 00:37:30.514041 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:30.514000 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zr7b4\" (UniqueName: \"kubernetes.io/projected/2bf0aca2-3542-4659-a5fc-eb4078a23488-kube-api-access-zr7b4\") pod \"maas-api-revoke-sub-6a81ead5-2ctzc\" (UID: \"2bf0aca2-3542-4659-a5fc-eb4078a23488\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a81ead5-2ctzc" Oct 31 00:37:30.514222 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:30.514053 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bf0aca2-3542-4659-a5fc-eb4078a23488-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a81ead5-2ctzc\" (UID: \"2bf0aca2-3542-4659-a5fc-eb4078a23488\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a81ead5-2ctzc" Oct 31 00:37:30.514272 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:30.514221 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bf0aca2-3542-4659-a5fc-eb4078a23488-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a81ead5-2ctzc\" (UID: \"2bf0aca2-3542-4659-a5fc-eb4078a23488\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a81ead5-2ctzc" Oct 31 00:37:30.523443 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:30.523415 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfz9z\"" Oct 31 00:37:30.523443 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:30.523429 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:37:30.523637 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:30.523493 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:37:30.523637 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:30.523495 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:37:30.615761 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:30.615727 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bf0aca2-3542-4659-a5fc-eb4078a23488-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a81ead5-2ctzc\" (UID: \"2bf0aca2-3542-4659-a5fc-eb4078a23488\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a81ead5-2ctzc" Oct 31 00:37:30.615956 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:30.615823 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bf0aca2-3542-4659-a5fc-eb4078a23488-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a81ead5-2ctzc\" (UID: \"2bf0aca2-3542-4659-a5fc-eb4078a23488\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a81ead5-2ctzc" Oct 31 00:37:30.615956 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:30.615932 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zr7b4\" (UniqueName: \"kubernetes.io/projected/2bf0aca2-3542-4659-a5fc-eb4078a23488-kube-api-access-zr7b4\") pod \"maas-api-revoke-sub-6a81ead5-2ctzc\" (UID: \"2bf0aca2-3542-4659-a5fc-eb4078a23488\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a81ead5-2ctzc" Oct 31 00:37:30.616435 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:30.616402 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bf0aca2-3542-4659-a5fc-eb4078a23488-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a81ead5-2ctzc\" (UID: \"2bf0aca2-3542-4659-a5fc-eb4078a23488\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a81ead5-2ctzc" Oct 31 00:37:30.618308 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:30.618285 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bf0aca2-3542-4659-a5fc-eb4078a23488-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a81ead5-2ctzc\" (UID: \"2bf0aca2-3542-4659-a5fc-eb4078a23488\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a81ead5-2ctzc" Oct 31 00:37:30.678416 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:30.678345 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zr7b4\" (UniqueName: \"kubernetes.io/projected/2bf0aca2-3542-4659-a5fc-eb4078a23488-kube-api-access-zr7b4\") pod \"maas-api-revoke-sub-6a81ead5-2ctzc\" (UID: \"2bf0aca2-3542-4659-a5fc-eb4078a23488\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a81ead5-2ctzc" Oct 31 00:37:30.735294 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:30.735257 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a81ead5-2ctzc" Oct 31 00:37:31.053601 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:31.053568 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49bb57d4-8qrc9"] Oct 31 00:37:31.057543 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:31.057518 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eabb169b-8rpmt"] Oct 31 00:37:31.061000 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:31.060977 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49bb57d4-8qrc9"] Oct 31 00:37:31.104647 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:31.104613 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eabb169b-8rpmt"] Oct 31 00:37:31.108180 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:31.108158 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-227efc5e-qg6ck"] Oct 31 00:37:31.114113 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:31.114084 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-227efc5e-qg6ck" Oct 31 00:37:31.158248 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:31.158200 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-227efc5e-qg6ck"] Oct 31 00:37:31.158449 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:37:31.158330 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bf0aca2_3542_4659_a5fc_eb4078a23488.slice/crio-cf46c675860e811675e0081971de8d770b0e6d077a45df4473dd59d0b2dffad0 WatchSource:0}: Error finding container cf46c675860e811675e0081971de8d770b0e6d077a45df4473dd59d0b2dffad0: Status 404 returned error can't find the container with id cf46c675860e811675e0081971de8d770b0e6d077a45df4473dd59d0b2dffad0 Oct 31 00:37:31.164676 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:31.164652 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a81ead5-2ctzc"] Oct 31 00:37:31.223443 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:31.223403 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cbba00e-45f7-492d-931d-3065700de2db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-227efc5e-qg6ck\" (UID: \"2cbba00e-45f7-492d-931d-3065700de2db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-227efc5e-qg6ck" Oct 31 00:37:31.223602 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:31.223549 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fx76\" (UniqueName: \"kubernetes.io/projected/2cbba00e-45f7-492d-931d-3065700de2db-kube-api-access-2fx76\") pod \"maas-api-revoke-sub-227efc5e-qg6ck\" (UID: \"2cbba00e-45f7-492d-931d-3065700de2db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-227efc5e-qg6ck" Oct 31 00:37:31.223602 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:31.223573 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cbba00e-45f7-492d-931d-3065700de2db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-227efc5e-qg6ck\" (UID: \"2cbba00e-45f7-492d-931d-3065700de2db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-227efc5e-qg6ck" Oct 31 00:37:31.325852 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:31.324725 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cbba00e-45f7-492d-931d-3065700de2db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-227efc5e-qg6ck\" (UID: \"2cbba00e-45f7-492d-931d-3065700de2db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-227efc5e-qg6ck" Oct 31 00:37:31.325852 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:31.324843 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fx76\" (UniqueName: \"kubernetes.io/projected/2cbba00e-45f7-492d-931d-3065700de2db-kube-api-access-2fx76\") pod \"maas-api-revoke-sub-227efc5e-qg6ck\" (UID: \"2cbba00e-45f7-492d-931d-3065700de2db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-227efc5e-qg6ck" Oct 31 00:37:31.325852 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:31.324875 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cbba00e-45f7-492d-931d-3065700de2db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-227efc5e-qg6ck\" (UID: \"2cbba00e-45f7-492d-931d-3065700de2db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-227efc5e-qg6ck" Oct 31 00:37:31.325852 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:31.325513 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cbba00e-45f7-492d-931d-3065700de2db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-227efc5e-qg6ck\" (UID: \"2cbba00e-45f7-492d-931d-3065700de2db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-227efc5e-qg6ck" Oct 31 00:37:31.328012 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:31.327989 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cbba00e-45f7-492d-931d-3065700de2db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-227efc5e-qg6ck\" (UID: \"2cbba00e-45f7-492d-931d-3065700de2db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-227efc5e-qg6ck" Oct 31 00:37:31.387922 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:31.387891 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fx76\" (UniqueName: \"kubernetes.io/projected/2cbba00e-45f7-492d-931d-3065700de2db-kube-api-access-2fx76\") pod \"maas-api-revoke-sub-227efc5e-qg6ck\" (UID: \"2cbba00e-45f7-492d-931d-3065700de2db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-227efc5e-qg6ck" Oct 31 00:37:31.428862 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:31.428827 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-227efc5e-qg6ck" Oct 31 00:37:31.550695 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:31.550657 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a81ead5-2ctzc" event={"ID":"2bf0aca2-3542-4659-a5fc-eb4078a23488","Type":"ContainerStarted","Data":"a67dba162cddb54c5ccac804e82cc87fb16b52f5b913436f43ea4b98d3b6ea54"} Oct 31 00:37:31.550863 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:31.550726 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a81ead5-2ctzc" event={"ID":"2bf0aca2-3542-4659-a5fc-eb4078a23488","Type":"ContainerStarted","Data":"cf46c675860e811675e0081971de8d770b0e6d077a45df4473dd59d0b2dffad0"} Oct 31 00:37:31.656769 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:31.656692 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a9a2e33-97eb-4882-b975-7f52b2e32dec" path="/var/lib/kubelet/pods/8a9a2e33-97eb-4882-b975-7f52b2e32dec/volumes" Oct 31 00:37:31.657149 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:31.657136 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fe83b3b-1e24-490e-a991-f3b86577405e" path="/var/lib/kubelet/pods/9fe83b3b-1e24-490e-a991-f3b86577405e/volumes" Oct 31 00:37:31.657538 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:31.657526 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad445cdf-d3a4-4363-8411-c66c7ac98ab8" path="/var/lib/kubelet/pods/ad445cdf-d3a4-4363-8411-c66c7ac98ab8/volumes" Oct 31 00:37:31.670843 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:31.670784 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a81ead5-2ctzc" podStartSLOduration=1.670766033 podStartE2EDuration="1.670766033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:37:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:37:31.67058605 +0000 UTC m=+1344.141755798" watchObservedRunningTime="2026-10-31 00:37:31.670766033 +0000 UTC m=+1344.141935785" Oct 31 00:37:31.809604 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:31.809578 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-227efc5e-qg6ck"] Oct 31 00:37:32.414913 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:32.414834 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22d9d793-snfv9"] Oct 31 00:37:32.419084 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:32.419063 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22d9d793-snfv9" Oct 31 00:37:32.473166 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:32.473138 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22d9d793-snfv9"] Oct 31 00:37:32.539693 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:32.539659 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbc6m\" (UniqueName: \"kubernetes.io/projected/6b5754c6-397f-40a7-87a3-23462c9dcc94-kube-api-access-zbc6m\") pod \"maas-api-revoke-sub-22d9d793-snfv9\" (UID: \"6b5754c6-397f-40a7-87a3-23462c9dcc94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22d9d793-snfv9" Oct 31 00:37:32.539865 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:32.539709 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b5754c6-397f-40a7-87a3-23462c9dcc94-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22d9d793-snfv9\" (UID: \"6b5754c6-397f-40a7-87a3-23462c9dcc94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22d9d793-snfv9" Oct 31 00:37:32.539929 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:32.539912 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b5754c6-397f-40a7-87a3-23462c9dcc94-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22d9d793-snfv9\" (UID: \"6b5754c6-397f-40a7-87a3-23462c9dcc94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22d9d793-snfv9" Oct 31 00:37:32.560514 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:32.560481 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-227efc5e-qg6ck" event={"ID":"2cbba00e-45f7-492d-931d-3065700de2db","Type":"ContainerStarted","Data":"f2fcc2b99cd9c5ecf6dc4a9ebd38f41e1e1bf0b824e2e04c301a12a68f7ae33d"} Oct 31 00:37:32.560746 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:32.560730 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-227efc5e-qg6ck" event={"ID":"2cbba00e-45f7-492d-931d-3065700de2db","Type":"ContainerStarted","Data":"5119a58310e831b9c32b298b6a94a4fc94a56c5865e7ef3b68fe84a6b87b3a93"} Oct 31 00:37:32.562145 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:32.562115 5222 generic.go:356] "Generic (PLEG): container finished" podID="2bf0aca2-3542-4659-a5fc-eb4078a23488" containerID="a67dba162cddb54c5ccac804e82cc87fb16b52f5b913436f43ea4b98d3b6ea54" exitCode=0 Oct 31 00:37:32.562270 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:32.562186 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a81ead5-2ctzc" event={"ID":"2bf0aca2-3542-4659-a5fc-eb4078a23488","Type":"ContainerDied","Data":"a67dba162cddb54c5ccac804e82cc87fb16b52f5b913436f43ea4b98d3b6ea54"} Oct 31 00:37:32.640936 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:32.640903 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b5754c6-397f-40a7-87a3-23462c9dcc94-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22d9d793-snfv9\" (UID: \"6b5754c6-397f-40a7-87a3-23462c9dcc94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22d9d793-snfv9" Oct 31 00:37:32.641094 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:32.641033 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbc6m\" (UniqueName: \"kubernetes.io/projected/6b5754c6-397f-40a7-87a3-23462c9dcc94-kube-api-access-zbc6m\") pod \"maas-api-revoke-sub-22d9d793-snfv9\" (UID: \"6b5754c6-397f-40a7-87a3-23462c9dcc94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22d9d793-snfv9" Oct 31 00:37:32.641094 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:32.641064 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b5754c6-397f-40a7-87a3-23462c9dcc94-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22d9d793-snfv9\" (UID: \"6b5754c6-397f-40a7-87a3-23462c9dcc94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22d9d793-snfv9" Oct 31 00:37:32.641774 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:32.641749 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b5754c6-397f-40a7-87a3-23462c9dcc94-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22d9d793-snfv9\" (UID: \"6b5754c6-397f-40a7-87a3-23462c9dcc94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22d9d793-snfv9" Oct 31 00:37:32.643498 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:32.643480 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b5754c6-397f-40a7-87a3-23462c9dcc94-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22d9d793-snfv9\" (UID: \"6b5754c6-397f-40a7-87a3-23462c9dcc94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22d9d793-snfv9" Oct 31 00:37:32.680709 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:32.680598 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-227efc5e-qg6ck" podStartSLOduration=1.6805816409999998 podStartE2EDuration="1.680581641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:37:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:37:32.679227192 +0000 UTC m=+1345.150396942" watchObservedRunningTime="2026-10-31 00:37:32.680581641 +0000 UTC m=+1345.151751389" Oct 31 00:37:32.704024 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:32.703991 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbc6m\" (UniqueName: \"kubernetes.io/projected/6b5754c6-397f-40a7-87a3-23462c9dcc94-kube-api-access-zbc6m\") pod \"maas-api-revoke-sub-22d9d793-snfv9\" (UID: \"6b5754c6-397f-40a7-87a3-23462c9dcc94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22d9d793-snfv9" Oct 31 00:37:32.732781 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:32.732750 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22d9d793-snfv9" Oct 31 00:37:33.127485 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:37:33.127455 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b5754c6_397f_40a7_87a3_23462c9dcc94.slice/crio-3e04b85129938bafee6a0efba70e98e4c9963b5238afa9ebaaf132e09298567d WatchSource:0}: Error finding container 3e04b85129938bafee6a0efba70e98e4c9963b5238afa9ebaaf132e09298567d: Status 404 returned error can't find the container with id 3e04b85129938bafee6a0efba70e98e4c9963b5238afa9ebaaf132e09298567d Oct 31 00:37:33.130931 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:33.130906 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22d9d793-snfv9"] Oct 31 00:37:33.572052 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:33.572017 5222 generic.go:356] "Generic (PLEG): container finished" podID="2cbba00e-45f7-492d-931d-3065700de2db" containerID="f2fcc2b99cd9c5ecf6dc4a9ebd38f41e1e1bf0b824e2e04c301a12a68f7ae33d" exitCode=0 Oct 31 00:37:33.572522 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:33.572113 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-227efc5e-qg6ck" event={"ID":"2cbba00e-45f7-492d-931d-3065700de2db","Type":"ContainerDied","Data":"f2fcc2b99cd9c5ecf6dc4a9ebd38f41e1e1bf0b824e2e04c301a12a68f7ae33d"} Oct 31 00:37:33.573821 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:33.573795 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22d9d793-snfv9" event={"ID":"6b5754c6-397f-40a7-87a3-23462c9dcc94","Type":"ContainerStarted","Data":"c49d2948aff3ebd3aa87e4514b97e61a6f8913329fdec514da1bd1fb7571a78e"} Oct 31 00:37:33.573923 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:33.573841 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22d9d793-snfv9" event={"ID":"6b5754c6-397f-40a7-87a3-23462c9dcc94","Type":"ContainerStarted","Data":"3e04b85129938bafee6a0efba70e98e4c9963b5238afa9ebaaf132e09298567d"} Oct 31 00:37:33.735160 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:33.735129 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a81ead5-2ctzc" Oct 31 00:37:33.818907 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:33.818852 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22d9d793-snfv9" podStartSLOduration=1.8188371220000001 podStartE2EDuration="1.818837122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:37:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:37:33.816454306 +0000 UTC m=+1346.287624057" watchObservedRunningTime="2026-10-31 00:37:33.818837122 +0000 UTC m=+1346.290006932" Oct 31 00:37:33.856760 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:33.856725 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2bf0aca2-3542-4659-a5fc-eb4078a23488-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bf0aca2-3542-4659-a5fc-eb4078a23488-maas-api-xks-ca\") pod \"2bf0aca2-3542-4659-a5fc-eb4078a23488\" (UID: \"2bf0aca2-3542-4659-a5fc-eb4078a23488\") " Oct 31 00:37:33.856942 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:33.856768 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bf0aca2-3542-4659-a5fc-eb4078a23488-kube-api-access-zr7b4\" (UniqueName: \"kubernetes.io/projected/2bf0aca2-3542-4659-a5fc-eb4078a23488-kube-api-access-zr7b4\") pod \"2bf0aca2-3542-4659-a5fc-eb4078a23488\" (UID: \"2bf0aca2-3542-4659-a5fc-eb4078a23488\") " Oct 31 00:37:33.856995 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:33.856944 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bf0aca2-3542-4659-a5fc-eb4078a23488-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bf0aca2-3542-4659-a5fc-eb4078a23488-maas-api-openshift-ca\") pod \"2bf0aca2-3542-4659-a5fc-eb4078a23488\" (UID: \"2bf0aca2-3542-4659-a5fc-eb4078a23488\") " Oct 31 00:37:33.857339 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:33.857308 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bf0aca2-3542-4659-a5fc-eb4078a23488-maas-api-openshift-ca" pod "2bf0aca2-3542-4659-a5fc-eb4078a23488" (UID: "2bf0aca2-3542-4659-a5fc-eb4078a23488"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:37:33.857479 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:33.857453 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bf0aca2-3542-4659-a5fc-eb4078a23488-maas-api-openshift-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:33.859058 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:33.859030 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bf0aca2-3542-4659-a5fc-eb4078a23488-maas-api-xks-ca" pod "2bf0aca2-3542-4659-a5fc-eb4078a23488" (UID: "2bf0aca2-3542-4659-a5fc-eb4078a23488"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:37:33.859168 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:33.859102 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bf0aca2-3542-4659-a5fc-eb4078a23488-kube-api-access-zr7b4" pod "2bf0aca2-3542-4659-a5fc-eb4078a23488" (UID: "2bf0aca2-3542-4659-a5fc-eb4078a23488"). InnerVolumeSpecName "kube-api-access-zr7b4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:37:33.959251 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:33.959137 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bf0aca2-3542-4659-a5fc-eb4078a23488-maas-api-xks-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:33.959251 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:33.959180 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zr7b4\" (UniqueName: \"kubernetes.io/projected/2bf0aca2-3542-4659-a5fc-eb4078a23488-kube-api-access-zr7b4\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:34.583584 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:34.583545 5222 generic.go:356] "Generic (PLEG): container finished" podID="6b5754c6-397f-40a7-87a3-23462c9dcc94" containerID="c49d2948aff3ebd3aa87e4514b97e61a6f8913329fdec514da1bd1fb7571a78e" exitCode=0 Oct 31 00:37:34.584022 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:34.583638 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22d9d793-snfv9" event={"ID":"6b5754c6-397f-40a7-87a3-23462c9dcc94","Type":"ContainerDied","Data":"c49d2948aff3ebd3aa87e4514b97e61a6f8913329fdec514da1bd1fb7571a78e"} Oct 31 00:37:34.585095 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:34.585063 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a81ead5-2ctzc" Oct 31 00:37:34.585220 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:34.585111 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a81ead5-2ctzc" event={"ID":"2bf0aca2-3542-4659-a5fc-eb4078a23488","Type":"ContainerDied","Data":"cf46c675860e811675e0081971de8d770b0e6d077a45df4473dd59d0b2dffad0"} Oct 31 00:37:34.585220 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:34.585146 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cf46c675860e811675e0081971de8d770b0e6d077a45df4473dd59d0b2dffad0" Oct 31 00:37:34.735757 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:34.735727 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-227efc5e-qg6ck" Oct 31 00:37:34.869604 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:34.869506 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cbba00e-45f7-492d-931d-3065700de2db-kube-api-access-2fx76\" (UniqueName: \"kubernetes.io/projected/2cbba00e-45f7-492d-931d-3065700de2db-kube-api-access-2fx76\") pod \"2cbba00e-45f7-492d-931d-3065700de2db\" (UID: \"2cbba00e-45f7-492d-931d-3065700de2db\") " Oct 31 00:37:34.869762 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:34.869654 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2cbba00e-45f7-492d-931d-3065700de2db-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cbba00e-45f7-492d-931d-3065700de2db-maas-api-openshift-ca\") pod \"2cbba00e-45f7-492d-931d-3065700de2db\" (UID: \"2cbba00e-45f7-492d-931d-3065700de2db\") " Oct 31 00:37:34.869762 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:34.869717 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2cbba00e-45f7-492d-931d-3065700de2db-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cbba00e-45f7-492d-931d-3065700de2db-maas-api-xks-ca\") pod \"2cbba00e-45f7-492d-931d-3065700de2db\" (UID: \"2cbba00e-45f7-492d-931d-3065700de2db\") " Oct 31 00:37:34.869996 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:34.869970 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cbba00e-45f7-492d-931d-3065700de2db-maas-api-openshift-ca" pod "2cbba00e-45f7-492d-931d-3065700de2db" (UID: "2cbba00e-45f7-492d-931d-3065700de2db"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:37:34.870095 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:34.870084 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cbba00e-45f7-492d-931d-3065700de2db-maas-api-openshift-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:34.872267 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:34.872178 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cbba00e-45f7-492d-931d-3065700de2db-kube-api-access-2fx76" pod "2cbba00e-45f7-492d-931d-3065700de2db" (UID: "2cbba00e-45f7-492d-931d-3065700de2db"). InnerVolumeSpecName "kube-api-access-2fx76". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:37:34.872426 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:34.872305 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cbba00e-45f7-492d-931d-3065700de2db-maas-api-xks-ca" pod "2cbba00e-45f7-492d-931d-3065700de2db" (UID: "2cbba00e-45f7-492d-931d-3065700de2db"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:37:34.971846 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:34.971816 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cbba00e-45f7-492d-931d-3065700de2db-maas-api-xks-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:34.971846 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:34.971844 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fx76\" (UniqueName: \"kubernetes.io/projected/2cbba00e-45f7-492d-931d-3065700de2db-kube-api-access-2fx76\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:35.594893 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:35.594861 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-227efc5e-qg6ck" Oct 31 00:37:35.594893 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:35.594886 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-227efc5e-qg6ck" event={"ID":"2cbba00e-45f7-492d-931d-3065700de2db","Type":"ContainerDied","Data":"5119a58310e831b9c32b298b6a94a4fc94a56c5865e7ef3b68fe84a6b87b3a93"} Oct 31 00:37:35.595335 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:35.594929 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5119a58310e831b9c32b298b6a94a4fc94a56c5865e7ef3b68fe84a6b87b3a93" Oct 31 00:37:35.760745 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:35.760715 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22d9d793-snfv9" Oct 31 00:37:35.883392 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:35.883297 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b5754c6-397f-40a7-87a3-23462c9dcc94-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b5754c6-397f-40a7-87a3-23462c9dcc94-maas-api-openshift-ca\") pod \"6b5754c6-397f-40a7-87a3-23462c9dcc94\" (UID: \"6b5754c6-397f-40a7-87a3-23462c9dcc94\") " Oct 31 00:37:35.883554 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:35.883409 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b5754c6-397f-40a7-87a3-23462c9dcc94-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b5754c6-397f-40a7-87a3-23462c9dcc94-maas-api-xks-ca\") pod \"6b5754c6-397f-40a7-87a3-23462c9dcc94\" (UID: \"6b5754c6-397f-40a7-87a3-23462c9dcc94\") " Oct 31 00:37:35.883554 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:35.883441 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b5754c6-397f-40a7-87a3-23462c9dcc94-kube-api-access-zbc6m\" (UniqueName: \"kubernetes.io/projected/6b5754c6-397f-40a7-87a3-23462c9dcc94-kube-api-access-zbc6m\") pod \"6b5754c6-397f-40a7-87a3-23462c9dcc94\" (UID: \"6b5754c6-397f-40a7-87a3-23462c9dcc94\") " Oct 31 00:37:35.883717 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:35.883689 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b5754c6-397f-40a7-87a3-23462c9dcc94-maas-api-openshift-ca" pod "6b5754c6-397f-40a7-87a3-23462c9dcc94" (UID: "6b5754c6-397f-40a7-87a3-23462c9dcc94"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:37:35.884039 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:35.884016 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b5754c6-397f-40a7-87a3-23462c9dcc94-maas-api-openshift-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:35.886283 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:35.886253 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b5754c6-397f-40a7-87a3-23462c9dcc94-maas-api-xks-ca" pod "6b5754c6-397f-40a7-87a3-23462c9dcc94" (UID: "6b5754c6-397f-40a7-87a3-23462c9dcc94"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:37:35.890551 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:35.890514 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b5754c6-397f-40a7-87a3-23462c9dcc94-kube-api-access-zbc6m" pod "6b5754c6-397f-40a7-87a3-23462c9dcc94" (UID: "6b5754c6-397f-40a7-87a3-23462c9dcc94"). InnerVolumeSpecName "kube-api-access-zbc6m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:37:35.905118 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:37:35.905078 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 00:37:35.985562 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:35.985523 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b5754c6-397f-40a7-87a3-23462c9dcc94-maas-api-xks-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:35.985562 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:35.985558 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbc6m\" (UniqueName: \"kubernetes.io/projected/6b5754c6-397f-40a7-87a3-23462c9dcc94-kube-api-access-zbc6m\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:36.375313 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:36.375269 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78659d5e-cpxdp"] Oct 31 00:37:36.375886 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:36.375867 5222 cpu_manager.go:414] "containerMap: removing container" podUID="6b5754c6-397f-40a7-87a3-23462c9dcc94" containerName="revoke-keys" Oct 31 00:37:36.375977 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:36.375887 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b5754c6-397f-40a7-87a3-23462c9dcc94" containerName="revoke-keys" Oct 31 00:37:36.375977 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:36.375927 5222 cpu_manager.go:414] "containerMap: removing container" podUID="2cbba00e-45f7-492d-931d-3065700de2db" containerName="revoke-keys" Oct 31 00:37:36.375977 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:36.375935 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cbba00e-45f7-492d-931d-3065700de2db" containerName="revoke-keys" Oct 31 00:37:36.375977 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:36.375966 5222 cpu_manager.go:414] "containerMap: removing container" podUID="2bf0aca2-3542-4659-a5fc-eb4078a23488" containerName="revoke-keys" Oct 31 00:37:36.375977 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:36.375973 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bf0aca2-3542-4659-a5fc-eb4078a23488" containerName="revoke-keys" Oct 31 00:37:36.376228 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:36.376105 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b5754c6-397f-40a7-87a3-23462c9dcc94" containerName="revoke-keys" Oct 31 00:37:36.376228 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:36.376125 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cbba00e-45f7-492d-931d-3065700de2db" containerName="revoke-keys" Oct 31 00:37:36.376228 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:36.376137 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bf0aca2-3542-4659-a5fc-eb4078a23488" containerName="revoke-keys" Oct 31 00:37:36.379864 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:36.379842 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78659d5e-cpxdp" Oct 31 00:37:36.432881 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:36.432838 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78659d5e-cpxdp"] Oct 31 00:37:36.492020 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:36.491973 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpwjh\" (UniqueName: \"kubernetes.io/projected/00ed5f54-bc6b-4bb7-a68c-4e1c78078178-kube-api-access-cpwjh\") pod \"maas-api-revoke-sub-78659d5e-cpxdp\" (UID: \"00ed5f54-bc6b-4bb7-a68c-4e1c78078178\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78659d5e-cpxdp" Oct 31 00:37:36.492020 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:36.492021 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00ed5f54-bc6b-4bb7-a68c-4e1c78078178-maas-api-xks-ca\") pod \"maas-api-revoke-sub-78659d5e-cpxdp\" (UID: \"00ed5f54-bc6b-4bb7-a68c-4e1c78078178\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78659d5e-cpxdp" Oct 31 00:37:36.492334 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:36.492258 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00ed5f54-bc6b-4bb7-a68c-4e1c78078178-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-78659d5e-cpxdp\" (UID: \"00ed5f54-bc6b-4bb7-a68c-4e1c78078178\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78659d5e-cpxdp" Oct 31 00:37:36.594083 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:36.594036 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00ed5f54-bc6b-4bb7-a68c-4e1c78078178-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-78659d5e-cpxdp\" (UID: \"00ed5f54-bc6b-4bb7-a68c-4e1c78078178\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78659d5e-cpxdp" Oct 31 00:37:36.594278 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:36.594099 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpwjh\" (UniqueName: \"kubernetes.io/projected/00ed5f54-bc6b-4bb7-a68c-4e1c78078178-kube-api-access-cpwjh\") pod \"maas-api-revoke-sub-78659d5e-cpxdp\" (UID: \"00ed5f54-bc6b-4bb7-a68c-4e1c78078178\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78659d5e-cpxdp" Oct 31 00:37:36.594278 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:36.594123 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00ed5f54-bc6b-4bb7-a68c-4e1c78078178-maas-api-xks-ca\") pod \"maas-api-revoke-sub-78659d5e-cpxdp\" (UID: \"00ed5f54-bc6b-4bb7-a68c-4e1c78078178\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78659d5e-cpxdp" Oct 31 00:37:36.594752 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:36.594725 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00ed5f54-bc6b-4bb7-a68c-4e1c78078178-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-78659d5e-cpxdp\" (UID: \"00ed5f54-bc6b-4bb7-a68c-4e1c78078178\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78659d5e-cpxdp" Oct 31 00:37:36.596814 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:36.596794 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00ed5f54-bc6b-4bb7-a68c-4e1c78078178-maas-api-xks-ca\") pod \"maas-api-revoke-sub-78659d5e-cpxdp\" (UID: \"00ed5f54-bc6b-4bb7-a68c-4e1c78078178\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78659d5e-cpxdp" Oct 31 00:37:36.607996 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:36.607959 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22d9d793-snfv9" event={"ID":"6b5754c6-397f-40a7-87a3-23462c9dcc94","Type":"ContainerDied","Data":"3e04b85129938bafee6a0efba70e98e4c9963b5238afa9ebaaf132e09298567d"} Oct 31 00:37:36.608098 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:36.607997 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22d9d793-snfv9" Oct 31 00:37:36.608098 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:36.608055 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e04b85129938bafee6a0efba70e98e4c9963b5238afa9ebaaf132e09298567d" Oct 31 00:37:36.657688 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:36.657619 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpwjh\" (UniqueName: \"kubernetes.io/projected/00ed5f54-bc6b-4bb7-a68c-4e1c78078178-kube-api-access-cpwjh\") pod \"maas-api-revoke-sub-78659d5e-cpxdp\" (UID: \"00ed5f54-bc6b-4bb7-a68c-4e1c78078178\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78659d5e-cpxdp" Oct 31 00:37:36.694579 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:36.694545 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78659d5e-cpxdp" Oct 31 00:37:37.074769 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:37:37.074729 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00ed5f54_bc6b_4bb7_a68c_4e1c78078178.slice/crio-5babf9be60a8fd6918918263b17b01f6ceb5cee6a816da22e4e36a7a716df798 WatchSource:0}: Error finding container 5babf9be60a8fd6918918263b17b01f6ceb5cee6a816da22e4e36a7a716df798: Status 404 returned error can't find the container with id 5babf9be60a8fd6918918263b17b01f6ceb5cee6a816da22e4e36a7a716df798 Oct 31 00:37:37.078922 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:37.078900 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78659d5e-cpxdp"] Oct 31 00:37:37.618456 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:37.618362 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78659d5e-cpxdp" event={"ID":"00ed5f54-bc6b-4bb7-a68c-4e1c78078178","Type":"ContainerStarted","Data":"57c2d1b4686ff56ee7d96686cc37a6f1dab92d848313f71470e5edf215d4e5d0"} Oct 31 00:37:37.618456 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:37.618426 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78659d5e-cpxdp" event={"ID":"00ed5f54-bc6b-4bb7-a68c-4e1c78078178","Type":"ContainerStarted","Data":"5babf9be60a8fd6918918263b17b01f6ceb5cee6a816da22e4e36a7a716df798"} Oct 31 00:37:37.737898 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:37.737842 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78659d5e-cpxdp" podStartSLOduration=1.737827896 podStartE2EDuration="1.737827896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:37:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:37:37.736495631 +0000 UTC m=+1350.207665375" watchObservedRunningTime="2026-10-31 00:37:37.737827896 +0000 UTC m=+1350.208997647" Oct 31 00:37:38.631605 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:38.631569 5222 generic.go:356] "Generic (PLEG): container finished" podID="00ed5f54-bc6b-4bb7-a68c-4e1c78078178" containerID="57c2d1b4686ff56ee7d96686cc37a6f1dab92d848313f71470e5edf215d4e5d0" exitCode=0 Oct 31 00:37:38.632012 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:38.631657 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78659d5e-cpxdp" event={"ID":"00ed5f54-bc6b-4bb7-a68c-4e1c78078178","Type":"ContainerDied","Data":"57c2d1b4686ff56ee7d96686cc37a6f1dab92d848313f71470e5edf215d4e5d0"} Oct 31 00:37:39.815052 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:39.815014 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78659d5e-cpxdp" Oct 31 00:37:39.936023 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:39.935932 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00ed5f54-bc6b-4bb7-a68c-4e1c78078178-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00ed5f54-bc6b-4bb7-a68c-4e1c78078178-maas-api-xks-ca\") pod \"00ed5f54-bc6b-4bb7-a68c-4e1c78078178\" (UID: \"00ed5f54-bc6b-4bb7-a68c-4e1c78078178\") " Oct 31 00:37:39.936178 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:39.936047 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00ed5f54-bc6b-4bb7-a68c-4e1c78078178-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00ed5f54-bc6b-4bb7-a68c-4e1c78078178-maas-api-openshift-ca\") pod \"00ed5f54-bc6b-4bb7-a68c-4e1c78078178\" (UID: \"00ed5f54-bc6b-4bb7-a68c-4e1c78078178\") " Oct 31 00:37:39.936178 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:39.936095 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00ed5f54-bc6b-4bb7-a68c-4e1c78078178-kube-api-access-cpwjh\" (UniqueName: \"kubernetes.io/projected/00ed5f54-bc6b-4bb7-a68c-4e1c78078178-kube-api-access-cpwjh\") pod \"00ed5f54-bc6b-4bb7-a68c-4e1c78078178\" (UID: \"00ed5f54-bc6b-4bb7-a68c-4e1c78078178\") " Oct 31 00:37:39.936533 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:39.936506 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00ed5f54-bc6b-4bb7-a68c-4e1c78078178-maas-api-openshift-ca" pod "00ed5f54-bc6b-4bb7-a68c-4e1c78078178" (UID: "00ed5f54-bc6b-4bb7-a68c-4e1c78078178"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:37:39.936766 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:39.936744 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00ed5f54-bc6b-4bb7-a68c-4e1c78078178-maas-api-openshift-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:39.938722 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:39.938690 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00ed5f54-bc6b-4bb7-a68c-4e1c78078178-maas-api-xks-ca" pod "00ed5f54-bc6b-4bb7-a68c-4e1c78078178" (UID: "00ed5f54-bc6b-4bb7-a68c-4e1c78078178"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:37:39.938849 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:39.938773 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00ed5f54-bc6b-4bb7-a68c-4e1c78078178-kube-api-access-cpwjh" pod "00ed5f54-bc6b-4bb7-a68c-4e1c78078178" (UID: "00ed5f54-bc6b-4bb7-a68c-4e1c78078178"). InnerVolumeSpecName "kube-api-access-cpwjh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:37:40.038019 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:40.037983 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00ed5f54-bc6b-4bb7-a68c-4e1c78078178-maas-api-xks-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:40.038019 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:40.038018 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpwjh\" (UniqueName: \"kubernetes.io/projected/00ed5f54-bc6b-4bb7-a68c-4e1c78078178-kube-api-access-cpwjh\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:40.649211 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:40.649175 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78659d5e-cpxdp" event={"ID":"00ed5f54-bc6b-4bb7-a68c-4e1c78078178","Type":"ContainerDied","Data":"5babf9be60a8fd6918918263b17b01f6ceb5cee6a816da22e4e36a7a716df798"} Oct 31 00:37:40.649211 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:40.649196 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78659d5e-cpxdp" Oct 31 00:37:40.649448 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:40.649221 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5babf9be60a8fd6918918263b17b01f6ceb5cee6a816da22e4e36a7a716df798" Oct 31 00:37:41.216436 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:41.211564 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70ab423e-lz5km"] Oct 31 00:37:41.216436 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:41.212458 5222 cpu_manager.go:414] "containerMap: removing container" podUID="00ed5f54-bc6b-4bb7-a68c-4e1c78078178" containerName="revoke-keys" Oct 31 00:37:41.216436 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:41.212479 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00ed5f54-bc6b-4bb7-a68c-4e1c78078178" containerName="revoke-keys" Oct 31 00:37:41.216436 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:41.212627 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="00ed5f54-bc6b-4bb7-a68c-4e1c78078178" containerName="revoke-keys" Oct 31 00:37:41.218635 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:41.218611 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70ab423e-lz5km" Oct 31 00:37:41.276227 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:41.276183 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:37:41.276408 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:41.276339 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfz9z\"" Oct 31 00:37:41.327593 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:41.327561 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:37:41.327789 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:41.327670 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:37:41.331997 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:41.331969 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70ab423e-lz5km"] Oct 31 00:37:41.357154 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:41.357123 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgjg6\" (UniqueName: \"kubernetes.io/projected/67d02085-6466-428c-8315-ab10655f0dd2-kube-api-access-tgjg6\") pod \"maas-api-revoke-sub-70ab423e-lz5km\" (UID: \"67d02085-6466-428c-8315-ab10655f0dd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70ab423e-lz5km" Oct 31 00:37:41.357314 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:41.357167 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67d02085-6466-428c-8315-ab10655f0dd2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70ab423e-lz5km\" (UID: \"67d02085-6466-428c-8315-ab10655f0dd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70ab423e-lz5km" Oct 31 00:37:41.357314 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:41.357194 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67d02085-6466-428c-8315-ab10655f0dd2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70ab423e-lz5km\" (UID: \"67d02085-6466-428c-8315-ab10655f0dd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70ab423e-lz5km" Oct 31 00:37:41.459034 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:41.458979 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgjg6\" (UniqueName: \"kubernetes.io/projected/67d02085-6466-428c-8315-ab10655f0dd2-kube-api-access-tgjg6\") pod \"maas-api-revoke-sub-70ab423e-lz5km\" (UID: \"67d02085-6466-428c-8315-ab10655f0dd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70ab423e-lz5km" Oct 31 00:37:41.459629 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:41.459602 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67d02085-6466-428c-8315-ab10655f0dd2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70ab423e-lz5km\" (UID: \"67d02085-6466-428c-8315-ab10655f0dd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70ab423e-lz5km" Oct 31 00:37:41.464105 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:41.464077 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67d02085-6466-428c-8315-ab10655f0dd2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70ab423e-lz5km\" (UID: \"67d02085-6466-428c-8315-ab10655f0dd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70ab423e-lz5km" Oct 31 00:37:41.464407 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:41.460642 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67d02085-6466-428c-8315-ab10655f0dd2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70ab423e-lz5km\" (UID: \"67d02085-6466-428c-8315-ab10655f0dd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70ab423e-lz5km" Oct 31 00:37:41.495964 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:41.495884 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67d02085-6466-428c-8315-ab10655f0dd2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70ab423e-lz5km\" (UID: \"67d02085-6466-428c-8315-ab10655f0dd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70ab423e-lz5km" Oct 31 00:37:41.529409 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:41.527153 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgjg6\" (UniqueName: \"kubernetes.io/projected/67d02085-6466-428c-8315-ab10655f0dd2-kube-api-access-tgjg6\") pod \"maas-api-revoke-sub-70ab423e-lz5km\" (UID: \"67d02085-6466-428c-8315-ab10655f0dd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70ab423e-lz5km" Oct 31 00:37:41.563636 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:41.563606 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70ab423e-lz5km" Oct 31 00:37:41.942281 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:37:41.942243 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67d02085_6466_428c_8315_ab10655f0dd2.slice/crio-3a384b88e81e6039e77ae474942b80f41cf0162dfac93254e36a65757ee805af WatchSource:0}: Error finding container 3a384b88e81e6039e77ae474942b80f41cf0162dfac93254e36a65757ee805af: Status 404 returned error can't find the container with id 3a384b88e81e6039e77ae474942b80f41cf0162dfac93254e36a65757ee805af Oct 31 00:37:41.945760 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:41.945733 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70ab423e-lz5km"] Oct 31 00:37:42.683217 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:42.683176 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70ab423e-lz5km" event={"ID":"67d02085-6466-428c-8315-ab10655f0dd2","Type":"ContainerStarted","Data":"a8fe5cded79bc9d4cea137ede29bda6e43993c4e2b106cdecb3ef722cdb3cc36"} Oct 31 00:37:42.683640 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:42.683240 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70ab423e-lz5km" event={"ID":"67d02085-6466-428c-8315-ab10655f0dd2","Type":"ContainerStarted","Data":"3a384b88e81e6039e77ae474942b80f41cf0162dfac93254e36a65757ee805af"} Oct 31 00:37:42.812807 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:42.812755 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70ab423e-lz5km" podStartSLOduration=1.8127401669999998 podStartE2EDuration="1.812740167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:37:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:37:42.80833879 +0000 UTC m=+1355.279508541" watchObservedRunningTime="2026-10-31 00:37:42.812740167 +0000 UTC m=+1355.283909915" Oct 31 00:37:43.210292 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:43.210261 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0cdffee-6zrmt"] Oct 31 00:37:43.214657 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:43.214641 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0cdffee-6zrmt" Oct 31 00:37:43.270142 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:43.270098 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0cdffee-6zrmt"] Oct 31 00:37:43.388472 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:43.388426 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b1e4efb-7f81-45dd-abde-15b6a07ded61-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0cdffee-6zrmt\" (UID: \"0b1e4efb-7f81-45dd-abde-15b6a07ded61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0cdffee-6zrmt" Oct 31 00:37:43.388472 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:43.388475 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b1e4efb-7f81-45dd-abde-15b6a07ded61-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0cdffee-6zrmt\" (UID: \"0b1e4efb-7f81-45dd-abde-15b6a07ded61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0cdffee-6zrmt" Oct 31 00:37:43.388706 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:43.388502 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkzcn\" (UniqueName: \"kubernetes.io/projected/0b1e4efb-7f81-45dd-abde-15b6a07ded61-kube-api-access-xkzcn\") pod \"maas-api-revoke-sub-f0cdffee-6zrmt\" (UID: \"0b1e4efb-7f81-45dd-abde-15b6a07ded61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0cdffee-6zrmt" Oct 31 00:37:43.489639 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:43.489535 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b1e4efb-7f81-45dd-abde-15b6a07ded61-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0cdffee-6zrmt\" (UID: \"0b1e4efb-7f81-45dd-abde-15b6a07ded61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0cdffee-6zrmt" Oct 31 00:37:43.489639 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:43.489599 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b1e4efb-7f81-45dd-abde-15b6a07ded61-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0cdffee-6zrmt\" (UID: \"0b1e4efb-7f81-45dd-abde-15b6a07ded61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0cdffee-6zrmt" Oct 31 00:37:43.489879 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:43.489640 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkzcn\" (UniqueName: \"kubernetes.io/projected/0b1e4efb-7f81-45dd-abde-15b6a07ded61-kube-api-access-xkzcn\") pod \"maas-api-revoke-sub-f0cdffee-6zrmt\" (UID: \"0b1e4efb-7f81-45dd-abde-15b6a07ded61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0cdffee-6zrmt" Oct 31 00:37:43.490396 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:43.490339 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b1e4efb-7f81-45dd-abde-15b6a07ded61-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0cdffee-6zrmt\" (UID: \"0b1e4efb-7f81-45dd-abde-15b6a07ded61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0cdffee-6zrmt" Oct 31 00:37:43.492134 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:43.492114 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b1e4efb-7f81-45dd-abde-15b6a07ded61-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0cdffee-6zrmt\" (UID: \"0b1e4efb-7f81-45dd-abde-15b6a07ded61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0cdffee-6zrmt" Oct 31 00:37:43.551991 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:43.551960 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkzcn\" (UniqueName: \"kubernetes.io/projected/0b1e4efb-7f81-45dd-abde-15b6a07ded61-kube-api-access-xkzcn\") pod \"maas-api-revoke-sub-f0cdffee-6zrmt\" (UID: \"0b1e4efb-7f81-45dd-abde-15b6a07ded61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0cdffee-6zrmt" Oct 31 00:37:43.692957 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:43.692918 5222 generic.go:356] "Generic (PLEG): container finished" podID="67d02085-6466-428c-8315-ab10655f0dd2" containerID="a8fe5cded79bc9d4cea137ede29bda6e43993c4e2b106cdecb3ef722cdb3cc36" exitCode=0 Oct 31 00:37:43.693396 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:43.692974 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70ab423e-lz5km" event={"ID":"67d02085-6466-428c-8315-ab10655f0dd2","Type":"ContainerDied","Data":"a8fe5cded79bc9d4cea137ede29bda6e43993c4e2b106cdecb3ef722cdb3cc36"} Oct 31 00:37:43.828538 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:43.828500 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0cdffee-6zrmt" Oct 31 00:37:44.215841 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:44.215812 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0cdffee-6zrmt"] Oct 31 00:37:44.502068 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:44.501964 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sxdqn"] Oct 31 00:37:44.507979 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:44.507949 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sxdqn" Oct 31 00:37:44.603685 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:44.603650 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sxdqn"] Oct 31 00:37:44.610039 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:44.610006 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05e35f54-2479-4d75-b372-0f10ab6819c3-istio-podinfo\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sxdqn\" (UID: \"05e35f54-2479-4d75-b372-0f10ab6819c3\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sxdqn" Oct 31 00:37:44.610220 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:44.610139 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05e35f54-2479-4d75-b372-0f10ab6819c3-istio-envoy\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sxdqn\" (UID: \"05e35f54-2479-4d75-b372-0f10ab6819c3\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sxdqn" Oct 31 00:37:44.610220 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:44.610180 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05e35f54-2479-4d75-b372-0f10ab6819c3-istio-token\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sxdqn\" (UID: \"05e35f54-2479-4d75-b372-0f10ab6819c3\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sxdqn" Oct 31 00:37:44.610365 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:44.610347 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05e35f54-2479-4d75-b372-0f10ab6819c3-credential-socket\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sxdqn\" (UID: \"05e35f54-2479-4d75-b372-0f10ab6819c3\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sxdqn" Oct 31 00:37:44.610458 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:44.610436 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05e35f54-2479-4d75-b372-0f10ab6819c3-istiod-ca-cert\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sxdqn\" (UID: \"05e35f54-2479-4d75-b372-0f10ab6819c3\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sxdqn" Oct 31 00:37:44.610521 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:44.610499 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05e35f54-2479-4d75-b372-0f10ab6819c3-workload-socket\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sxdqn\" (UID: \"05e35f54-2479-4d75-b372-0f10ab6819c3\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sxdqn" Oct 31 00:37:44.610561 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:44.610542 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05e35f54-2479-4d75-b372-0f10ab6819c3-workload-certs\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sxdqn\" (UID: \"05e35f54-2479-4d75-b372-0f10ab6819c3\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sxdqn" Oct 31 00:37:44.610606 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:44.610563 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05e35f54-2479-4d75-b372-0f10ab6819c3-istio-data\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sxdqn\" (UID: \"05e35f54-2479-4d75-b372-0f10ab6819c3\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sxdqn" Oct 31 00:37:44.610653 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:44.610640 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcwh7\" (UniqueName: \"kubernetes.io/projected/05e35f54-2479-4d75-b372-0f10ab6819c3-kube-api-access-gcwh7\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sxdqn\" (UID: \"05e35f54-2479-4d75-b372-0f10ab6819c3\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sxdqn" Oct 31 00:37:44.703890 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:44.703855 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0cdffee-6zrmt" event={"ID":"0b1e4efb-7f81-45dd-abde-15b6a07ded61","Type":"ContainerStarted","Data":"5004c0e7ca67d9557448cf97f326c352505544c489345e1f4f7e91c1923a41d6"} Oct 31 00:37:44.704347 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:44.703915 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0cdffee-6zrmt" event={"ID":"0b1e4efb-7f81-45dd-abde-15b6a07ded61","Type":"ContainerStarted","Data":"4eb071eeca2a41370ddcb410d9bdf42802965a2bff2006fa5b23ff74455730d4"} Oct 31 00:37:44.711981 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:44.711947 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05e35f54-2479-4d75-b372-0f10ab6819c3-credential-socket\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sxdqn\" (UID: \"05e35f54-2479-4d75-b372-0f10ab6819c3\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sxdqn" Oct 31 00:37:44.712109 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:44.712034 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05e35f54-2479-4d75-b372-0f10ab6819c3-istiod-ca-cert\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sxdqn\" (UID: \"05e35f54-2479-4d75-b372-0f10ab6819c3\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sxdqn" Oct 31 00:37:44.712109 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:44.712067 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05e35f54-2479-4d75-b372-0f10ab6819c3-workload-socket\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sxdqn\" (UID: \"05e35f54-2479-4d75-b372-0f10ab6819c3\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sxdqn" Oct 31 00:37:44.712109 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:44.712104 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05e35f54-2479-4d75-b372-0f10ab6819c3-workload-certs\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sxdqn\" (UID: \"05e35f54-2479-4d75-b372-0f10ab6819c3\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sxdqn" Oct 31 00:37:44.712278 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:44.712134 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05e35f54-2479-4d75-b372-0f10ab6819c3-istio-data\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sxdqn\" (UID: \"05e35f54-2479-4d75-b372-0f10ab6819c3\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sxdqn" Oct 31 00:37:44.712278 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:44.712190 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcwh7\" (UniqueName: \"kubernetes.io/projected/05e35f54-2479-4d75-b372-0f10ab6819c3-kube-api-access-gcwh7\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sxdqn\" (UID: \"05e35f54-2479-4d75-b372-0f10ab6819c3\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sxdqn" Oct 31 00:37:44.712278 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:44.712262 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05e35f54-2479-4d75-b372-0f10ab6819c3-istio-podinfo\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sxdqn\" (UID: \"05e35f54-2479-4d75-b372-0f10ab6819c3\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sxdqn" Oct 31 00:37:44.712464 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:44.712322 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05e35f54-2479-4d75-b372-0f10ab6819c3-istio-envoy\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sxdqn\" (UID: \"05e35f54-2479-4d75-b372-0f10ab6819c3\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sxdqn" Oct 31 00:37:44.712464 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:44.712352 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05e35f54-2479-4d75-b372-0f10ab6819c3-istio-token\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sxdqn\" (UID: \"05e35f54-2479-4d75-b372-0f10ab6819c3\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sxdqn" Oct 31 00:37:44.712464 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:44.712414 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05e35f54-2479-4d75-b372-0f10ab6819c3-credential-socket\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sxdqn\" (UID: \"05e35f54-2479-4d75-b372-0f10ab6819c3\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sxdqn" Oct 31 00:37:44.712622 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:44.712528 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05e35f54-2479-4d75-b372-0f10ab6819c3-workload-socket\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sxdqn\" (UID: \"05e35f54-2479-4d75-b372-0f10ab6819c3\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sxdqn" Oct 31 00:37:44.712622 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:44.712550 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05e35f54-2479-4d75-b372-0f10ab6819c3-workload-certs\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sxdqn\" (UID: \"05e35f54-2479-4d75-b372-0f10ab6819c3\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sxdqn" Oct 31 00:37:44.712812 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:44.712788 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05e35f54-2479-4d75-b372-0f10ab6819c3-istio-data\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sxdqn\" (UID: \"05e35f54-2479-4d75-b372-0f10ab6819c3\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sxdqn" Oct 31 00:37:44.712921 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:44.712898 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05e35f54-2479-4d75-b372-0f10ab6819c3-istiod-ca-cert\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sxdqn\" (UID: \"05e35f54-2479-4d75-b372-0f10ab6819c3\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sxdqn" Oct 31 00:37:44.715457 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:44.715426 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05e35f54-2479-4d75-b372-0f10ab6819c3-istio-envoy\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sxdqn\" (UID: \"05e35f54-2479-4d75-b372-0f10ab6819c3\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sxdqn" Oct 31 00:37:44.715883 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:44.715640 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05e35f54-2479-4d75-b372-0f10ab6819c3-istio-podinfo\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sxdqn\" (UID: \"05e35f54-2479-4d75-b372-0f10ab6819c3\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sxdqn" Oct 31 00:37:44.801393 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:44.794255 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05e35f54-2479-4d75-b372-0f10ab6819c3-istio-token\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sxdqn\" (UID: \"05e35f54-2479-4d75-b372-0f10ab6819c3\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sxdqn" Oct 31 00:37:44.817595 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:44.815832 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcwh7\" (UniqueName: \"kubernetes.io/projected/05e35f54-2479-4d75-b372-0f10ab6819c3-kube-api-access-gcwh7\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sxdqn\" (UID: \"05e35f54-2479-4d75-b372-0f10ab6819c3\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sxdqn" Oct 31 00:37:44.826990 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:44.826823 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sxdqn" Oct 31 00:37:44.835539 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:44.833913 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0cdffee-6zrmt" podStartSLOduration=1.833892758 podStartE2EDuration="1.833892758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:37:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:37:44.831772987 +0000 UTC m=+1357.302942766" watchObservedRunningTime="2026-10-31 00:37:44.833892758 +0000 UTC m=+1357.305062509" Oct 31 00:37:44.962151 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:44.962122 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70ab423e-lz5km" Oct 31 00:37:45.033820 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:45.033781 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67d02085-6466-428c-8315-ab10655f0dd2-kube-api-access-tgjg6\" (UniqueName: \"kubernetes.io/projected/67d02085-6466-428c-8315-ab10655f0dd2-kube-api-access-tgjg6\") pod \"67d02085-6466-428c-8315-ab10655f0dd2\" (UID: \"67d02085-6466-428c-8315-ab10655f0dd2\") " Oct 31 00:37:45.034005 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:45.033922 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67d02085-6466-428c-8315-ab10655f0dd2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67d02085-6466-428c-8315-ab10655f0dd2-maas-api-xks-ca\") pod \"67d02085-6466-428c-8315-ab10655f0dd2\" (UID: \"67d02085-6466-428c-8315-ab10655f0dd2\") " Oct 31 00:37:45.034085 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:45.034027 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67d02085-6466-428c-8315-ab10655f0dd2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67d02085-6466-428c-8315-ab10655f0dd2-maas-api-openshift-ca\") pod \"67d02085-6466-428c-8315-ab10655f0dd2\" (UID: \"67d02085-6466-428c-8315-ab10655f0dd2\") " Oct 31 00:37:45.034510 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:45.034479 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67d02085-6466-428c-8315-ab10655f0dd2-maas-api-openshift-ca" pod "67d02085-6466-428c-8315-ab10655f0dd2" (UID: "67d02085-6466-428c-8315-ab10655f0dd2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:37:45.034699 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:45.034604 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67d02085-6466-428c-8315-ab10655f0dd2-maas-api-openshift-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:45.036769 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:45.036740 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67d02085-6466-428c-8315-ab10655f0dd2-maas-api-xks-ca" pod "67d02085-6466-428c-8315-ab10655f0dd2" (UID: "67d02085-6466-428c-8315-ab10655f0dd2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:37:45.036893 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:45.036770 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67d02085-6466-428c-8315-ab10655f0dd2-kube-api-access-tgjg6" pod "67d02085-6466-428c-8315-ab10655f0dd2" (UID: "67d02085-6466-428c-8315-ab10655f0dd2"). InnerVolumeSpecName "kube-api-access-tgjg6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:37:45.135905 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:45.135822 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgjg6\" (UniqueName: \"kubernetes.io/projected/67d02085-6466-428c-8315-ab10655f0dd2-kube-api-access-tgjg6\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:45.135905 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:45.135855 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67d02085-6466-428c-8315-ab10655f0dd2-maas-api-xks-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:45.359774 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:45.359744 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sxdqn"] Oct 31 00:37:45.362605 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:37:45.362569 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05e35f54_2479_4d75_b372_0f10ab6819c3.slice/crio-04e54fd43944373cd5d77d1b48bbaa742d0ec5fd46a0fb12f5b15671f8ddaeeb WatchSource:0}: Error finding container 04e54fd43944373cd5d77d1b48bbaa742d0ec5fd46a0fb12f5b15671f8ddaeeb: Status 404 returned error can't find the container with id 04e54fd43944373cd5d77d1b48bbaa742d0ec5fd46a0fb12f5b15671f8ddaeeb Oct 31 00:37:45.365203 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:45.365169 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:37:45.365279 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:45.365236 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:37:45.365279 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:45.365268 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:37:45.714970 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:45.714939 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70ab423e-lz5km" Oct 31 00:37:45.715821 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:45.714966 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70ab423e-lz5km" event={"ID":"67d02085-6466-428c-8315-ab10655f0dd2","Type":"ContainerDied","Data":"3a384b88e81e6039e77ae474942b80f41cf0162dfac93254e36a65757ee805af"} Oct 31 00:37:45.715821 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:45.715022 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3a384b88e81e6039e77ae474942b80f41cf0162dfac93254e36a65757ee805af" Oct 31 00:37:45.717111 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:45.717087 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sxdqn" event={"ID":"05e35f54-2479-4d75-b372-0f10ab6819c3","Type":"ContainerStarted","Data":"3682d6cda2b31af1a401bc2920770da23a88ca5efa60fab2c4d49bab22167f3c"} Oct 31 00:37:45.717251 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:45.717151 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sxdqn" event={"ID":"05e35f54-2479-4d75-b372-0f10ab6819c3","Type":"ContainerStarted","Data":"04e54fd43944373cd5d77d1b48bbaa742d0ec5fd46a0fb12f5b15671f8ddaeeb"} Oct 31 00:37:45.719246 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:45.719224 5222 generic.go:356] "Generic (PLEG): container finished" podID="0b1e4efb-7f81-45dd-abde-15b6a07ded61" containerID="5004c0e7ca67d9557448cf97f326c352505544c489345e1f4f7e91c1923a41d6" exitCode=0 Oct 31 00:37:45.719353 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:45.719307 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0cdffee-6zrmt" event={"ID":"0b1e4efb-7f81-45dd-abde-15b6a07ded61","Type":"ContainerDied","Data":"5004c0e7ca67d9557448cf97f326c352505544c489345e1f4f7e91c1923a41d6"} Oct 31 00:37:45.827640 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:45.827605 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sxdqn" Oct 31 00:37:45.881187 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:45.881118 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sxdqn" podStartSLOduration=1.881099197 podStartE2EDuration="1.881099197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:37:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:37:45.875528368 +0000 UTC m=+1358.346698117" watchObservedRunningTime="2026-10-31 00:37:45.881099197 +0000 UTC m=+1358.352268947" Oct 31 00:37:46.066644 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:46.066544 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afa82603-jr9rb"] Oct 31 00:37:46.075126 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:46.075082 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afa82603-jr9rb"] Oct 31 00:37:46.078946 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:46.078922 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14eec3f8-z5kkm"] Oct 31 00:37:46.082406 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:46.082386 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14eec3f8-z5kkm"] Oct 31 00:37:46.829905 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:46.829696 5222 patch_prober.go:29] interesting pod/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sxdqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Oct 31 00:37:46.829905 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:46.829745 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sxdqn" podUID="05e35f54-2479-4d75-b372-0f10ab6819c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Oct 31 00:37:46.932765 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:46.932732 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0cdffee-6zrmt" Oct 31 00:37:46.979079 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:46.978994 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b1e4efb-7f81-45dd-abde-15b6a07ded61-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b1e4efb-7f81-45dd-abde-15b6a07ded61-maas-api-openshift-ca\") pod \"0b1e4efb-7f81-45dd-abde-15b6a07ded61\" (UID: \"0b1e4efb-7f81-45dd-abde-15b6a07ded61\") " Oct 31 00:37:46.979224 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:46.979158 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b1e4efb-7f81-45dd-abde-15b6a07ded61-kube-api-access-xkzcn\" (UniqueName: \"kubernetes.io/projected/0b1e4efb-7f81-45dd-abde-15b6a07ded61-kube-api-access-xkzcn\") pod \"0b1e4efb-7f81-45dd-abde-15b6a07ded61\" (UID: \"0b1e4efb-7f81-45dd-abde-15b6a07ded61\") " Oct 31 00:37:46.979345 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:46.979327 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b1e4efb-7f81-45dd-abde-15b6a07ded61-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b1e4efb-7f81-45dd-abde-15b6a07ded61-maas-api-xks-ca\") pod \"0b1e4efb-7f81-45dd-abde-15b6a07ded61\" (UID: \"0b1e4efb-7f81-45dd-abde-15b6a07ded61\") " Oct 31 00:37:46.979428 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:46.979405 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b1e4efb-7f81-45dd-abde-15b6a07ded61-maas-api-openshift-ca" pod "0b1e4efb-7f81-45dd-abde-15b6a07ded61" (UID: "0b1e4efb-7f81-45dd-abde-15b6a07ded61"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:37:46.979859 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:46.979769 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b1e4efb-7f81-45dd-abde-15b6a07ded61-maas-api-openshift-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:46.981477 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:46.981453 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b1e4efb-7f81-45dd-abde-15b6a07ded61-kube-api-access-xkzcn" pod "0b1e4efb-7f81-45dd-abde-15b6a07ded61" (UID: "0b1e4efb-7f81-45dd-abde-15b6a07ded61"). InnerVolumeSpecName "kube-api-access-xkzcn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:37:46.981477 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:46.981462 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b1e4efb-7f81-45dd-abde-15b6a07ded61-maas-api-xks-ca" pod "0b1e4efb-7f81-45dd-abde-15b6a07ded61" (UID: "0b1e4efb-7f81-45dd-abde-15b6a07ded61"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:37:47.080841 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:47.080808 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b1e4efb-7f81-45dd-abde-15b6a07ded61-maas-api-xks-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:47.080841 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:47.080839 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkzcn\" (UniqueName: \"kubernetes.io/projected/0b1e4efb-7f81-45dd-abde-15b6a07ded61-kube-api-access-xkzcn\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:47.675487 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:47.675452 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62140133-beb2-49fc-a4c9-4864ed0dd580" path="/var/lib/kubelet/pods/62140133-beb2-49fc-a4c9-4864ed0dd580/volumes" Oct 31 00:37:47.675940 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:47.675927 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d22f0a86-6e20-4f79-8104-b47a4431d338" path="/var/lib/kubelet/pods/d22f0a86-6e20-4f79-8104-b47a4431d338/volumes" Oct 31 00:37:47.739398 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:47.738517 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0cdffee-6zrmt" Oct 31 00:37:47.739398 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:47.739125 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0cdffee-6zrmt" event={"ID":"0b1e4efb-7f81-45dd-abde-15b6a07ded61","Type":"ContainerDied","Data":"4eb071eeca2a41370ddcb410d9bdf42802965a2bff2006fa5b23ff74455730d4"} Oct 31 00:37:47.739398 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:47.739167 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4eb071eeca2a41370ddcb410d9bdf42802965a2bff2006fa5b23ff74455730d4" Oct 31 00:37:47.828474 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:47.828438 5222 patch_prober.go:29] interesting pod/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sxdqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Oct 31 00:37:47.828675 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:47.828512 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sxdqn" podUID="05e35f54-2479-4d75-b372-0f10ab6819c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Oct 31 00:37:48.830148 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:48.830017 5222 patch_prober.go:29] interesting pod/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sxdqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Oct 31 00:37:48.837501 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:48.835247 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sxdqn" podUID="05e35f54-2479-4d75-b372-0f10ab6819c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Oct 31 00:37:49.828363 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:49.828315 5222 patch_prober.go:29] interesting pod/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sxdqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Oct 31 00:37:49.828558 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:49.828400 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sxdqn" podUID="05e35f54-2479-4d75-b372-0f10ab6819c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Oct 31 00:37:50.831537 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:50.828829 5222 patch_prober.go:29] interesting pod/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sxdqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Oct 31 00:37:50.831537 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:50.828894 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sxdqn" podUID="05e35f54-2479-4d75-b372-0f10ab6819c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Oct 31 00:37:51.533223 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:51.533188 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-26fb7237-7878f946bc-j9xgk"] Oct 31 00:37:51.533779 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:51.533764 5222 cpu_manager.go:414] "containerMap: removing container" podUID="67d02085-6466-428c-8315-ab10655f0dd2" containerName="revoke-keys" Oct 31 00:37:51.533830 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:51.533781 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67d02085-6466-428c-8315-ab10655f0dd2" containerName="revoke-keys" Oct 31 00:37:51.533830 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:51.533809 5222 cpu_manager.go:414] "containerMap: removing container" podUID="0b1e4efb-7f81-45dd-abde-15b6a07ded61" containerName="revoke-keys" Oct 31 00:37:51.533830 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:51.533815 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b1e4efb-7f81-45dd-abde-15b6a07ded61" containerName="revoke-keys" Oct 31 00:37:51.533922 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:51.533910 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="67d02085-6466-428c-8315-ab10655f0dd2" containerName="revoke-keys" Oct 31 00:37:51.533922 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:51.533925 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b1e4efb-7f81-45dd-abde-15b6a07ded61" containerName="revoke-keys" Oct 31 00:37:51.538734 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:51.538713 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-26fb7237-7878f946bc-j9xgk" Oct 31 00:37:51.591769 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:51.591735 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-26fb7237-7878f946bc-j9xgk"] Oct 31 00:37:51.675476 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:51.637194 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c507c4c8-d49b-4d8f-9c71-bb8779483055-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-26fb7237-7878f946bc-j9xgk\" (UID: \"c507c4c8-d49b-4d8f-9c71-bb8779483055\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-26fb7237-7878f946bc-j9xgk" Oct 31 00:37:51.700724 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:51.700678 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lt9g6\" (UniqueName: \"kubernetes.io/projected/c507c4c8-d49b-4d8f-9c71-bb8779483055-kube-api-access-lt9g6\") pod \"payload-pre-processing-e2e-migrate-26fb7237-7878f946bc-j9xgk\" (UID: \"c507c4c8-d49b-4d8f-9c71-bb8779483055\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-26fb7237-7878f946bc-j9xgk" Oct 31 00:37:51.713896 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:51.667847 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-26fb7237-dockercfg-4m746\"" Oct 31 00:37:51.714129 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:51.640611 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-26fb7237\"" Oct 31 00:37:51.776416 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:51.774237 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-26fb7237-75b5fbc687-xdh58"] Oct 31 00:37:51.779040 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:51.779008 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-26fb7237-75b5fbc687-xdh58" Oct 31 00:37:51.811409 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:51.810306 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-26fb7237-75b5fbc687-xdh58"] Oct 31 00:37:51.815027 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:51.814983 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c507c4c8-d49b-4d8f-9c71-bb8779483055-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-26fb7237-7878f946bc-j9xgk\" (UID: \"c507c4c8-d49b-4d8f-9c71-bb8779483055\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-26fb7237-7878f946bc-j9xgk" Oct 31 00:37:51.815168 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:51.815128 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nj4np\" (UniqueName: \"kubernetes.io/projected/9d26f843-46bb-4fd0-a33b-81b8b6e67d99-kube-api-access-nj4np\") pod \"payload-processing-e2e-migrate-26fb7237-75b5fbc687-xdh58\" (UID: \"9d26f843-46bb-4fd0-a33b-81b8b6e67d99\") " pod="openshift-ingress/payload-processing-e2e-migrate-26fb7237-75b5fbc687-xdh58" Oct 31 00:37:51.815402 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:51.815346 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d26f843-46bb-4fd0-a33b-81b8b6e67d99-plugins-config-volume\") pod \"payload-processing-e2e-migrate-26fb7237-75b5fbc687-xdh58\" (UID: \"9d26f843-46bb-4fd0-a33b-81b8b6e67d99\") " pod="openshift-ingress/payload-processing-e2e-migrate-26fb7237-75b5fbc687-xdh58" Oct 31 00:37:51.815507 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:51.815433 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lt9g6\" (UniqueName: \"kubernetes.io/projected/c507c4c8-d49b-4d8f-9c71-bb8779483055-kube-api-access-lt9g6\") pod \"payload-pre-processing-e2e-migrate-26fb7237-7878f946bc-j9xgk\" (UID: \"c507c4c8-d49b-4d8f-9c71-bb8779483055\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-26fb7237-7878f946bc-j9xgk" Oct 31 00:37:51.816342 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:51.816142 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c507c4c8-d49b-4d8f-9c71-bb8779483055-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-26fb7237-7878f946bc-j9xgk\" (UID: \"c507c4c8-d49b-4d8f-9c71-bb8779483055\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-26fb7237-7878f946bc-j9xgk" Oct 31 00:37:51.827904 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:51.827877 5222 patch_prober.go:29] interesting pod/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sxdqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Oct 31 00:37:51.828039 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:51.827931 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sxdqn" podUID="05e35f54-2479-4d75-b372-0f10ab6819c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Oct 31 00:37:51.878844 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:51.878805 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lt9g6\" (UniqueName: \"kubernetes.io/projected/c507c4c8-d49b-4d8f-9c71-bb8779483055-kube-api-access-lt9g6\") pod \"payload-pre-processing-e2e-migrate-26fb7237-7878f946bc-j9xgk\" (UID: \"c507c4c8-d49b-4d8f-9c71-bb8779483055\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-26fb7237-7878f946bc-j9xgk" Oct 31 00:37:51.916440 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:51.916393 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nj4np\" (UniqueName: \"kubernetes.io/projected/9d26f843-46bb-4fd0-a33b-81b8b6e67d99-kube-api-access-nj4np\") pod \"payload-processing-e2e-migrate-26fb7237-75b5fbc687-xdh58\" (UID: \"9d26f843-46bb-4fd0-a33b-81b8b6e67d99\") " pod="openshift-ingress/payload-processing-e2e-migrate-26fb7237-75b5fbc687-xdh58" Oct 31 00:37:51.916639 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:51.916460 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d26f843-46bb-4fd0-a33b-81b8b6e67d99-plugins-config-volume\") pod \"payload-processing-e2e-migrate-26fb7237-75b5fbc687-xdh58\" (UID: \"9d26f843-46bb-4fd0-a33b-81b8b6e67d99\") " pod="openshift-ingress/payload-processing-e2e-migrate-26fb7237-75b5fbc687-xdh58" Oct 31 00:37:51.917350 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:51.917328 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d26f843-46bb-4fd0-a33b-81b8b6e67d99-plugins-config-volume\") pod \"payload-processing-e2e-migrate-26fb7237-75b5fbc687-xdh58\" (UID: \"9d26f843-46bb-4fd0-a33b-81b8b6e67d99\") " pod="openshift-ingress/payload-processing-e2e-migrate-26fb7237-75b5fbc687-xdh58" Oct 31 00:37:51.980856 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:51.980818 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nj4np\" (UniqueName: \"kubernetes.io/projected/9d26f843-46bb-4fd0-a33b-81b8b6e67d99-kube-api-access-nj4np\") pod \"payload-processing-e2e-migrate-26fb7237-75b5fbc687-xdh58\" (UID: \"9d26f843-46bb-4fd0-a33b-81b8b6e67d99\") " pod="openshift-ingress/payload-processing-e2e-migrate-26fb7237-75b5fbc687-xdh58" Oct 31 00:37:52.031473 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:52.031435 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3088a498-m4ms8"] Oct 31 00:37:52.036605 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:52.036586 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3088a498-m4ms8" Oct 31 00:37:52.089438 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:52.089344 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3088a498-m4ms8"] Oct 31 00:37:52.090842 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:52.090809 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:37:52.091067 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:52.091048 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:37:52.097327 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:52.097302 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-26fb7237-75b5fbc687-xdh58" Oct 31 00:37:52.109679 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:52.109653 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfz9z\"" Oct 31 00:37:52.109978 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:52.109936 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:37:52.118045 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:52.118013 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55f4e5f6-0ba4-4e7f-acef-07b16a1a523e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3088a498-m4ms8\" (UID: \"55f4e5f6-0ba4-4e7f-acef-07b16a1a523e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3088a498-m4ms8" Oct 31 00:37:52.118283 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:52.118257 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55f4e5f6-0ba4-4e7f-acef-07b16a1a523e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3088a498-m4ms8\" (UID: \"55f4e5f6-0ba4-4e7f-acef-07b16a1a523e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3088a498-m4ms8" Oct 31 00:37:52.118415 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:52.118320 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bmzw\" (UniqueName: \"kubernetes.io/projected/55f4e5f6-0ba4-4e7f-acef-07b16a1a523e-kube-api-access-4bmzw\") pod \"maas-api-revoke-sub-3088a498-m4ms8\" (UID: \"55f4e5f6-0ba4-4e7f-acef-07b16a1a523e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3088a498-m4ms8" Oct 31 00:37:52.155672 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:52.155407 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-26fb7237-7878f946bc-j9xgk" Oct 31 00:37:52.221516 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:52.220579 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55f4e5f6-0ba4-4e7f-acef-07b16a1a523e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3088a498-m4ms8\" (UID: \"55f4e5f6-0ba4-4e7f-acef-07b16a1a523e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3088a498-m4ms8" Oct 31 00:37:52.221516 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:52.220651 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bmzw\" (UniqueName: \"kubernetes.io/projected/55f4e5f6-0ba4-4e7f-acef-07b16a1a523e-kube-api-access-4bmzw\") pod \"maas-api-revoke-sub-3088a498-m4ms8\" (UID: \"55f4e5f6-0ba4-4e7f-acef-07b16a1a523e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3088a498-m4ms8" Oct 31 00:37:52.221516 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:52.220694 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55f4e5f6-0ba4-4e7f-acef-07b16a1a523e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3088a498-m4ms8\" (UID: \"55f4e5f6-0ba4-4e7f-acef-07b16a1a523e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3088a498-m4ms8" Oct 31 00:37:52.221828 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:52.221603 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55f4e5f6-0ba4-4e7f-acef-07b16a1a523e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3088a498-m4ms8\" (UID: \"55f4e5f6-0ba4-4e7f-acef-07b16a1a523e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3088a498-m4ms8" Oct 31 00:37:52.227091 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:52.226792 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55f4e5f6-0ba4-4e7f-acef-07b16a1a523e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3088a498-m4ms8\" (UID: \"55f4e5f6-0ba4-4e7f-acef-07b16a1a523e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3088a498-m4ms8" Oct 31 00:37:52.284169 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:52.284137 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bmzw\" (UniqueName: \"kubernetes.io/projected/55f4e5f6-0ba4-4e7f-acef-07b16a1a523e-kube-api-access-4bmzw\") pod \"maas-api-revoke-sub-3088a498-m4ms8\" (UID: \"55f4e5f6-0ba4-4e7f-acef-07b16a1a523e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3088a498-m4ms8" Oct 31 00:37:52.351303 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:52.351192 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3088a498-m4ms8" Oct 31 00:37:52.493315 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:52.493290 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-26fb7237-75b5fbc687-xdh58"] Oct 31 00:37:52.583990 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:37:52.583959 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc507c4c8_d49b_4d8f_9c71_bb8779483055.slice/crio-8ac5c4a4c3480886ee4eb45e6ef11a96b8b4bf81868171b220c74d77fc430db6 WatchSource:0}: Error finding container 8ac5c4a4c3480886ee4eb45e6ef11a96b8b4bf81868171b220c74d77fc430db6: Status 404 returned error can't find the container with id 8ac5c4a4c3480886ee4eb45e6ef11a96b8b4bf81868171b220c74d77fc430db6 Oct 31 00:37:52.590126 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:52.590091 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-26fb7237-7878f946bc-j9xgk"] Oct 31 00:37:52.747780 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:52.747734 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3088a498-m4ms8"] Oct 31 00:37:52.765043 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:37:52.765003 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55f4e5f6_0ba4_4e7f_acef_07b16a1a523e.slice/crio-332baf7907679526edb002fc2c5dbd677ca70354a06a7450664767b73dd63e16 WatchSource:0}: Error finding container 332baf7907679526edb002fc2c5dbd677ca70354a06a7450664767b73dd63e16: Status 404 returned error can't find the container with id 332baf7907679526edb002fc2c5dbd677ca70354a06a7450664767b73dd63e16 Oct 31 00:37:52.942290 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:52.942248 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-26fb7237-7878f946bc-j9xgk" event={"ID":"c507c4c8-d49b-4d8f-9c71-bb8779483055","Type":"ContainerStarted","Data":"abb82a8d7d1b09bf03b5b5d2ed11a89edcc1bc5bc01c7a5050c91e9d6df34387"} Oct 31 00:37:52.942792 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:52.942321 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-26fb7237-7878f946bc-j9xgk" Oct 31 00:37:52.942792 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:52.942345 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-26fb7237-7878f946bc-j9xgk" event={"ID":"c507c4c8-d49b-4d8f-9c71-bb8779483055","Type":"ContainerStarted","Data":"8ac5c4a4c3480886ee4eb45e6ef11a96b8b4bf81868171b220c74d77fc430db6"} Oct 31 00:37:52.944066 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:52.944038 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3088a498-m4ms8" event={"ID":"55f4e5f6-0ba4-4e7f-acef-07b16a1a523e","Type":"ContainerStarted","Data":"332baf7907679526edb002fc2c5dbd677ca70354a06a7450664767b73dd63e16"} Oct 31 00:37:52.946017 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:52.945989 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-26fb7237-75b5fbc687-xdh58" event={"ID":"9d26f843-46bb-4fd0-a33b-81b8b6e67d99","Type":"ContainerStarted","Data":"540bb11f8f5c877f63eafc69d7c2d84063fdc25f48a88c38d3dbc368184dea07"} Oct 31 00:37:52.946115 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:52.946043 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-26fb7237-75b5fbc687-xdh58" event={"ID":"9d26f843-46bb-4fd0-a33b-81b8b6e67d99","Type":"ContainerStarted","Data":"e23b284246844b09dfc501419e5f2a5ce3389f62532311b160d682aa49e4487f"} Oct 31 00:37:52.946115 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:52.946083 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-26fb7237-75b5fbc687-xdh58" Oct 31 00:37:53.026502 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:53.026473 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sxdqn" Oct 31 00:37:53.026916 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:53.026881 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sxdqn" Oct 31 00:37:53.028054 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:53.028032 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sxdqn" Oct 31 00:37:53.067155 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:53.067094 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-26fb7237-7878f946bc-j9xgk" podStartSLOduration=2.06707594 podStartE2EDuration="2.06707594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:37:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:37:53.062600467 +0000 UTC m=+1365.533770219" watchObservedRunningTime="2026-10-31 00:37:53.06707594 +0000 UTC m=+1365.538245693" Oct 31 00:37:53.352223 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:53.352145 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-26fb7237-75b5fbc687-xdh58" podStartSLOduration=2.352124229 podStartE2EDuration="2.352124229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:37:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:37:53.305347615 +0000 UTC m=+1365.776517365" watchObservedRunningTime="2026-10-31 00:37:53.352124229 +0000 UTC m=+1365.823293980" Oct 31 00:37:53.970134 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:53.967534 5222 generic.go:356] "Generic (PLEG): container finished" podID="55f4e5f6-0ba4-4e7f-acef-07b16a1a523e" containerID="36bd3135f4f9451b799b97129e8b50b338c2777e89c27a7e136a1ded6d4ba621" exitCode=0 Oct 31 00:37:53.970134 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:53.968930 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3088a498-m4ms8" event={"ID":"55f4e5f6-0ba4-4e7f-acef-07b16a1a523e","Type":"ContainerDied","Data":"36bd3135f4f9451b799b97129e8b50b338c2777e89c27a7e136a1ded6d4ba621"} Oct 31 00:37:55.137275 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:55.137247 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3088a498-m4ms8" Oct 31 00:37:55.238861 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:55.238770 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55f4e5f6-0ba4-4e7f-acef-07b16a1a523e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55f4e5f6-0ba4-4e7f-acef-07b16a1a523e-maas-api-xks-ca\") pod \"55f4e5f6-0ba4-4e7f-acef-07b16a1a523e\" (UID: \"55f4e5f6-0ba4-4e7f-acef-07b16a1a523e\") " Oct 31 00:37:55.239050 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:55.238965 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55f4e5f6-0ba4-4e7f-acef-07b16a1a523e-kube-api-access-4bmzw\" (UniqueName: \"kubernetes.io/projected/55f4e5f6-0ba4-4e7f-acef-07b16a1a523e-kube-api-access-4bmzw\") pod \"55f4e5f6-0ba4-4e7f-acef-07b16a1a523e\" (UID: \"55f4e5f6-0ba4-4e7f-acef-07b16a1a523e\") " Oct 31 00:37:55.239050 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:55.239031 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55f4e5f6-0ba4-4e7f-acef-07b16a1a523e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55f4e5f6-0ba4-4e7f-acef-07b16a1a523e-maas-api-openshift-ca\") pod \"55f4e5f6-0ba4-4e7f-acef-07b16a1a523e\" (UID: \"55f4e5f6-0ba4-4e7f-acef-07b16a1a523e\") " Oct 31 00:37:55.239409 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:55.239339 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55f4e5f6-0ba4-4e7f-acef-07b16a1a523e-maas-api-openshift-ca" pod "55f4e5f6-0ba4-4e7f-acef-07b16a1a523e" (UID: "55f4e5f6-0ba4-4e7f-acef-07b16a1a523e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:37:55.241206 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:55.241183 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55f4e5f6-0ba4-4e7f-acef-07b16a1a523e-maas-api-xks-ca" pod "55f4e5f6-0ba4-4e7f-acef-07b16a1a523e" (UID: "55f4e5f6-0ba4-4e7f-acef-07b16a1a523e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:37:55.241460 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:55.241440 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55f4e5f6-0ba4-4e7f-acef-07b16a1a523e-kube-api-access-4bmzw" pod "55f4e5f6-0ba4-4e7f-acef-07b16a1a523e" (UID: "55f4e5f6-0ba4-4e7f-acef-07b16a1a523e"). InnerVolumeSpecName "kube-api-access-4bmzw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:37:55.340748 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:55.340712 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bmzw\" (UniqueName: \"kubernetes.io/projected/55f4e5f6-0ba4-4e7f-acef-07b16a1a523e-kube-api-access-4bmzw\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:55.340748 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:55.340741 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55f4e5f6-0ba4-4e7f-acef-07b16a1a523e-maas-api-openshift-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:55.340748 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:55.340751 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55f4e5f6-0ba4-4e7f-acef-07b16a1a523e-maas-api-xks-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:55.993173 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:55.993092 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3088a498-m4ms8" Oct 31 00:37:55.993173 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:55.993133 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3088a498-m4ms8" event={"ID":"55f4e5f6-0ba4-4e7f-acef-07b16a1a523e","Type":"ContainerDied","Data":"332baf7907679526edb002fc2c5dbd677ca70354a06a7450664767b73dd63e16"} Oct 31 00:37:55.993389 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:55.993209 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="332baf7907679526edb002fc2c5dbd677ca70354a06a7450664767b73dd63e16" Oct 31 00:37:57.780196 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:57.780158 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-vkvcx"] Oct 31 00:37:57.781165 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:57.781144 5222 cpu_manager.go:414] "containerMap: removing container" podUID="55f4e5f6-0ba4-4e7f-acef-07b16a1a523e" containerName="revoke-keys" Oct 31 00:37:57.781165 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:57.781163 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55f4e5f6-0ba4-4e7f-acef-07b16a1a523e" containerName="revoke-keys" Oct 31 00:37:57.781334 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:57.781272 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="55f4e5f6-0ba4-4e7f-acef-07b16a1a523e" containerName="revoke-keys" Oct 31 00:37:57.785344 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:57.785326 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-vkvcx" Oct 31 00:37:57.817811 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:57.817764 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-vkvcx"] Oct 31 00:37:57.872475 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:57.872439 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f45187f1-0814-4c12-bdcb-158eeca665f4-istiod-ca-cert\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-vkvcx\" (UID: \"f45187f1-0814-4c12-bdcb-158eeca665f4\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-vkvcx" Oct 31 00:37:57.872640 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:57.872508 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f45187f1-0814-4c12-bdcb-158eeca665f4-workload-certs\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-vkvcx\" (UID: \"f45187f1-0814-4c12-bdcb-158eeca665f4\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-vkvcx" Oct 31 00:37:57.872640 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:57.872613 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7b7s\" (UniqueName: \"kubernetes.io/projected/f45187f1-0814-4c12-bdcb-158eeca665f4-kube-api-access-d7b7s\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-vkvcx\" (UID: \"f45187f1-0814-4c12-bdcb-158eeca665f4\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-vkvcx" Oct 31 00:37:57.872723 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:57.872670 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f45187f1-0814-4c12-bdcb-158eeca665f4-istio-token\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-vkvcx\" (UID: \"f45187f1-0814-4c12-bdcb-158eeca665f4\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-vkvcx" Oct 31 00:37:57.872723 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:57.872709 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f45187f1-0814-4c12-bdcb-158eeca665f4-istio-envoy\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-vkvcx\" (UID: \"f45187f1-0814-4c12-bdcb-158eeca665f4\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-vkvcx" Oct 31 00:37:57.872894 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:57.872872 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f45187f1-0814-4c12-bdcb-158eeca665f4-istio-podinfo\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-vkvcx\" (UID: \"f45187f1-0814-4c12-bdcb-158eeca665f4\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-vkvcx" Oct 31 00:37:57.872932 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:57.872910 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f45187f1-0814-4c12-bdcb-158eeca665f4-workload-socket\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-vkvcx\" (UID: \"f45187f1-0814-4c12-bdcb-158eeca665f4\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-vkvcx" Oct 31 00:37:57.873009 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:57.872994 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f45187f1-0814-4c12-bdcb-158eeca665f4-credential-socket\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-vkvcx\" (UID: \"f45187f1-0814-4c12-bdcb-158eeca665f4\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-vkvcx" Oct 31 00:37:57.873067 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:57.873052 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f45187f1-0814-4c12-bdcb-158eeca665f4-istio-data\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-vkvcx\" (UID: \"f45187f1-0814-4c12-bdcb-158eeca665f4\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-vkvcx" Oct 31 00:37:57.974810 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:57.974757 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f45187f1-0814-4c12-bdcb-158eeca665f4-istiod-ca-cert\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-vkvcx\" (UID: \"f45187f1-0814-4c12-bdcb-158eeca665f4\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-vkvcx" Oct 31 00:37:57.975021 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:57.974828 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f45187f1-0814-4c12-bdcb-158eeca665f4-workload-certs\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-vkvcx\" (UID: \"f45187f1-0814-4c12-bdcb-158eeca665f4\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-vkvcx" Oct 31 00:37:57.975021 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:57.974859 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7b7s\" (UniqueName: \"kubernetes.io/projected/f45187f1-0814-4c12-bdcb-158eeca665f4-kube-api-access-d7b7s\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-vkvcx\" (UID: \"f45187f1-0814-4c12-bdcb-158eeca665f4\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-vkvcx" Oct 31 00:37:57.975021 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:57.974892 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f45187f1-0814-4c12-bdcb-158eeca665f4-istio-token\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-vkvcx\" (UID: \"f45187f1-0814-4c12-bdcb-158eeca665f4\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-vkvcx" Oct 31 00:37:57.975021 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:57.974927 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f45187f1-0814-4c12-bdcb-158eeca665f4-istio-envoy\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-vkvcx\" (UID: \"f45187f1-0814-4c12-bdcb-158eeca665f4\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-vkvcx" Oct 31 00:37:57.975021 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:57.974968 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f45187f1-0814-4c12-bdcb-158eeca665f4-istio-podinfo\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-vkvcx\" (UID: \"f45187f1-0814-4c12-bdcb-158eeca665f4\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-vkvcx" Oct 31 00:37:57.975293 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:57.975101 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f45187f1-0814-4c12-bdcb-158eeca665f4-workload-socket\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-vkvcx\" (UID: \"f45187f1-0814-4c12-bdcb-158eeca665f4\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-vkvcx" Oct 31 00:37:57.975293 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:57.975177 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f45187f1-0814-4c12-bdcb-158eeca665f4-credential-socket\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-vkvcx\" (UID: \"f45187f1-0814-4c12-bdcb-158eeca665f4\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-vkvcx" Oct 31 00:37:57.975293 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:57.975239 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f45187f1-0814-4c12-bdcb-158eeca665f4-istio-data\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-vkvcx\" (UID: \"f45187f1-0814-4c12-bdcb-158eeca665f4\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-vkvcx" Oct 31 00:37:57.975459 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:57.975314 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f45187f1-0814-4c12-bdcb-158eeca665f4-workload-certs\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-vkvcx\" (UID: \"f45187f1-0814-4c12-bdcb-158eeca665f4\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-vkvcx" Oct 31 00:37:57.975545 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:57.975515 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f45187f1-0814-4c12-bdcb-158eeca665f4-workload-socket\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-vkvcx\" (UID: \"f45187f1-0814-4c12-bdcb-158eeca665f4\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-vkvcx" Oct 31 00:37:57.975678 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:57.975652 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f45187f1-0814-4c12-bdcb-158eeca665f4-istiod-ca-cert\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-vkvcx\" (UID: \"f45187f1-0814-4c12-bdcb-158eeca665f4\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-vkvcx" Oct 31 00:37:57.975978 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:57.975955 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f45187f1-0814-4c12-bdcb-158eeca665f4-istio-data\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-vkvcx\" (UID: \"f45187f1-0814-4c12-bdcb-158eeca665f4\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-vkvcx" Oct 31 00:37:57.976088 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:57.976068 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f45187f1-0814-4c12-bdcb-158eeca665f4-credential-socket\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-vkvcx\" (UID: \"f45187f1-0814-4c12-bdcb-158eeca665f4\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-vkvcx" Oct 31 00:37:57.977566 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:57.977545 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f45187f1-0814-4c12-bdcb-158eeca665f4-istio-envoy\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-vkvcx\" (UID: \"f45187f1-0814-4c12-bdcb-158eeca665f4\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-vkvcx" Oct 31 00:37:57.977804 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:57.977778 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f45187f1-0814-4c12-bdcb-158eeca665f4-istio-podinfo\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-vkvcx\" (UID: \"f45187f1-0814-4c12-bdcb-158eeca665f4\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-vkvcx" Oct 31 00:37:58.037673 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:58.037603 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f45187f1-0814-4c12-bdcb-158eeca665f4-istio-token\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-vkvcx\" (UID: \"f45187f1-0814-4c12-bdcb-158eeca665f4\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-vkvcx" Oct 31 00:37:58.037673 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:58.037660 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7b7s\" (UniqueName: \"kubernetes.io/projected/f45187f1-0814-4c12-bdcb-158eeca665f4-kube-api-access-d7b7s\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-vkvcx\" (UID: \"f45187f1-0814-4c12-bdcb-158eeca665f4\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-vkvcx" Oct 31 00:37:58.100215 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:58.100175 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-vkvcx" Oct 31 00:37:58.565094 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:37:58.565055 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf45187f1_0814_4c12_bdcb_158eeca665f4.slice/crio-2c763a48762f42eb92ee903b42118beeb1a1c64b77d2d5ddefaca4f09dffbe19 WatchSource:0}: Error finding container 2c763a48762f42eb92ee903b42118beeb1a1c64b77d2d5ddefaca4f09dffbe19: Status 404 returned error can't find the container with id 2c763a48762f42eb92ee903b42118beeb1a1c64b77d2d5ddefaca4f09dffbe19 Oct 31 00:37:58.567691 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:58.567661 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:37:58.567777 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:58.567730 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:37:58.567777 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:58.567766 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:37:58.569222 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:58.569203 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-vkvcx"] Oct 31 00:37:59.020472 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:59.020423 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-vkvcx" event={"ID":"f45187f1-0814-4c12-bdcb-158eeca665f4","Type":"ContainerStarted","Data":"3d55a1d7aa5f49d1d7975b99c891dcd99554b570386df2c326ae130bf1efec56"} Oct 31 00:37:59.020879 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:59.020501 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-vkvcx" event={"ID":"f45187f1-0814-4c12-bdcb-158eeca665f4","Type":"ContainerStarted","Data":"2c763a48762f42eb92ee903b42118beeb1a1c64b77d2d5ddefaca4f09dffbe19"} Oct 31 00:37:59.100597 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:59.100570 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-vkvcx" Oct 31 00:37:59.102072 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:59.102049 5222 patch_prober.go:29] interesting pod/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-vkvcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Oct 31 00:37:59.102208 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:59.102102 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-vkvcx" podUID="f45187f1-0814-4c12-bdcb-158eeca665f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Oct 31 00:37:59.210012 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:37:59.209952 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-vkvcx" podStartSLOduration=2.209937708 podStartE2EDuration="2.209937708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:37:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:37:59.194143641 +0000 UTC m=+1371.665313389" watchObservedRunningTime="2026-10-31 00:37:59.209937708 +0000 UTC m=+1371.681107457" Oct 31 00:38:00.101385 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:00.101344 5222 patch_prober.go:29] interesting pod/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-vkvcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Oct 31 00:38:00.101796 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:00.101442 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-vkvcx" podUID="f45187f1-0814-4c12-bdcb-158eeca665f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Oct 31 00:38:00.885493 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:00.885457 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-349c9c5e-22ggj"] Oct 31 00:38:00.922406 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:00.902276 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-349c9c5e-22ggj" Oct 31 00:38:00.936979 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:00.936948 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-349c9c5e-22ggj"] Oct 31 00:38:00.972706 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:00.972566 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:38:00.972884 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:00.972839 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:38:01.028558 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:01.027634 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfz9z\"" Oct 31 00:38:01.028558 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:01.027807 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:38:01.099406 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:01.094192 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9da6618-94f5-4e0d-9203-16642c36f901-maas-api-xks-ca\") pod \"maas-api-revoke-sub-349c9c5e-22ggj\" (UID: \"c9da6618-94f5-4e0d-9203-16642c36f901\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-349c9c5e-22ggj" Oct 31 00:38:01.099406 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:01.094314 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k26kc\" (UniqueName: \"kubernetes.io/projected/c9da6618-94f5-4e0d-9203-16642c36f901-kube-api-access-k26kc\") pod \"maas-api-revoke-sub-349c9c5e-22ggj\" (UID: \"c9da6618-94f5-4e0d-9203-16642c36f901\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-349c9c5e-22ggj" Oct 31 00:38:01.099406 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:01.094515 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9da6618-94f5-4e0d-9203-16642c36f901-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-349c9c5e-22ggj\" (UID: \"c9da6618-94f5-4e0d-9203-16642c36f901\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-349c9c5e-22ggj" Oct 31 00:38:01.111918 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:01.111809 5222 patch_prober.go:29] interesting pod/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-vkvcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Oct 31 00:38:01.112548 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:01.111895 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-vkvcx" podUID="f45187f1-0814-4c12-bdcb-158eeca665f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Oct 31 00:38:01.114893 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:01.114872 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-49bb6b1a-6d969799c5-6lkcs"] Oct 31 00:38:01.119130 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:01.119112 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-49bb6b1a-6d969799c5-6lkcs" Oct 31 00:38:01.173691 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:01.173605 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-49bb6b1a-6d969799c5-6lkcs"] Oct 31 00:38:01.190395 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:01.186732 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-49bb6b1a-dockercfg-k69ds\"" Oct 31 00:38:01.190395 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:01.187900 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-49bb6b1a\"" Oct 31 00:38:01.212405 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:01.205216 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9da6618-94f5-4e0d-9203-16642c36f901-maas-api-xks-ca\") pod \"maas-api-revoke-sub-349c9c5e-22ggj\" (UID: \"c9da6618-94f5-4e0d-9203-16642c36f901\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-349c9c5e-22ggj" Oct 31 00:38:01.212405 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:01.205326 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k26kc\" (UniqueName: \"kubernetes.io/projected/c9da6618-94f5-4e0d-9203-16642c36f901-kube-api-access-k26kc\") pod \"maas-api-revoke-sub-349c9c5e-22ggj\" (UID: \"c9da6618-94f5-4e0d-9203-16642c36f901\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-349c9c5e-22ggj" Oct 31 00:38:01.212405 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:01.205505 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9da6618-94f5-4e0d-9203-16642c36f901-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-349c9c5e-22ggj\" (UID: \"c9da6618-94f5-4e0d-9203-16642c36f901\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-349c9c5e-22ggj" Oct 31 00:38:01.212405 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:01.206335 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9da6618-94f5-4e0d-9203-16642c36f901-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-349c9c5e-22ggj\" (UID: \"c9da6618-94f5-4e0d-9203-16642c36f901\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-349c9c5e-22ggj" Oct 31 00:38:01.269914 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:01.269882 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9da6618-94f5-4e0d-9203-16642c36f901-maas-api-xks-ca\") pod \"maas-api-revoke-sub-349c9c5e-22ggj\" (UID: \"c9da6618-94f5-4e0d-9203-16642c36f901\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-349c9c5e-22ggj" Oct 31 00:38:01.323502 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:01.321736 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rb95k\" (UniqueName: \"kubernetes.io/projected/0e8bd878-f701-4547-bc62-6ea4de9bb5e0-kube-api-access-rb95k\") pod \"payload-pre-processing-e2e-ait-49bb6b1a-6d969799c5-6lkcs\" (UID: \"0e8bd878-f701-4547-bc62-6ea4de9bb5e0\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-49bb6b1a-6d969799c5-6lkcs" Oct 31 00:38:01.323502 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:01.321929 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e8bd878-f701-4547-bc62-6ea4de9bb5e0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-49bb6b1a-6d969799c5-6lkcs\" (UID: \"0e8bd878-f701-4547-bc62-6ea4de9bb5e0\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-49bb6b1a-6d969799c5-6lkcs" Oct 31 00:38:01.323787 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:01.320457 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-49bb6b1a-775dfc4dd5-7jxf7"] Oct 31 00:38:01.407704 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:01.407666 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k26kc\" (UniqueName: \"kubernetes.io/projected/c9da6618-94f5-4e0d-9203-16642c36f901-kube-api-access-k26kc\") pod \"maas-api-revoke-sub-349c9c5e-22ggj\" (UID: \"c9da6618-94f5-4e0d-9203-16642c36f901\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-349c9c5e-22ggj" Oct 31 00:38:01.410014 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:01.409307 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-49bb6b1a-775dfc4dd5-7jxf7" Oct 31 00:38:01.418571 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:01.413806 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-79tk8"] Oct 31 00:38:01.418571 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:01.418444 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-49bb6b1a-775dfc4dd5-7jxf7"] Oct 31 00:38:01.423134 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:01.423110 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e8bd878-f701-4547-bc62-6ea4de9bb5e0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-49bb6b1a-6d969799c5-6lkcs\" (UID: \"0e8bd878-f701-4547-bc62-6ea4de9bb5e0\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-49bb6b1a-6d969799c5-6lkcs" Oct 31 00:38:01.423492 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:01.423474 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rb95k\" (UniqueName: \"kubernetes.io/projected/0e8bd878-f701-4547-bc62-6ea4de9bb5e0-kube-api-access-rb95k\") pod \"payload-pre-processing-e2e-ait-49bb6b1a-6d969799c5-6lkcs\" (UID: \"0e8bd878-f701-4547-bc62-6ea4de9bb5e0\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-49bb6b1a-6d969799c5-6lkcs" Oct 31 00:38:01.424486 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:01.424439 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e8bd878-f701-4547-bc62-6ea4de9bb5e0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-49bb6b1a-6d969799c5-6lkcs\" (UID: \"0e8bd878-f701-4547-bc62-6ea4de9bb5e0\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-49bb6b1a-6d969799c5-6lkcs" Oct 31 00:38:01.486313 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:01.486282 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rb95k\" (UniqueName: \"kubernetes.io/projected/0e8bd878-f701-4547-bc62-6ea4de9bb5e0-kube-api-access-rb95k\") pod \"payload-pre-processing-e2e-ait-49bb6b1a-6d969799c5-6lkcs\" (UID: \"0e8bd878-f701-4547-bc62-6ea4de9bb5e0\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-49bb6b1a-6d969799c5-6lkcs" Oct 31 00:38:01.525253 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:01.525209 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltklt\" (UniqueName: \"kubernetes.io/projected/2f791ab3-6734-4655-afb6-3267d91c961a-kube-api-access-ltklt\") pod \"payload-processing-e2e-ait-49bb6b1a-775dfc4dd5-7jxf7\" (UID: \"2f791ab3-6734-4655-afb6-3267d91c961a\") " pod="openshift-ingress/payload-processing-e2e-ait-49bb6b1a-775dfc4dd5-7jxf7" Oct 31 00:38:01.525495 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:01.525338 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f791ab3-6734-4655-afb6-3267d91c961a-plugins-config-volume\") pod \"payload-processing-e2e-ait-49bb6b1a-775dfc4dd5-7jxf7\" (UID: \"2f791ab3-6734-4655-afb6-3267d91c961a\") " pod="openshift-ingress/payload-processing-e2e-ait-49bb6b1a-775dfc4dd5-7jxf7" Oct 31 00:38:01.626309 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:01.626270 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltklt\" (UniqueName: \"kubernetes.io/projected/2f791ab3-6734-4655-afb6-3267d91c961a-kube-api-access-ltklt\") pod \"payload-processing-e2e-ait-49bb6b1a-775dfc4dd5-7jxf7\" (UID: \"2f791ab3-6734-4655-afb6-3267d91c961a\") " pod="openshift-ingress/payload-processing-e2e-ait-49bb6b1a-775dfc4dd5-7jxf7" Oct 31 00:38:01.626550 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:01.626333 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f791ab3-6734-4655-afb6-3267d91c961a-plugins-config-volume\") pod \"payload-processing-e2e-ait-49bb6b1a-775dfc4dd5-7jxf7\" (UID: \"2f791ab3-6734-4655-afb6-3267d91c961a\") " pod="openshift-ingress/payload-processing-e2e-ait-49bb6b1a-775dfc4dd5-7jxf7" Oct 31 00:38:01.627027 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:01.627004 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f791ab3-6734-4655-afb6-3267d91c961a-plugins-config-volume\") pod \"payload-processing-e2e-ait-49bb6b1a-775dfc4dd5-7jxf7\" (UID: \"2f791ab3-6734-4655-afb6-3267d91c961a\") " pod="openshift-ingress/payload-processing-e2e-ait-49bb6b1a-775dfc4dd5-7jxf7" Oct 31 00:38:01.694800 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:01.694717 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-349c9c5e-22ggj" Oct 31 00:38:01.711608 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:01.711584 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltklt\" (UniqueName: \"kubernetes.io/projected/2f791ab3-6734-4655-afb6-3267d91c961a-kube-api-access-ltklt\") pod \"payload-processing-e2e-ait-49bb6b1a-775dfc4dd5-7jxf7\" (UID: \"2f791ab3-6734-4655-afb6-3267d91c961a\") " pod="openshift-ingress/payload-processing-e2e-ait-49bb6b1a-775dfc4dd5-7jxf7" Oct 31 00:38:01.733187 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:01.733159 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-49bb6b1a-775dfc4dd5-7jxf7" Oct 31 00:38:01.733399 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:01.733289 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-49bb6b1a-6d969799c5-6lkcs" Oct 31 00:38:02.101108 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:02.101072 5222 patch_prober.go:29] interesting pod/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-vkvcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Oct 31 00:38:02.101259 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:02.101139 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-vkvcx" podUID="f45187f1-0814-4c12-bdcb-158eeca665f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Oct 31 00:38:02.105989 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:38:02.105956 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9da6618_94f5_4e0d_9203_16642c36f901.slice/crio-62c923f84e270a10288e806b1b3908157e92701775a3f5ec406d4cc594c6d27f WatchSource:0}: Error finding container 62c923f84e270a10288e806b1b3908157e92701775a3f5ec406d4cc594c6d27f: Status 404 returned error can't find the container with id 62c923f84e270a10288e806b1b3908157e92701775a3f5ec406d4cc594c6d27f Oct 31 00:38:02.108931 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:02.108856 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-349c9c5e-22ggj"] Oct 31 00:38:02.129312 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:02.129270 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-349c9c5e-22ggj" event={"ID":"c9da6618-94f5-4e0d-9203-16642c36f901","Type":"ContainerStarted","Data":"62c923f84e270a10288e806b1b3908157e92701775a3f5ec406d4cc594c6d27f"} Oct 31 00:38:02.203360 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:38:02.203323 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f791ab3_6734_4655_afb6_3267d91c961a.slice/crio-7be7f03571e3404bb32270fd39236cc647ee860bd878deff85f136cf31949af4 WatchSource:0}: Error finding container 7be7f03571e3404bb32270fd39236cc647ee860bd878deff85f136cf31949af4: Status 404 returned error can't find the container with id 7be7f03571e3404bb32270fd39236cc647ee860bd878deff85f136cf31949af4 Oct 31 00:38:02.207152 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:02.207124 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-49bb6b1a-775dfc4dd5-7jxf7"] Oct 31 00:38:02.223126 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:02.219781 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-49bb6b1a-6d969799c5-6lkcs"] Oct 31 00:38:02.223126 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:38:02.222199 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e8bd878_f701_4547_bc62_6ea4de9bb5e0.slice/crio-5eae1632d6eefcdba16574cc647a89b04b2377d9a683e57862f8f24f1306c962 WatchSource:0}: Error finding container 5eae1632d6eefcdba16574cc647a89b04b2377d9a683e57862f8f24f1306c962: Status 404 returned error can't find the container with id 5eae1632d6eefcdba16574cc647a89b04b2377d9a683e57862f8f24f1306c962 Oct 31 00:38:03.122397 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:03.112870 5222 patch_prober.go:29] interesting pod/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-vkvcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Oct 31 00:38:03.122397 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:03.112949 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-vkvcx" podUID="f45187f1-0814-4c12-bdcb-158eeca665f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Oct 31 00:38:03.210403 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:03.209521 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-349c9c5e-22ggj" event={"ID":"c9da6618-94f5-4e0d-9203-16642c36f901","Type":"ContainerStarted","Data":"cfc69315fe615a4dcb3175d08c307bd3b167d4ea910e586086d97c115127a80f"} Oct 31 00:38:03.228975 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:03.228125 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-49bb6b1a-775dfc4dd5-7jxf7" event={"ID":"2f791ab3-6734-4655-afb6-3267d91c961a","Type":"ContainerStarted","Data":"ad353db7d100dd0bcfa6bdc6755aa8e35d949be3807c4fd1b9bf05a0ca7e3651"} Oct 31 00:38:03.228975 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:03.228206 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-49bb6b1a-775dfc4dd5-7jxf7" event={"ID":"2f791ab3-6734-4655-afb6-3267d91c961a","Type":"ContainerStarted","Data":"7be7f03571e3404bb32270fd39236cc647ee860bd878deff85f136cf31949af4"} Oct 31 00:38:03.233395 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:03.230382 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-49bb6b1a-775dfc4dd5-7jxf7" Oct 31 00:38:03.256398 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:03.233889 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-49bb6b1a-6d969799c5-6lkcs" event={"ID":"0e8bd878-f701-4547-bc62-6ea4de9bb5e0","Type":"ContainerStarted","Data":"5ece13bb5b292401dac8c5ed51dcc1dd95bbe45f04cbd3dafe3fc25a89c923e6"} Oct 31 00:38:03.256398 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:03.233957 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-49bb6b1a-6d969799c5-6lkcs" event={"ID":"0e8bd878-f701-4547-bc62-6ea4de9bb5e0","Type":"ContainerStarted","Data":"5eae1632d6eefcdba16574cc647a89b04b2377d9a683e57862f8f24f1306c962"} Oct 31 00:38:03.256398 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:03.236858 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-49bb6b1a-6d969799c5-6lkcs" Oct 31 00:38:03.389454 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:03.383819 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-349c9c5e-22ggj" podStartSLOduration=3.383794477 podStartE2EDuration="3.383794477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:38:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:38:03.340772931 +0000 UTC m=+1375.811942680" watchObservedRunningTime="2026-10-31 00:38:03.383794477 +0000 UTC m=+1375.854964226" Oct 31 00:38:03.506143 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:03.505829 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-49bb6b1a-775dfc4dd5-7jxf7" podStartSLOduration=2.505808833 podStartE2EDuration="2.505808833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:38:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:38:03.472444911 +0000 UTC m=+1375.943614659" watchObservedRunningTime="2026-10-31 00:38:03.505808833 +0000 UTC m=+1375.976978583" Oct 31 00:38:03.543432 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:38:03.533677 5222 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9da6618_94f5_4e0d_9203_16642c36f901.slice/crio-conmon-cfc69315fe615a4dcb3175d08c307bd3b167d4ea910e586086d97c115127a80f.scope\": RecentStats: unable to find data in memory cache]" Oct 31 00:38:03.626398 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:03.625149 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-49bb6b1a-6d969799c5-6lkcs" podStartSLOduration=2.62512805 podStartE2EDuration="2.62512805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:38:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:38:03.623726678 +0000 UTC m=+1376.094896467" watchObservedRunningTime="2026-10-31 00:38:03.62512805 +0000 UTC m=+1376.096297816" Oct 31 00:38:04.002395 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:03.988012 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-26fb7237-75b5fbc687-xdh58" Oct 31 00:38:04.002395 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:03.995581 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-26fb7237-7878f946bc-j9xgk" Oct 31 00:38:04.105394 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:04.104803 5222 patch_prober.go:29] interesting pod/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-vkvcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Oct 31 00:38:04.105394 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:04.104867 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-vkvcx" podUID="f45187f1-0814-4c12-bdcb-158eeca665f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Oct 31 00:38:04.465049 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:04.464981 5222 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1e5e75/distinct-w3-1e5e75-kserve-7b55d58bbb-hchmb" podUID="79ffb21b-10c3-4174-89be-05ba9e852a98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:38:04.466091 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:04.465469 5222 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6f4a5/simulator-w6-a6f4a5-kserve-7769689d48-bpkc6" podUID="24079bcb-658a-4109-9b14-a167a38cfac7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:38:04.620750 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:04.620113 5222 generic.go:356] "Generic (PLEG): container finished" podID="c9da6618-94f5-4e0d-9203-16642c36f901" containerID="cfc69315fe615a4dcb3175d08c307bd3b167d4ea910e586086d97c115127a80f" exitCode=0 Oct 31 00:38:04.621757 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:04.621523 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-349c9c5e-22ggj" event={"ID":"c9da6618-94f5-4e0d-9203-16642c36f901","Type":"ContainerDied","Data":"cfc69315fe615a4dcb3175d08c307bd3b167d4ea910e586086d97c115127a80f"} Oct 31 00:38:04.644626 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:04.644562 5222 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da9d07/simulator-w4-da9d07-kserve-7c7588b894-snqq4" podUID="3d37ba9a-a648-44b2-aa04-4acf2b503a08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:38:04.777416 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:04.771709 5222 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-412743/distinct-w5-412743-kserve-64d65f7d8f-c4tfs" podUID="d0e8d5c5-aa08-41f6-badf-792648a20a68" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:38:05.101948 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:05.101852 5222 patch_prober.go:29] interesting pod/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-vkvcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Oct 31 00:38:05.102214 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:05.102181 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-vkvcx" podUID="f45187f1-0814-4c12-bdcb-158eeca665f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Oct 31 00:38:05.799619 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:05.799592 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-349c9c5e-22ggj" Oct 31 00:38:05.892807 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:38:05.892706 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 00:38:05.934117 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:05.934078 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9da6618-94f5-4e0d-9203-16642c36f901-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9da6618-94f5-4e0d-9203-16642c36f901-maas-api-openshift-ca\") pod \"c9da6618-94f5-4e0d-9203-16642c36f901\" (UID: \"c9da6618-94f5-4e0d-9203-16642c36f901\") " Oct 31 00:38:05.934287 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:05.934247 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9da6618-94f5-4e0d-9203-16642c36f901-kube-api-access-k26kc\" (UniqueName: \"kubernetes.io/projected/c9da6618-94f5-4e0d-9203-16642c36f901-kube-api-access-k26kc\") pod \"c9da6618-94f5-4e0d-9203-16642c36f901\" (UID: \"c9da6618-94f5-4e0d-9203-16642c36f901\") " Oct 31 00:38:05.934388 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:05.934294 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9da6618-94f5-4e0d-9203-16642c36f901-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9da6618-94f5-4e0d-9203-16642c36f901-maas-api-xks-ca\") pod \"c9da6618-94f5-4e0d-9203-16642c36f901\" (UID: \"c9da6618-94f5-4e0d-9203-16642c36f901\") " Oct 31 00:38:05.934611 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:05.934554 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9da6618-94f5-4e0d-9203-16642c36f901-maas-api-openshift-ca" pod "c9da6618-94f5-4e0d-9203-16642c36f901" (UID: "c9da6618-94f5-4e0d-9203-16642c36f901"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:38:05.934922 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:05.934896 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9da6618-94f5-4e0d-9203-16642c36f901-maas-api-openshift-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:38:05.936682 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:05.936657 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9da6618-94f5-4e0d-9203-16642c36f901-maas-api-xks-ca" pod "c9da6618-94f5-4e0d-9203-16642c36f901" (UID: "c9da6618-94f5-4e0d-9203-16642c36f901"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:38:05.937104 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:05.937084 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9da6618-94f5-4e0d-9203-16642c36f901-kube-api-access-k26kc" pod "c9da6618-94f5-4e0d-9203-16642c36f901" (UID: "c9da6618-94f5-4e0d-9203-16642c36f901"). InnerVolumeSpecName "kube-api-access-k26kc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:38:06.036588 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:06.036550 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k26kc\" (UniqueName: \"kubernetes.io/projected/c9da6618-94f5-4e0d-9203-16642c36f901-kube-api-access-k26kc\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:38:06.036588 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:06.036582 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9da6618-94f5-4e0d-9203-16642c36f901-maas-api-xks-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:38:06.101484 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:06.101440 5222 patch_prober.go:29] interesting pod/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-vkvcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Oct 31 00:38:06.101688 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:06.101506 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-vkvcx" podUID="f45187f1-0814-4c12-bdcb-158eeca665f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Oct 31 00:38:06.641759 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:06.641725 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-349c9c5e-22ggj" Oct 31 00:38:06.641945 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:06.641754 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-349c9c5e-22ggj" event={"ID":"c9da6618-94f5-4e0d-9203-16642c36f901","Type":"ContainerDied","Data":"62c923f84e270a10288e806b1b3908157e92701775a3f5ec406d4cc594c6d27f"} Oct 31 00:38:06.641945 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:06.641809 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="62c923f84e270a10288e806b1b3908157e92701775a3f5ec406d4cc594c6d27f" Oct 31 00:38:07.105639 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:07.105606 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-vkvcx" Oct 31 00:38:07.106140 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:07.106016 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-vkvcx" Oct 31 00:38:07.106750 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:07.106731 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-vkvcx" Oct 31 00:38:07.925713 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:07.925682 5222 scope.go:122] "RemoveContainer" containerID="9513938b764987ac85d4d4635b2936364bf983688b1d5b4080a9ab5d0fa913b1" Oct 31 00:38:07.936735 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:07.936710 5222 scope.go:122] "RemoveContainer" containerID="4e0667aa29e4ff76f6fe216b47514652c469fd2b5398d90e3c0f7e0f300ee5bb" Oct 31 00:38:07.947785 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:07.947753 5222 scope.go:122] "RemoveContainer" containerID="012e592bd4d58e7011f26601b2c2e7bebe0df450fbe7d4f9c5fa75c3f01de8b6" Oct 31 00:38:07.958425 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:07.958393 5222 scope.go:122] "RemoveContainer" containerID="f991ce90313763e7cca6fa78c7b201564cc071fe2c2b234ac452ceab8171600f" Oct 31 00:38:07.969020 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:07.968992 5222 scope.go:122] "RemoveContainer" containerID="46591766be9065bdf2812ee0a20812d995ca7db03e2257b566edd4223d18c8c8" Oct 31 00:38:07.979810 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:07.979788 5222 scope.go:122] "RemoveContainer" containerID="4ffc3baf8b66849b9896e0bf196049946e39462e71295d5ea20095a85f3fa6d2" Oct 31 00:38:07.990456 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:07.990433 5222 scope.go:122] "RemoveContainer" containerID="2ed38402b1edb1bbc029705953f1a0b94cf4e7b7030fa33f904daa015148df8e" Oct 31 00:38:08.004082 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:08.004060 5222 scope.go:122] "RemoveContainer" containerID="8f58f08d71ab0fdf926cf0be72b607fd43ebeb004875348db78548d204ea059b" Oct 31 00:38:08.014559 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:08.014538 5222 scope.go:122] "RemoveContainer" containerID="8a41330184fa93477ea658d81dc8a739f5af95f3b537b2575b917eff601b9e13" Oct 31 00:38:08.025148 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:08.025127 5222 scope.go:122] "RemoveContainer" containerID="7d7ab68a6eb6e3db473a1401cffd059cec48363481f339d3fb00f01315ad0034" Oct 31 00:38:13.443709 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:13.443671 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-zvppf"] Oct 31 00:38:13.444309 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:13.444295 5222 cpu_manager.go:414] "containerMap: removing container" podUID="c9da6618-94f5-4e0d-9203-16642c36f901" containerName="revoke-keys" Oct 31 00:38:13.444354 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:13.444309 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9da6618-94f5-4e0d-9203-16642c36f901" containerName="revoke-keys" Oct 31 00:38:13.444450 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:13.444438 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9da6618-94f5-4e0d-9203-16642c36f901" containerName="revoke-keys" Oct 31 00:38:13.450850 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:13.450827 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-zvppf" Oct 31 00:38:13.536130 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:13.536085 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d3c7901-be21-4231-b9c9-ae15d4d19f92-istio-token\") pod \"e2e-worker-w3-1e5e75-openshift-default-879c67bc5-zvppf\" (UID: \"4d3c7901-be21-4231-b9c9-ae15d4d19f92\") " pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-zvppf" Oct 31 00:38:13.536323 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:13.536151 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qgw9\" (UniqueName: \"kubernetes.io/projected/4d3c7901-be21-4231-b9c9-ae15d4d19f92-kube-api-access-8qgw9\") pod \"e2e-worker-w3-1e5e75-openshift-default-879c67bc5-zvppf\" (UID: \"4d3c7901-be21-4231-b9c9-ae15d4d19f92\") " pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-zvppf" Oct 31 00:38:13.536323 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:13.536219 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d3c7901-be21-4231-b9c9-ae15d4d19f92-credential-socket\") pod \"e2e-worker-w3-1e5e75-openshift-default-879c67bc5-zvppf\" (UID: \"4d3c7901-be21-4231-b9c9-ae15d4d19f92\") " pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-zvppf" Oct 31 00:38:13.536323 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:13.536257 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d3c7901-be21-4231-b9c9-ae15d4d19f92-workload-certs\") pod \"e2e-worker-w3-1e5e75-openshift-default-879c67bc5-zvppf\" (UID: \"4d3c7901-be21-4231-b9c9-ae15d4d19f92\") " pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-zvppf" Oct 31 00:38:13.536323 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:13.536299 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d3c7901-be21-4231-b9c9-ae15d4d19f92-istio-data\") pod \"e2e-worker-w3-1e5e75-openshift-default-879c67bc5-zvppf\" (UID: \"4d3c7901-be21-4231-b9c9-ae15d4d19f92\") " pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-zvppf" Oct 31 00:38:13.536557 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:13.536358 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d3c7901-be21-4231-b9c9-ae15d4d19f92-workload-socket\") pod \"e2e-worker-w3-1e5e75-openshift-default-879c67bc5-zvppf\" (UID: \"4d3c7901-be21-4231-b9c9-ae15d4d19f92\") " pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-zvppf" Oct 31 00:38:13.536557 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:13.536434 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d3c7901-be21-4231-b9c9-ae15d4d19f92-istiod-ca-cert\") pod \"e2e-worker-w3-1e5e75-openshift-default-879c67bc5-zvppf\" (UID: \"4d3c7901-be21-4231-b9c9-ae15d4d19f92\") " pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-zvppf" Oct 31 00:38:13.536557 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:13.536495 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d3c7901-be21-4231-b9c9-ae15d4d19f92-istio-envoy\") pod \"e2e-worker-w3-1e5e75-openshift-default-879c67bc5-zvppf\" (UID: \"4d3c7901-be21-4231-b9c9-ae15d4d19f92\") " pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-zvppf" Oct 31 00:38:13.536688 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:13.536662 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d3c7901-be21-4231-b9c9-ae15d4d19f92-istio-podinfo\") pod \"e2e-worker-w3-1e5e75-openshift-default-879c67bc5-zvppf\" (UID: \"4d3c7901-be21-4231-b9c9-ae15d4d19f92\") " pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-zvppf" Oct 31 00:38:13.540537 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:13.540510 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-zvppf"] Oct 31 00:38:13.638489 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:13.638453 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d3c7901-be21-4231-b9c9-ae15d4d19f92-credential-socket\") pod \"e2e-worker-w3-1e5e75-openshift-default-879c67bc5-zvppf\" (UID: \"4d3c7901-be21-4231-b9c9-ae15d4d19f92\") " pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-zvppf" Oct 31 00:38:13.638686 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:13.638513 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d3c7901-be21-4231-b9c9-ae15d4d19f92-workload-certs\") pod \"e2e-worker-w3-1e5e75-openshift-default-879c67bc5-zvppf\" (UID: \"4d3c7901-be21-4231-b9c9-ae15d4d19f92\") " pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-zvppf" Oct 31 00:38:13.638686 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:13.638543 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d3c7901-be21-4231-b9c9-ae15d4d19f92-istio-data\") pod \"e2e-worker-w3-1e5e75-openshift-default-879c67bc5-zvppf\" (UID: \"4d3c7901-be21-4231-b9c9-ae15d4d19f92\") " pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-zvppf" Oct 31 00:38:13.638686 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:13.638579 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d3c7901-be21-4231-b9c9-ae15d4d19f92-workload-socket\") pod \"e2e-worker-w3-1e5e75-openshift-default-879c67bc5-zvppf\" (UID: \"4d3c7901-be21-4231-b9c9-ae15d4d19f92\") " pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-zvppf" Oct 31 00:38:13.638686 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:13.638598 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d3c7901-be21-4231-b9c9-ae15d4d19f92-istiod-ca-cert\") pod \"e2e-worker-w3-1e5e75-openshift-default-879c67bc5-zvppf\" (UID: \"4d3c7901-be21-4231-b9c9-ae15d4d19f92\") " pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-zvppf" Oct 31 00:38:13.638686 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:13.638643 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d3c7901-be21-4231-b9c9-ae15d4d19f92-istio-envoy\") pod \"e2e-worker-w3-1e5e75-openshift-default-879c67bc5-zvppf\" (UID: \"4d3c7901-be21-4231-b9c9-ae15d4d19f92\") " pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-zvppf" Oct 31 00:38:13.638958 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:13.638762 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d3c7901-be21-4231-b9c9-ae15d4d19f92-istio-podinfo\") pod \"e2e-worker-w3-1e5e75-openshift-default-879c67bc5-zvppf\" (UID: \"4d3c7901-be21-4231-b9c9-ae15d4d19f92\") " pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-zvppf" Oct 31 00:38:13.638958 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:13.638849 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d3c7901-be21-4231-b9c9-ae15d4d19f92-istio-token\") pod \"e2e-worker-w3-1e5e75-openshift-default-879c67bc5-zvppf\" (UID: \"4d3c7901-be21-4231-b9c9-ae15d4d19f92\") " pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-zvppf" Oct 31 00:38:13.638958 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:13.638916 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qgw9\" (UniqueName: \"kubernetes.io/projected/4d3c7901-be21-4231-b9c9-ae15d4d19f92-kube-api-access-8qgw9\") pod \"e2e-worker-w3-1e5e75-openshift-default-879c67bc5-zvppf\" (UID: \"4d3c7901-be21-4231-b9c9-ae15d4d19f92\") " pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-zvppf" Oct 31 00:38:13.638958 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:13.638928 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d3c7901-be21-4231-b9c9-ae15d4d19f92-credential-socket\") pod \"e2e-worker-w3-1e5e75-openshift-default-879c67bc5-zvppf\" (UID: \"4d3c7901-be21-4231-b9c9-ae15d4d19f92\") " pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-zvppf" Oct 31 00:38:13.638958 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:13.638945 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d3c7901-be21-4231-b9c9-ae15d4d19f92-workload-certs\") pod \"e2e-worker-w3-1e5e75-openshift-default-879c67bc5-zvppf\" (UID: \"4d3c7901-be21-4231-b9c9-ae15d4d19f92\") " pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-zvppf" Oct 31 00:38:13.639210 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:13.639026 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d3c7901-be21-4231-b9c9-ae15d4d19f92-istio-data\") pod \"e2e-worker-w3-1e5e75-openshift-default-879c67bc5-zvppf\" (UID: \"4d3c7901-be21-4231-b9c9-ae15d4d19f92\") " pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-zvppf" Oct 31 00:38:13.639315 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:13.639286 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d3c7901-be21-4231-b9c9-ae15d4d19f92-istiod-ca-cert\") pod \"e2e-worker-w3-1e5e75-openshift-default-879c67bc5-zvppf\" (UID: \"4d3c7901-be21-4231-b9c9-ae15d4d19f92\") " pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-zvppf" Oct 31 00:38:13.639424 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:13.639405 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d3c7901-be21-4231-b9c9-ae15d4d19f92-workload-socket\") pod \"e2e-worker-w3-1e5e75-openshift-default-879c67bc5-zvppf\" (UID: \"4d3c7901-be21-4231-b9c9-ae15d4d19f92\") " pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-zvppf" Oct 31 00:38:13.641269 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:13.641244 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d3c7901-be21-4231-b9c9-ae15d4d19f92-istio-envoy\") pod \"e2e-worker-w3-1e5e75-openshift-default-879c67bc5-zvppf\" (UID: \"4d3c7901-be21-4231-b9c9-ae15d4d19f92\") " pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-zvppf" Oct 31 00:38:13.641621 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:13.641602 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d3c7901-be21-4231-b9c9-ae15d4d19f92-istio-podinfo\") pod \"e2e-worker-w3-1e5e75-openshift-default-879c67bc5-zvppf\" (UID: \"4d3c7901-be21-4231-b9c9-ae15d4d19f92\") " pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-zvppf" Oct 31 00:38:13.725761 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:13.725686 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d3c7901-be21-4231-b9c9-ae15d4d19f92-istio-token\") pod \"e2e-worker-w3-1e5e75-openshift-default-879c67bc5-zvppf\" (UID: \"4d3c7901-be21-4231-b9c9-ae15d4d19f92\") " pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-zvppf" Oct 31 00:38:13.725894 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:13.725792 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qgw9\" (UniqueName: \"kubernetes.io/projected/4d3c7901-be21-4231-b9c9-ae15d4d19f92-kube-api-access-8qgw9\") pod \"e2e-worker-w3-1e5e75-openshift-default-879c67bc5-zvppf\" (UID: \"4d3c7901-be21-4231-b9c9-ae15d4d19f92\") " pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-zvppf" Oct 31 00:38:13.764518 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:13.764484 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-zvppf" Oct 31 00:38:14.093690 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:14.093655 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-4hsbr"] Oct 31 00:38:14.099276 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:14.099255 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-4hsbr" Oct 31 00:38:14.159167 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:14.159132 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-4hsbr"] Oct 31 00:38:14.242441 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:38:14.242405 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d3c7901_be21_4231_b9c9_ae15d4d19f92.slice/crio-0bfd0f49b8fcd76f03f9697d774e6081f3496fd5df80abb805139cd2f2cff18f WatchSource:0}: Error finding container 0bfd0f49b8fcd76f03f9697d774e6081f3496fd5df80abb805139cd2f2cff18f: Status 404 returned error can't find the container with id 0bfd0f49b8fcd76f03f9697d774e6081f3496fd5df80abb805139cd2f2cff18f Oct 31 00:38:14.245163 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:14.245129 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:38:14.245245 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:14.245198 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:38:14.245245 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:14.245227 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:38:14.246762 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:14.246737 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-zvppf"] Oct 31 00:38:14.249225 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:14.249206 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6825ee8a-8ef9-4d40-8046-3e5bd337b6e0-istio-envoy\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-4hsbr\" (UID: \"6825ee8a-8ef9-4d40-8046-3e5bd337b6e0\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-4hsbr" Oct 31 00:38:14.249293 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:14.249244 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6825ee8a-8ef9-4d40-8046-3e5bd337b6e0-istio-data\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-4hsbr\" (UID: \"6825ee8a-8ef9-4d40-8046-3e5bd337b6e0\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-4hsbr" Oct 31 00:38:14.249293 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:14.249262 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6825ee8a-8ef9-4d40-8046-3e5bd337b6e0-istiod-ca-cert\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-4hsbr\" (UID: \"6825ee8a-8ef9-4d40-8046-3e5bd337b6e0\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-4hsbr" Oct 31 00:38:14.249438 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:14.249421 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6825ee8a-8ef9-4d40-8046-3e5bd337b6e0-workload-socket\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-4hsbr\" (UID: \"6825ee8a-8ef9-4d40-8046-3e5bd337b6e0\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-4hsbr" Oct 31 00:38:14.249488 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:14.249476 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6825ee8a-8ef9-4d40-8046-3e5bd337b6e0-istio-token\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-4hsbr\" (UID: \"6825ee8a-8ef9-4d40-8046-3e5bd337b6e0\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-4hsbr" Oct 31 00:38:14.249594 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:14.249570 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6825ee8a-8ef9-4d40-8046-3e5bd337b6e0-credential-socket\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-4hsbr\" (UID: \"6825ee8a-8ef9-4d40-8046-3e5bd337b6e0\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-4hsbr" Oct 31 00:38:14.249674 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:14.249654 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89td9\" (UniqueName: \"kubernetes.io/projected/6825ee8a-8ef9-4d40-8046-3e5bd337b6e0-kube-api-access-89td9\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-4hsbr\" (UID: \"6825ee8a-8ef9-4d40-8046-3e5bd337b6e0\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-4hsbr" Oct 31 00:38:14.249764 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:14.249748 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6825ee8a-8ef9-4d40-8046-3e5bd337b6e0-istio-podinfo\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-4hsbr\" (UID: \"6825ee8a-8ef9-4d40-8046-3e5bd337b6e0\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-4hsbr" Oct 31 00:38:14.249853 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:14.249838 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6825ee8a-8ef9-4d40-8046-3e5bd337b6e0-workload-certs\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-4hsbr\" (UID: \"6825ee8a-8ef9-4d40-8046-3e5bd337b6e0\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-4hsbr" Oct 31 00:38:14.351690 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:14.351601 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6825ee8a-8ef9-4d40-8046-3e5bd337b6e0-credential-socket\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-4hsbr\" (UID: \"6825ee8a-8ef9-4d40-8046-3e5bd337b6e0\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-4hsbr" Oct 31 00:38:14.351847 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:14.351687 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89td9\" (UniqueName: \"kubernetes.io/projected/6825ee8a-8ef9-4d40-8046-3e5bd337b6e0-kube-api-access-89td9\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-4hsbr\" (UID: \"6825ee8a-8ef9-4d40-8046-3e5bd337b6e0\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-4hsbr" Oct 31 00:38:14.351847 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:14.351756 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6825ee8a-8ef9-4d40-8046-3e5bd337b6e0-istio-podinfo\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-4hsbr\" (UID: \"6825ee8a-8ef9-4d40-8046-3e5bd337b6e0\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-4hsbr" Oct 31 00:38:14.351847 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:14.351829 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6825ee8a-8ef9-4d40-8046-3e5bd337b6e0-workload-certs\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-4hsbr\" (UID: \"6825ee8a-8ef9-4d40-8046-3e5bd337b6e0\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-4hsbr" Oct 31 00:38:14.352070 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:14.352035 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6825ee8a-8ef9-4d40-8046-3e5bd337b6e0-credential-socket\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-4hsbr\" (UID: \"6825ee8a-8ef9-4d40-8046-3e5bd337b6e0\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-4hsbr" Oct 31 00:38:14.352070 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:14.352027 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6825ee8a-8ef9-4d40-8046-3e5bd337b6e0-istio-envoy\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-4hsbr\" (UID: \"6825ee8a-8ef9-4d40-8046-3e5bd337b6e0\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-4hsbr" Oct 31 00:38:14.352177 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:14.352112 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6825ee8a-8ef9-4d40-8046-3e5bd337b6e0-istio-data\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-4hsbr\" (UID: \"6825ee8a-8ef9-4d40-8046-3e5bd337b6e0\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-4hsbr" Oct 31 00:38:14.352177 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:14.352138 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6825ee8a-8ef9-4d40-8046-3e5bd337b6e0-istiod-ca-cert\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-4hsbr\" (UID: \"6825ee8a-8ef9-4d40-8046-3e5bd337b6e0\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-4hsbr" Oct 31 00:38:14.352275 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:14.352198 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6825ee8a-8ef9-4d40-8046-3e5bd337b6e0-workload-socket\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-4hsbr\" (UID: \"6825ee8a-8ef9-4d40-8046-3e5bd337b6e0\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-4hsbr" Oct 31 00:38:14.352275 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:14.352195 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6825ee8a-8ef9-4d40-8046-3e5bd337b6e0-workload-certs\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-4hsbr\" (UID: \"6825ee8a-8ef9-4d40-8046-3e5bd337b6e0\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-4hsbr" Oct 31 00:38:14.352416 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:14.352282 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6825ee8a-8ef9-4d40-8046-3e5bd337b6e0-istio-token\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-4hsbr\" (UID: \"6825ee8a-8ef9-4d40-8046-3e5bd337b6e0\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-4hsbr" Oct 31 00:38:14.352485 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:14.352451 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6825ee8a-8ef9-4d40-8046-3e5bd337b6e0-istio-data\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-4hsbr\" (UID: \"6825ee8a-8ef9-4d40-8046-3e5bd337b6e0\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-4hsbr" Oct 31 00:38:14.352555 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:14.352534 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6825ee8a-8ef9-4d40-8046-3e5bd337b6e0-workload-socket\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-4hsbr\" (UID: \"6825ee8a-8ef9-4d40-8046-3e5bd337b6e0\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-4hsbr" Oct 31 00:38:14.352816 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:14.352797 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6825ee8a-8ef9-4d40-8046-3e5bd337b6e0-istiod-ca-cert\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-4hsbr\" (UID: \"6825ee8a-8ef9-4d40-8046-3e5bd337b6e0\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-4hsbr" Oct 31 00:38:14.354912 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:14.354892 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6825ee8a-8ef9-4d40-8046-3e5bd337b6e0-istio-podinfo\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-4hsbr\" (UID: \"6825ee8a-8ef9-4d40-8046-3e5bd337b6e0\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-4hsbr" Oct 31 00:38:14.354993 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:14.354947 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6825ee8a-8ef9-4d40-8046-3e5bd337b6e0-istio-envoy\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-4hsbr\" (UID: \"6825ee8a-8ef9-4d40-8046-3e5bd337b6e0\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-4hsbr" Oct 31 00:38:14.414782 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:14.414739 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6825ee8a-8ef9-4d40-8046-3e5bd337b6e0-istio-token\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-4hsbr\" (UID: \"6825ee8a-8ef9-4d40-8046-3e5bd337b6e0\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-4hsbr" Oct 31 00:38:14.414961 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:14.414787 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89td9\" (UniqueName: \"kubernetes.io/projected/6825ee8a-8ef9-4d40-8046-3e5bd337b6e0-kube-api-access-89td9\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-4hsbr\" (UID: \"6825ee8a-8ef9-4d40-8046-3e5bd337b6e0\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-4hsbr" Oct 31 00:38:14.624756 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:14.624663 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-49bb6b1a-6d969799c5-6lkcs" Oct 31 00:38:14.709773 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:14.709730 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-zvppf" event={"ID":"4d3c7901-be21-4231-b9c9-ae15d4d19f92","Type":"ContainerStarted","Data":"3d57accae2ce7643502e38db5cc656d3d0fdd74e8057b02c8f9d15b435c6d5e7"} Oct 31 00:38:14.709974 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:14.709797 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-zvppf" event={"ID":"4d3c7901-be21-4231-b9c9-ae15d4d19f92","Type":"ContainerStarted","Data":"0bfd0f49b8fcd76f03f9697d774e6081f3496fd5df80abb805139cd2f2cff18f"} Oct 31 00:38:14.714245 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:14.714221 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-4hsbr" Oct 31 00:38:14.785447 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:14.779537 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-zvppf" Oct 31 00:38:14.874628 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:14.874577 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-zvppf" podStartSLOduration=1.874561047 podStartE2EDuration="1.874561047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:38:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:38:14.871711859 +0000 UTC m=+1387.342881610" watchObservedRunningTime="2026-10-31 00:38:14.874561047 +0000 UTC m=+1387.345730796" Oct 31 00:38:15.164465 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:38:15.164424 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6825ee8a_8ef9_4d40_8046_3e5bd337b6e0.slice/crio-f76da1cccd0053be48c1d8b9923abe51c6b80d0e727efa90eb8546b23e2a6527 WatchSource:0}: Error finding container f76da1cccd0053be48c1d8b9923abe51c6b80d0e727efa90eb8546b23e2a6527: Status 404 returned error can't find the container with id f76da1cccd0053be48c1d8b9923abe51c6b80d0e727efa90eb8546b23e2a6527 Oct 31 00:38:15.167743 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:15.167704 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:38:15.167843 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:15.167791 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:38:15.167889 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:15.167842 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:38:15.170241 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:15.170205 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-4hsbr"] Oct 31 00:38:15.633126 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:15.633100 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-49bb6b1a-775dfc4dd5-7jxf7" Oct 31 00:38:15.721739 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:15.721703 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-4hsbr" event={"ID":"6825ee8a-8ef9-4d40-8046-3e5bd337b6e0","Type":"ContainerStarted","Data":"55271c1d80b8c03372e3b9481f2b215a19a47fbec4c26b0c808c432459732daf"} Oct 31 00:38:15.721904 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:15.721755 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-4hsbr" event={"ID":"6825ee8a-8ef9-4d40-8046-3e5bd337b6e0","Type":"ContainerStarted","Data":"f76da1cccd0053be48c1d8b9923abe51c6b80d0e727efa90eb8546b23e2a6527"} Oct 31 00:38:15.771463 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:15.770480 5222 patch_prober.go:29] interesting pod/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-zvppf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Oct 31 00:38:15.771463 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:15.770549 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-zvppf" podUID="4d3c7901-be21-4231-b9c9-ae15d4d19f92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Oct 31 00:38:15.894360 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:15.894256 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-4hsbr" podStartSLOduration=1.894237309 podStartE2EDuration="1.894237309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:38:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:38:15.889007699 +0000 UTC m=+1388.360177462" watchObservedRunningTime="2026-10-31 00:38:15.894237309 +0000 UTC m=+1388.365407060" Oct 31 00:38:16.715452 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:16.715418 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-4hsbr" Oct 31 00:38:16.716855 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:16.716831 5222 patch_prober.go:29] interesting pod/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-4hsbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body= Oct 31 00:38:16.716904 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:16.716879 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-4hsbr" podUID="6825ee8a-8ef9-4d40-8046-3e5bd337b6e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" Oct 31 00:38:16.767394 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:16.766792 5222 patch_prober.go:29] interesting pod/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-zvppf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Oct 31 00:38:16.767394 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:16.766856 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-zvppf" podUID="4d3c7901-be21-4231-b9c9-ae15d4d19f92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Oct 31 00:38:17.715281 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:17.715247 5222 patch_prober.go:29] interesting pod/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-4hsbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body= Oct 31 00:38:17.715502 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:17.715301 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-4hsbr" podUID="6825ee8a-8ef9-4d40-8046-3e5bd337b6e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" Oct 31 00:38:17.771851 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:17.771811 5222 patch_prober.go:29] interesting pod/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-zvppf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Oct 31 00:38:17.772190 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:17.772156 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-zvppf" podUID="4d3c7901-be21-4231-b9c9-ae15d4d19f92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Oct 31 00:38:18.714882 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:18.714714 5222 patch_prober.go:29] interesting pod/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-4hsbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body= Oct 31 00:38:18.714882 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:18.714792 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-4hsbr" podUID="6825ee8a-8ef9-4d40-8046-3e5bd337b6e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" Oct 31 00:38:18.766847 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:18.766707 5222 patch_prober.go:29] interesting pod/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-zvppf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Oct 31 00:38:18.766847 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:18.766784 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-zvppf" podUID="4d3c7901-be21-4231-b9c9-ae15d4d19f92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Oct 31 00:38:19.715386 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:19.715339 5222 patch_prober.go:29] interesting pod/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-4hsbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body= Oct 31 00:38:19.715567 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:19.715417 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-4hsbr" podUID="6825ee8a-8ef9-4d40-8046-3e5bd337b6e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" Oct 31 00:38:19.766396 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:19.765353 5222 patch_prober.go:29] interesting pod/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-zvppf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Oct 31 00:38:19.766396 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:19.765442 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-zvppf" podUID="4d3c7901-be21-4231-b9c9-ae15d4d19f92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Oct 31 00:38:20.715602 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:20.715562 5222 patch_prober.go:29] interesting pod/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-4hsbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body= Oct 31 00:38:20.716021 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:20.715618 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-4hsbr" podUID="6825ee8a-8ef9-4d40-8046-3e5bd337b6e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" Oct 31 00:38:20.765812 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:20.765771 5222 patch_prober.go:29] interesting pod/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-zvppf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Oct 31 00:38:20.766123 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:20.766090 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-zvppf" podUID="4d3c7901-be21-4231-b9c9-ae15d4d19f92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Oct 31 00:38:21.715459 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:21.715422 5222 patch_prober.go:29] interesting pod/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-4hsbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body= Oct 31 00:38:21.715681 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:21.715477 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-4hsbr" podUID="6825ee8a-8ef9-4d40-8046-3e5bd337b6e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" Oct 31 00:38:21.765657 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:21.765620 5222 patch_prober.go:29] interesting pod/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-zvppf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Oct 31 00:38:21.765865 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:21.765681 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-zvppf" podUID="4d3c7901-be21-4231-b9c9-ae15d4d19f92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Oct 31 00:38:22.629560 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:22.629507 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qbn7v"] Oct 31 00:38:22.635181 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:22.635154 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qbn7v" Oct 31 00:38:22.692924 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:22.692874 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qbn7v"] Oct 31 00:38:22.722556 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:22.722509 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7304a135-5e25-401d-b3cc-4503f1c0a794-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-qbn7v\" (UID: \"7304a135-5e25-401d-b3cc-4503f1c0a794\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qbn7v" Oct 31 00:38:22.723019 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:22.722561 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7304a135-5e25-401d-b3cc-4503f1c0a794-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-qbn7v\" (UID: \"7304a135-5e25-401d-b3cc-4503f1c0a794\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qbn7v" Oct 31 00:38:22.723019 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:22.722683 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7304a135-5e25-401d-b3cc-4503f1c0a794-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-qbn7v\" (UID: \"7304a135-5e25-401d-b3cc-4503f1c0a794\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qbn7v" Oct 31 00:38:22.723019 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:22.722730 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7304a135-5e25-401d-b3cc-4503f1c0a794-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-qbn7v\" (UID: \"7304a135-5e25-401d-b3cc-4503f1c0a794\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qbn7v" Oct 31 00:38:22.723019 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:22.722785 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7304a135-5e25-401d-b3cc-4503f1c0a794-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-qbn7v\" (UID: \"7304a135-5e25-401d-b3cc-4503f1c0a794\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qbn7v" Oct 31 00:38:22.723019 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:22.722852 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7304a135-5e25-401d-b3cc-4503f1c0a794-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-qbn7v\" (UID: \"7304a135-5e25-401d-b3cc-4503f1c0a794\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qbn7v" Oct 31 00:38:22.723019 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:22.722907 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvn2d\" (UniqueName: \"kubernetes.io/projected/7304a135-5e25-401d-b3cc-4503f1c0a794-kube-api-access-pvn2d\") pod \"maas-default-gateway-openshift-default-6886b96f9-qbn7v\" (UID: \"7304a135-5e25-401d-b3cc-4503f1c0a794\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qbn7v" Oct 31 00:38:22.723300 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:22.723082 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7304a135-5e25-401d-b3cc-4503f1c0a794-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-qbn7v\" (UID: \"7304a135-5e25-401d-b3cc-4503f1c0a794\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qbn7v" Oct 31 00:38:22.723300 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:22.723257 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7304a135-5e25-401d-b3cc-4503f1c0a794-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-qbn7v\" (UID: \"7304a135-5e25-401d-b3cc-4503f1c0a794\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qbn7v" Oct 31 00:38:22.736093 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:22.736042 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-fxh6q\"" Oct 31 00:38:22.770821 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:22.770786 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-zvppf" Oct 31 00:38:22.771241 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:22.771220 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-zvppf" Oct 31 00:38:22.771909 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:22.771891 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-zvppf" Oct 31 00:38:22.824636 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:22.824592 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvn2d\" (UniqueName: \"kubernetes.io/projected/7304a135-5e25-401d-b3cc-4503f1c0a794-kube-api-access-pvn2d\") pod \"maas-default-gateway-openshift-default-6886b96f9-qbn7v\" (UID: \"7304a135-5e25-401d-b3cc-4503f1c0a794\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qbn7v" Oct 31 00:38:22.824847 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:22.824651 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7304a135-5e25-401d-b3cc-4503f1c0a794-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-qbn7v\" (UID: \"7304a135-5e25-401d-b3cc-4503f1c0a794\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qbn7v" Oct 31 00:38:22.824847 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:22.824727 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7304a135-5e25-401d-b3cc-4503f1c0a794-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-qbn7v\" (UID: \"7304a135-5e25-401d-b3cc-4503f1c0a794\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qbn7v" Oct 31 00:38:22.824847 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:22.824833 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7304a135-5e25-401d-b3cc-4503f1c0a794-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-qbn7v\" (UID: \"7304a135-5e25-401d-b3cc-4503f1c0a794\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qbn7v" Oct 31 00:38:22.825022 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:22.824858 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7304a135-5e25-401d-b3cc-4503f1c0a794-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-qbn7v\" (UID: \"7304a135-5e25-401d-b3cc-4503f1c0a794\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qbn7v" Oct 31 00:38:22.825022 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:22.824903 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7304a135-5e25-401d-b3cc-4503f1c0a794-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-qbn7v\" (UID: \"7304a135-5e25-401d-b3cc-4503f1c0a794\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qbn7v" Oct 31 00:38:22.825022 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:22.824930 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7304a135-5e25-401d-b3cc-4503f1c0a794-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-qbn7v\" (UID: \"7304a135-5e25-401d-b3cc-4503f1c0a794\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qbn7v" Oct 31 00:38:22.825022 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:22.824986 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7304a135-5e25-401d-b3cc-4503f1c0a794-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-qbn7v\" (UID: \"7304a135-5e25-401d-b3cc-4503f1c0a794\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qbn7v" Oct 31 00:38:22.825205 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:22.825065 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7304a135-5e25-401d-b3cc-4503f1c0a794-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-qbn7v\" (UID: \"7304a135-5e25-401d-b3cc-4503f1c0a794\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qbn7v" Oct 31 00:38:22.825245 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:22.825208 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7304a135-5e25-401d-b3cc-4503f1c0a794-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-qbn7v\" (UID: \"7304a135-5e25-401d-b3cc-4503f1c0a794\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qbn7v" Oct 31 00:38:22.825362 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:22.825335 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7304a135-5e25-401d-b3cc-4503f1c0a794-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-qbn7v\" (UID: \"7304a135-5e25-401d-b3cc-4503f1c0a794\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qbn7v" Oct 31 00:38:22.825545 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:22.825521 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7304a135-5e25-401d-b3cc-4503f1c0a794-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-qbn7v\" (UID: \"7304a135-5e25-401d-b3cc-4503f1c0a794\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qbn7v" Oct 31 00:38:22.825765 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:22.825736 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7304a135-5e25-401d-b3cc-4503f1c0a794-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-qbn7v\" (UID: \"7304a135-5e25-401d-b3cc-4503f1c0a794\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qbn7v" Oct 31 00:38:22.826005 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:22.825980 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7304a135-5e25-401d-b3cc-4503f1c0a794-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-qbn7v\" (UID: \"7304a135-5e25-401d-b3cc-4503f1c0a794\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qbn7v" Oct 31 00:38:22.827976 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:22.827957 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7304a135-5e25-401d-b3cc-4503f1c0a794-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-qbn7v\" (UID: \"7304a135-5e25-401d-b3cc-4503f1c0a794\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qbn7v" Oct 31 00:38:22.827976 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:22.827970 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7304a135-5e25-401d-b3cc-4503f1c0a794-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-qbn7v\" (UID: \"7304a135-5e25-401d-b3cc-4503f1c0a794\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qbn7v" Oct 31 00:38:22.888632 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:22.888559 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvn2d\" (UniqueName: \"kubernetes.io/projected/7304a135-5e25-401d-b3cc-4503f1c0a794-kube-api-access-pvn2d\") pod \"maas-default-gateway-openshift-default-6886b96f9-qbn7v\" (UID: \"7304a135-5e25-401d-b3cc-4503f1c0a794\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qbn7v" Oct 31 00:38:22.888632 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:22.888558 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7304a135-5e25-401d-b3cc-4503f1c0a794-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-qbn7v\" (UID: \"7304a135-5e25-401d-b3cc-4503f1c0a794\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qbn7v" Oct 31 00:38:22.964232 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:22.964200 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qbn7v" Oct 31 00:38:22.990152 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:22.989958 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-4hsbr" Oct 31 00:38:22.990554 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:22.990435 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-4hsbr" Oct 31 00:38:22.991858 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:22.991836 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-4hsbr" Oct 31 00:38:23.426518 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:38:23.426444 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7304a135_5e25_401d_b3cc_4503f1c0a794.slice/crio-521c72d371d2835631a3d1ef53426e656a8f2be9a064706b3005f3dc51c2a4a9 WatchSource:0}: Error finding container 521c72d371d2835631a3d1ef53426e656a8f2be9a064706b3005f3dc51c2a4a9: Status 404 returned error can't find the container with id 521c72d371d2835631a3d1ef53426e656a8f2be9a064706b3005f3dc51c2a4a9 Oct 31 00:38:23.428971 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:23.428938 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:38:23.429031 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:23.429007 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:38:23.429069 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:23.429038 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:38:23.430045 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:23.430019 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qbn7v"] Oct 31 00:38:23.852206 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:23.852161 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qbn7v" event={"ID":"7304a135-5e25-401d-b3cc-4503f1c0a794","Type":"ContainerStarted","Data":"9ed89e2df0b3ba97125b027063bfd9aaa236be3b603aa0ff3d178d097bb2ed3e"} Oct 31 00:38:23.852750 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:23.852228 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qbn7v" event={"ID":"7304a135-5e25-401d-b3cc-4503f1c0a794","Type":"ContainerStarted","Data":"521c72d371d2835631a3d1ef53426e656a8f2be9a064706b3005f3dc51c2a4a9"} Oct 31 00:38:23.964979 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:23.964945 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qbn7v" Oct 31 00:38:23.979436 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:23.979347 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qbn7v" podStartSLOduration=1.9793285859999998 podStartE2EDuration="1.979328586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:38:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:38:23.975171795 +0000 UTC m=+1396.446341548" watchObservedRunningTime="2026-10-31 00:38:23.979328586 +0000 UTC m=+1396.450498336" Oct 31 00:38:24.966924 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:24.966884 5222 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-qbn7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Oct 31 00:38:24.967339 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:24.966941 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qbn7v" podUID="7304a135-5e25-401d-b3cc-4503f1c0a794" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Oct 31 00:38:25.965596 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:25.965556 5222 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-qbn7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Oct 31 00:38:25.965806 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:25.965617 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qbn7v" podUID="7304a135-5e25-401d-b3cc-4503f1c0a794" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Oct 31 00:38:26.965223 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:26.965176 5222 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-qbn7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Oct 31 00:38:26.965744 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:26.965234 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qbn7v" podUID="7304a135-5e25-401d-b3cc-4503f1c0a794" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Oct 31 00:38:27.965554 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:27.965515 5222 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-qbn7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Oct 31 00:38:27.966129 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:27.965588 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qbn7v" podUID="7304a135-5e25-401d-b3cc-4503f1c0a794" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Oct 31 00:38:28.965595 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:28.965564 5222 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-qbn7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Oct 31 00:38:28.965994 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:28.965635 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qbn7v" podUID="7304a135-5e25-401d-b3cc-4503f1c0a794" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Oct 31 00:38:29.965386 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:29.965342 5222 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-qbn7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Oct 31 00:38:29.965573 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:29.965440 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qbn7v" podUID="7304a135-5e25-401d-b3cc-4503f1c0a794" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Oct 31 00:38:30.965330 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:30.965285 5222 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-qbn7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Oct 31 00:38:30.965793 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:30.965344 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qbn7v" podUID="7304a135-5e25-401d-b3cc-4503f1c0a794" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Oct 31 00:38:32.217652 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:32.217615 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qbn7v" Oct 31 00:38:32.218855 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:32.218824 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qbn7v" Oct 31 00:38:32.220669 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:32.220647 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qbn7v" Oct 31 00:38:35.891124 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:38:35.891088 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 00:38:48.259548 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:48.259507 5222 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-qbn7v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:38:48.260120 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:48.259591 5222 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qbn7v" podUID="7304a135-5e25-401d-b3cc-4503f1c0a794" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:38:59.255466 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:59.255421 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-hx9qr"] Oct 31 00:38:59.260313 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:59.260288 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-hx9qr" Oct 31 00:38:59.370900 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:59.370870 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-hx9qr"] Oct 31 00:38:59.413188 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:59.413152 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/267e2bb7-9215-4fc3-9874-ae4c2ad13749-istio-envoy\") pod \"e2e-worker-w4-da9d07-openshift-default-798d9948bc-hx9qr\" (UID: \"267e2bb7-9215-4fc3-9874-ae4c2ad13749\") " pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-hx9qr" Oct 31 00:38:59.413399 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:59.413197 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/267e2bb7-9215-4fc3-9874-ae4c2ad13749-credential-socket\") pod \"e2e-worker-w4-da9d07-openshift-default-798d9948bc-hx9qr\" (UID: \"267e2bb7-9215-4fc3-9874-ae4c2ad13749\") " pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-hx9qr" Oct 31 00:38:59.413399 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:59.413216 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/267e2bb7-9215-4fc3-9874-ae4c2ad13749-istio-token\") pod \"e2e-worker-w4-da9d07-openshift-default-798d9948bc-hx9qr\" (UID: \"267e2bb7-9215-4fc3-9874-ae4c2ad13749\") " pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-hx9qr" Oct 31 00:38:59.413491 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:59.413469 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/267e2bb7-9215-4fc3-9874-ae4c2ad13749-istio-data\") pod \"e2e-worker-w4-da9d07-openshift-default-798d9948bc-hx9qr\" (UID: \"267e2bb7-9215-4fc3-9874-ae4c2ad13749\") " pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-hx9qr" Oct 31 00:38:59.413525 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:59.413512 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/267e2bb7-9215-4fc3-9874-ae4c2ad13749-workload-certs\") pod \"e2e-worker-w4-da9d07-openshift-default-798d9948bc-hx9qr\" (UID: \"267e2bb7-9215-4fc3-9874-ae4c2ad13749\") " pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-hx9qr" Oct 31 00:38:59.413610 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:59.413593 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/267e2bb7-9215-4fc3-9874-ae4c2ad13749-workload-socket\") pod \"e2e-worker-w4-da9d07-openshift-default-798d9948bc-hx9qr\" (UID: \"267e2bb7-9215-4fc3-9874-ae4c2ad13749\") " pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-hx9qr" Oct 31 00:38:59.413658 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:59.413618 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74wrk\" (UniqueName: \"kubernetes.io/projected/267e2bb7-9215-4fc3-9874-ae4c2ad13749-kube-api-access-74wrk\") pod \"e2e-worker-w4-da9d07-openshift-default-798d9948bc-hx9qr\" (UID: \"267e2bb7-9215-4fc3-9874-ae4c2ad13749\") " pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-hx9qr" Oct 31 00:38:59.413658 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:59.413641 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/267e2bb7-9215-4fc3-9874-ae4c2ad13749-istio-podinfo\") pod \"e2e-worker-w4-da9d07-openshift-default-798d9948bc-hx9qr\" (UID: \"267e2bb7-9215-4fc3-9874-ae4c2ad13749\") " pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-hx9qr" Oct 31 00:38:59.413790 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:59.413769 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/267e2bb7-9215-4fc3-9874-ae4c2ad13749-istiod-ca-cert\") pod \"e2e-worker-w4-da9d07-openshift-default-798d9948bc-hx9qr\" (UID: \"267e2bb7-9215-4fc3-9874-ae4c2ad13749\") " pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-hx9qr" Oct 31 00:38:59.518461 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:59.517872 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/267e2bb7-9215-4fc3-9874-ae4c2ad13749-istio-data\") pod \"e2e-worker-w4-da9d07-openshift-default-798d9948bc-hx9qr\" (UID: \"267e2bb7-9215-4fc3-9874-ae4c2ad13749\") " pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-hx9qr" Oct 31 00:38:59.518461 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:59.517948 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/267e2bb7-9215-4fc3-9874-ae4c2ad13749-workload-certs\") pod \"e2e-worker-w4-da9d07-openshift-default-798d9948bc-hx9qr\" (UID: \"267e2bb7-9215-4fc3-9874-ae4c2ad13749\") " pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-hx9qr" Oct 31 00:38:59.518461 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:59.517987 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/267e2bb7-9215-4fc3-9874-ae4c2ad13749-workload-socket\") pod \"e2e-worker-w4-da9d07-openshift-default-798d9948bc-hx9qr\" (UID: \"267e2bb7-9215-4fc3-9874-ae4c2ad13749\") " pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-hx9qr" Oct 31 00:38:59.518461 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:59.518013 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74wrk\" (UniqueName: \"kubernetes.io/projected/267e2bb7-9215-4fc3-9874-ae4c2ad13749-kube-api-access-74wrk\") pod \"e2e-worker-w4-da9d07-openshift-default-798d9948bc-hx9qr\" (UID: \"267e2bb7-9215-4fc3-9874-ae4c2ad13749\") " pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-hx9qr" Oct 31 00:38:59.518461 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:59.518046 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/267e2bb7-9215-4fc3-9874-ae4c2ad13749-istio-podinfo\") pod \"e2e-worker-w4-da9d07-openshift-default-798d9948bc-hx9qr\" (UID: \"267e2bb7-9215-4fc3-9874-ae4c2ad13749\") " pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-hx9qr" Oct 31 00:38:59.518461 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:59.518105 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/267e2bb7-9215-4fc3-9874-ae4c2ad13749-istiod-ca-cert\") pod \"e2e-worker-w4-da9d07-openshift-default-798d9948bc-hx9qr\" (UID: \"267e2bb7-9215-4fc3-9874-ae4c2ad13749\") " pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-hx9qr" Oct 31 00:38:59.518461 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:59.518181 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/267e2bb7-9215-4fc3-9874-ae4c2ad13749-istio-envoy\") pod \"e2e-worker-w4-da9d07-openshift-default-798d9948bc-hx9qr\" (UID: \"267e2bb7-9215-4fc3-9874-ae4c2ad13749\") " pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-hx9qr" Oct 31 00:38:59.518461 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:59.518220 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/267e2bb7-9215-4fc3-9874-ae4c2ad13749-credential-socket\") pod \"e2e-worker-w4-da9d07-openshift-default-798d9948bc-hx9qr\" (UID: \"267e2bb7-9215-4fc3-9874-ae4c2ad13749\") " pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-hx9qr" Oct 31 00:38:59.518461 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:59.518247 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/267e2bb7-9215-4fc3-9874-ae4c2ad13749-istio-token\") pod \"e2e-worker-w4-da9d07-openshift-default-798d9948bc-hx9qr\" (UID: \"267e2bb7-9215-4fc3-9874-ae4c2ad13749\") " pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-hx9qr" Oct 31 00:38:59.519150 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:59.518954 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/267e2bb7-9215-4fc3-9874-ae4c2ad13749-istio-data\") pod \"e2e-worker-w4-da9d07-openshift-default-798d9948bc-hx9qr\" (UID: \"267e2bb7-9215-4fc3-9874-ae4c2ad13749\") " pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-hx9qr" Oct 31 00:38:59.519206 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:59.519192 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/267e2bb7-9215-4fc3-9874-ae4c2ad13749-workload-certs\") pod \"e2e-worker-w4-da9d07-openshift-default-798d9948bc-hx9qr\" (UID: \"267e2bb7-9215-4fc3-9874-ae4c2ad13749\") " pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-hx9qr" Oct 31 00:38:59.519460 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:59.519438 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/267e2bb7-9215-4fc3-9874-ae4c2ad13749-workload-socket\") pod \"e2e-worker-w4-da9d07-openshift-default-798d9948bc-hx9qr\" (UID: \"267e2bb7-9215-4fc3-9874-ae4c2ad13749\") " pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-hx9qr" Oct 31 00:38:59.521619 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:59.520578 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/267e2bb7-9215-4fc3-9874-ae4c2ad13749-istiod-ca-cert\") pod \"e2e-worker-w4-da9d07-openshift-default-798d9948bc-hx9qr\" (UID: \"267e2bb7-9215-4fc3-9874-ae4c2ad13749\") " pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-hx9qr" Oct 31 00:38:59.521786 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:59.520820 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/267e2bb7-9215-4fc3-9874-ae4c2ad13749-credential-socket\") pod \"e2e-worker-w4-da9d07-openshift-default-798d9948bc-hx9qr\" (UID: \"267e2bb7-9215-4fc3-9874-ae4c2ad13749\") " pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-hx9qr" Oct 31 00:38:59.527212 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:59.527180 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/267e2bb7-9215-4fc3-9874-ae4c2ad13749-istio-envoy\") pod \"e2e-worker-w4-da9d07-openshift-default-798d9948bc-hx9qr\" (UID: \"267e2bb7-9215-4fc3-9874-ae4c2ad13749\") " pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-hx9qr" Oct 31 00:38:59.528307 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:59.528280 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/267e2bb7-9215-4fc3-9874-ae4c2ad13749-istio-podinfo\") pod \"e2e-worker-w4-da9d07-openshift-default-798d9948bc-hx9qr\" (UID: \"267e2bb7-9215-4fc3-9874-ae4c2ad13749\") " pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-hx9qr" Oct 31 00:38:59.581781 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:59.581742 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/267e2bb7-9215-4fc3-9874-ae4c2ad13749-istio-token\") pod \"e2e-worker-w4-da9d07-openshift-default-798d9948bc-hx9qr\" (UID: \"267e2bb7-9215-4fc3-9874-ae4c2ad13749\") " pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-hx9qr" Oct 31 00:38:59.581940 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:59.581881 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74wrk\" (UniqueName: \"kubernetes.io/projected/267e2bb7-9215-4fc3-9874-ae4c2ad13749-kube-api-access-74wrk\") pod \"e2e-worker-w4-da9d07-openshift-default-798d9948bc-hx9qr\" (UID: \"267e2bb7-9215-4fc3-9874-ae4c2ad13749\") " pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-hx9qr" Oct 31 00:38:59.875403 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:38:59.875284 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-hx9qr" Oct 31 00:39:00.303863 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:39:00.303829 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod267e2bb7_9215_4fc3_9874_ae4c2ad13749.slice/crio-c790a5edfe1ec3b5f18c2fb3313d6fad6c15f92f05788c6edfd60f1d3f8d8ae7 WatchSource:0}: Error finding container c790a5edfe1ec3b5f18c2fb3313d6fad6c15f92f05788c6edfd60f1d3f8d8ae7: Status 404 returned error can't find the container with id c790a5edfe1ec3b5f18c2fb3313d6fad6c15f92f05788c6edfd60f1d3f8d8ae7 Oct 31 00:39:00.306789 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:00.306757 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:39:00.306861 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:00.306823 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:39:00.306861 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:00.306856 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:39:00.310683 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:00.310658 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-hx9qr"] Oct 31 00:39:01.237338 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:01.237296 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-hx9qr" event={"ID":"267e2bb7-9215-4fc3-9874-ae4c2ad13749","Type":"ContainerStarted","Data":"e284b72bce31ba8bf8d2c22f1ea971f1ae0e704f32a8a59ccc0da90903bd74aa"} Oct 31 00:39:01.237588 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:01.237429 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-hx9qr" event={"ID":"267e2bb7-9215-4fc3-9874-ae4c2ad13749","Type":"ContainerStarted","Data":"c790a5edfe1ec3b5f18c2fb3313d6fad6c15f92f05788c6edfd60f1d3f8d8ae7"} Oct 31 00:39:01.482222 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:01.482143 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-hx9qr" podStartSLOduration=2.482125861 podStartE2EDuration="2.482125861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:38:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:39:01.429837661 +0000 UTC m=+1433.901007409" watchObservedRunningTime="2026-10-31 00:39:01.482125861 +0000 UTC m=+1433.953295609" Oct 31 00:39:01.876460 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:01.876415 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-hx9qr" Oct 31 00:39:01.878015 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:01.877988 5222 patch_prober.go:29] interesting pod/e2e-worker-w4-da9d07-openshift-default-798d9948bc-hx9qr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Oct 31 00:39:01.878148 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:01.878051 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-hx9qr" podUID="267e2bb7-9215-4fc3-9874-ae4c2ad13749" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Oct 31 00:39:02.876329 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:02.876293 5222 patch_prober.go:29] interesting pod/e2e-worker-w4-da9d07-openshift-default-798d9948bc-hx9qr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Oct 31 00:39:02.876825 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:02.876356 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-hx9qr" podUID="267e2bb7-9215-4fc3-9874-ae4c2ad13749" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Oct 31 00:39:03.875648 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:03.875614 5222 patch_prober.go:29] interesting pod/e2e-worker-w4-da9d07-openshift-default-798d9948bc-hx9qr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Oct 31 00:39:03.875875 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:03.875666 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-hx9qr" podUID="267e2bb7-9215-4fc3-9874-ae4c2ad13749" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Oct 31 00:39:04.876028 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:04.875986 5222 patch_prober.go:29] interesting pod/e2e-worker-w4-da9d07-openshift-default-798d9948bc-hx9qr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Oct 31 00:39:04.876487 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:04.876058 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-hx9qr" podUID="267e2bb7-9215-4fc3-9874-ae4c2ad13749" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Oct 31 00:39:05.876585 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:05.876551 5222 patch_prober.go:29] interesting pod/e2e-worker-w4-da9d07-openshift-default-798d9948bc-hx9qr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Oct 31 00:39:05.877037 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:05.876615 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-hx9qr" podUID="267e2bb7-9215-4fc3-9874-ae4c2ad13749" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Oct 31 00:39:05.890675 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:39:05.890637 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 00:39:06.876098 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:06.876062 5222 patch_prober.go:29] interesting pod/e2e-worker-w4-da9d07-openshift-default-798d9948bc-hx9qr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Oct 31 00:39:06.876295 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:06.876135 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-hx9qr" podUID="267e2bb7-9215-4fc3-9874-ae4c2ad13749" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Oct 31 00:39:07.880743 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:07.880712 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-hx9qr" Oct 31 00:39:07.881222 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:07.881202 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-hx9qr" Oct 31 00:39:07.882036 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:07.882018 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-hx9qr" Oct 31 00:39:07.993581 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:07.993550 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lkfxl"] Oct 31 00:39:07.998328 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:07.998306 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lkfxl" Oct 31 00:39:08.055809 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:08.055762 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21900006-51c4-44ab-848d-68f885effa5f-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-lkfxl\" (UID: \"21900006-51c4-44ab-848d-68f885effa5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lkfxl" Oct 31 00:39:08.056000 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:08.055832 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21900006-51c4-44ab-848d-68f885effa5f-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-lkfxl\" (UID: \"21900006-51c4-44ab-848d-68f885effa5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lkfxl" Oct 31 00:39:08.056000 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:08.055855 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21900006-51c4-44ab-848d-68f885effa5f-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-lkfxl\" (UID: \"21900006-51c4-44ab-848d-68f885effa5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lkfxl" Oct 31 00:39:08.056000 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:08.055923 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21900006-51c4-44ab-848d-68f885effa5f-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-lkfxl\" (UID: \"21900006-51c4-44ab-848d-68f885effa5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lkfxl" Oct 31 00:39:08.056000 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:08.055973 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21900006-51c4-44ab-848d-68f885effa5f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-lkfxl\" (UID: \"21900006-51c4-44ab-848d-68f885effa5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lkfxl" Oct 31 00:39:08.056242 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:08.056095 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21900006-51c4-44ab-848d-68f885effa5f-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-lkfxl\" (UID: \"21900006-51c4-44ab-848d-68f885effa5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lkfxl" Oct 31 00:39:08.056242 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:08.056134 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21900006-51c4-44ab-848d-68f885effa5f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-lkfxl\" (UID: \"21900006-51c4-44ab-848d-68f885effa5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lkfxl" Oct 31 00:39:08.056242 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:08.056218 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21900006-51c4-44ab-848d-68f885effa5f-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-lkfxl\" (UID: \"21900006-51c4-44ab-848d-68f885effa5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lkfxl" Oct 31 00:39:08.056359 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:08.056280 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9mnp\" (UniqueName: \"kubernetes.io/projected/21900006-51c4-44ab-848d-68f885effa5f-kube-api-access-m9mnp\") pod \"maas-default-gateway-openshift-default-6886b96f9-lkfxl\" (UID: \"21900006-51c4-44ab-848d-68f885effa5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lkfxl" Oct 31 00:39:08.115718 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:08.115683 5222 scope.go:122] "RemoveContainer" containerID="0746566f1cac3816fc98f651c7bfc0750fffbcb76bef5a51d538a0ead6891987" Oct 31 00:39:08.116894 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:08.116862 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lkfxl"] Oct 31 00:39:08.129290 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:08.129260 5222 scope.go:122] "RemoveContainer" containerID="0b5ce50f468b6eda9f9ea7b6ddb322e82a65ab5897aa9dc265225db5758d8a22" Oct 31 00:39:08.140192 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:08.140172 5222 scope.go:122] "RemoveContainer" containerID="ae1fb3a0805b693656a2cab1f3dd851283af6bfaee832b69438f156f74414c12" Oct 31 00:39:08.150529 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:08.150508 5222 scope.go:122] "RemoveContainer" containerID="50ae8c2d47a857f59d86ba7b31f953675e05abd936d50805185fb728f5a43649" Oct 31 00:39:08.157464 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:08.157437 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21900006-51c4-44ab-848d-68f885effa5f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-lkfxl\" (UID: \"21900006-51c4-44ab-848d-68f885effa5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lkfxl" Oct 31 00:39:08.157597 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:08.157486 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21900006-51c4-44ab-848d-68f885effa5f-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-lkfxl\" (UID: \"21900006-51c4-44ab-848d-68f885effa5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lkfxl" Oct 31 00:39:08.157597 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:08.157540 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9mnp\" (UniqueName: \"kubernetes.io/projected/21900006-51c4-44ab-848d-68f885effa5f-kube-api-access-m9mnp\") pod \"maas-default-gateway-openshift-default-6886b96f9-lkfxl\" (UID: \"21900006-51c4-44ab-848d-68f885effa5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lkfxl" Oct 31 00:39:08.157742 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:08.157612 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21900006-51c4-44ab-848d-68f885effa5f-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-lkfxl\" (UID: \"21900006-51c4-44ab-848d-68f885effa5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lkfxl" Oct 31 00:39:08.157742 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:08.157628 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21900006-51c4-44ab-848d-68f885effa5f-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-lkfxl\" (UID: \"21900006-51c4-44ab-848d-68f885effa5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lkfxl" Oct 31 00:39:08.157742 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:08.157652 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21900006-51c4-44ab-848d-68f885effa5f-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-lkfxl\" (UID: \"21900006-51c4-44ab-848d-68f885effa5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lkfxl" Oct 31 00:39:08.157898 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:08.157797 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21900006-51c4-44ab-848d-68f885effa5f-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-lkfxl\" (UID: \"21900006-51c4-44ab-848d-68f885effa5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lkfxl" Oct 31 00:39:08.157958 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:08.157916 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21900006-51c4-44ab-848d-68f885effa5f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-lkfxl\" (UID: \"21900006-51c4-44ab-848d-68f885effa5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lkfxl" Oct 31 00:39:08.157958 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:08.157938 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21900006-51c4-44ab-848d-68f885effa5f-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-lkfxl\" (UID: \"21900006-51c4-44ab-848d-68f885effa5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lkfxl" Oct 31 00:39:08.158076 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:08.158036 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21900006-51c4-44ab-848d-68f885effa5f-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-lkfxl\" (UID: \"21900006-51c4-44ab-848d-68f885effa5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lkfxl" Oct 31 00:39:08.158251 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:08.158227 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21900006-51c4-44ab-848d-68f885effa5f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-lkfxl\" (UID: \"21900006-51c4-44ab-848d-68f885effa5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lkfxl" Oct 31 00:39:08.158342 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:08.158318 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21900006-51c4-44ab-848d-68f885effa5f-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-lkfxl\" (UID: \"21900006-51c4-44ab-848d-68f885effa5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lkfxl" Oct 31 00:39:08.158342 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:08.158282 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21900006-51c4-44ab-848d-68f885effa5f-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-lkfxl\" (UID: \"21900006-51c4-44ab-848d-68f885effa5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lkfxl" Oct 31 00:39:08.158680 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:08.158654 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21900006-51c4-44ab-848d-68f885effa5f-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-lkfxl\" (UID: \"21900006-51c4-44ab-848d-68f885effa5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lkfxl" Oct 31 00:39:08.160080 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:08.160049 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21900006-51c4-44ab-848d-68f885effa5f-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-lkfxl\" (UID: \"21900006-51c4-44ab-848d-68f885effa5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lkfxl" Oct 31 00:39:08.160349 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:08.160332 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21900006-51c4-44ab-848d-68f885effa5f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-lkfxl\" (UID: \"21900006-51c4-44ab-848d-68f885effa5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lkfxl" Oct 31 00:39:08.161550 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:08.161528 5222 scope.go:122] "RemoveContainer" containerID="5baba50f3c9b6095727232636feab713930ac0d1e31e8adde43d0baac0399c7b" Oct 31 00:39:08.202499 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:08.202463 5222 scope.go:122] "RemoveContainer" containerID="26635f2b2b53ff11d5d104155e95ae94721a63a7cf4151fb28a01d8fc721fa79" Oct 31 00:39:08.225062 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:08.225022 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21900006-51c4-44ab-848d-68f885effa5f-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-lkfxl\" (UID: \"21900006-51c4-44ab-848d-68f885effa5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lkfxl" Oct 31 00:39:08.225237 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:08.225033 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9mnp\" (UniqueName: \"kubernetes.io/projected/21900006-51c4-44ab-848d-68f885effa5f-kube-api-access-m9mnp\") pod \"maas-default-gateway-openshift-default-6886b96f9-lkfxl\" (UID: \"21900006-51c4-44ab-848d-68f885effa5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lkfxl" Oct 31 00:39:08.228593 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:08.228568 5222 scope.go:122] "RemoveContainer" containerID="0b96fccce54fcbb8e519b18f3d66eed5339168be902319e84064ae7d936f73b9" Oct 31 00:39:08.259667 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:08.259636 5222 scope.go:122] "RemoveContainer" containerID="f751d5ac35992fbfa181259fc9f6e4ef0f3e8cca8fbb6b4d86ebb2807298c7a1" Oct 31 00:39:08.270074 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:08.270047 5222 scope.go:122] "RemoveContainer" containerID="b36524d123b292dbddd465b82268fffbad40658f525fb1620645fa50bf2bc83a" Oct 31 00:39:08.312707 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:08.312671 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lkfxl" Oct 31 00:39:08.761562 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:39:08.761521 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21900006_51c4_44ab_848d_68f885effa5f.slice/crio-7bb498b4c1301febf43dcef03cca8d286bdbd13629d96c9765be5d5e9a87fecc WatchSource:0}: Error finding container 7bb498b4c1301febf43dcef03cca8d286bdbd13629d96c9765be5d5e9a87fecc: Status 404 returned error can't find the container with id 7bb498b4c1301febf43dcef03cca8d286bdbd13629d96c9765be5d5e9a87fecc Oct 31 00:39:08.764436 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:08.764392 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:39:08.764538 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:08.764492 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:39:08.764538 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:08.764522 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:39:08.766510 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:08.766488 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lkfxl"] Oct 31 00:39:09.357395 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:09.356057 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lkfxl" event={"ID":"21900006-51c4-44ab-848d-68f885effa5f","Type":"ContainerStarted","Data":"b462c5e3696412164e49cd679a7dec56a51ec00d8f2db0ac8e72f6a41edb9bb5"} Oct 31 00:39:09.357395 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:09.356134 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lkfxl" event={"ID":"21900006-51c4-44ab-848d-68f885effa5f","Type":"ContainerStarted","Data":"7bb498b4c1301febf43dcef03cca8d286bdbd13629d96c9765be5d5e9a87fecc"} Oct 31 00:39:09.480053 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:09.479997 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lkfxl" podStartSLOduration=2.479973291 podStartE2EDuration="2.479973291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:39:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:39:09.479169119 +0000 UTC m=+1441.950338869" watchObservedRunningTime="2026-10-31 00:39:09.479973291 +0000 UTC m=+1441.951143039" Oct 31 00:39:10.313977 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:10.313948 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lkfxl" Oct 31 00:39:10.315791 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:10.315765 5222 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-lkfxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Oct 31 00:39:10.315900 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:10.315826 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lkfxl" podUID="21900006-51c4-44ab-848d-68f885effa5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Oct 31 00:39:11.314318 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:11.314286 5222 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-lkfxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Oct 31 00:39:11.314821 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:11.314348 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lkfxl" podUID="21900006-51c4-44ab-848d-68f885effa5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Oct 31 00:39:12.313115 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:12.313079 5222 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-lkfxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Oct 31 00:39:12.313323 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:12.313132 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lkfxl" podUID="21900006-51c4-44ab-848d-68f885effa5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Oct 31 00:39:13.313345 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:13.313307 5222 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-lkfxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Oct 31 00:39:13.313770 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:13.313410 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lkfxl" podUID="21900006-51c4-44ab-848d-68f885effa5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Oct 31 00:39:14.314193 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:14.314156 5222 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-lkfxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Oct 31 00:39:14.314626 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:14.314226 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lkfxl" podUID="21900006-51c4-44ab-848d-68f885effa5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Oct 31 00:39:14.482623 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:14.482550 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-bcc9n"] Oct 31 00:39:14.487745 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:14.487722 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-bcc9n" Oct 31 00:39:14.553394 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:14.548053 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vbgx5"] Oct 31 00:39:14.561929 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:14.561422 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vbgx5" Oct 31 00:39:14.577018 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:14.576823 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc854b8d-c4e0-4355-a059-09b118cf87b7-workload-socket\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-bcc9n\" (UID: \"bc854b8d-c4e0-4355-a059-09b118cf87b7\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-bcc9n" Oct 31 00:39:14.577018 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:14.576895 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dz85\" (UniqueName: \"kubernetes.io/projected/bc854b8d-c4e0-4355-a059-09b118cf87b7-kube-api-access-6dz85\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-bcc9n\" (UID: \"bc854b8d-c4e0-4355-a059-09b118cf87b7\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-bcc9n" Oct 31 00:39:14.577018 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:14.576977 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc854b8d-c4e0-4355-a059-09b118cf87b7-istio-data\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-bcc9n\" (UID: \"bc854b8d-c4e0-4355-a059-09b118cf87b7\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-bcc9n" Oct 31 00:39:14.577300 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:14.577026 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc854b8d-c4e0-4355-a059-09b118cf87b7-credential-socket\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-bcc9n\" (UID: \"bc854b8d-c4e0-4355-a059-09b118cf87b7\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-bcc9n" Oct 31 00:39:14.577300 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:14.577068 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc854b8d-c4e0-4355-a059-09b118cf87b7-workload-certs\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-bcc9n\" (UID: \"bc854b8d-c4e0-4355-a059-09b118cf87b7\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-bcc9n" Oct 31 00:39:14.577300 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:14.577106 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc854b8d-c4e0-4355-a059-09b118cf87b7-istio-envoy\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-bcc9n\" (UID: \"bc854b8d-c4e0-4355-a059-09b118cf87b7\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-bcc9n" Oct 31 00:39:14.577300 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:14.577135 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc854b8d-c4e0-4355-a059-09b118cf87b7-istiod-ca-cert\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-bcc9n\" (UID: \"bc854b8d-c4e0-4355-a059-09b118cf87b7\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-bcc9n" Oct 31 00:39:14.577300 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:14.577283 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc854b8d-c4e0-4355-a059-09b118cf87b7-istio-token\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-bcc9n\" (UID: \"bc854b8d-c4e0-4355-a059-09b118cf87b7\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-bcc9n" Oct 31 00:39:14.577642 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:14.577331 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc854b8d-c4e0-4355-a059-09b118cf87b7-istio-podinfo\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-bcc9n\" (UID: \"bc854b8d-c4e0-4355-a059-09b118cf87b7\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-bcc9n" Oct 31 00:39:14.629477 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:14.629440 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-bcc9n"] Oct 31 00:39:14.673433 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:14.650484 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vbgx5"] Oct 31 00:39:14.678361 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:14.678322 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf-workload-socket\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vbgx5\" (UID: \"1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vbgx5" Oct 31 00:39:14.678571 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:14.678417 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc854b8d-c4e0-4355-a059-09b118cf87b7-credential-socket\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-bcc9n\" (UID: \"bc854b8d-c4e0-4355-a059-09b118cf87b7\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-bcc9n" Oct 31 00:39:14.678571 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:14.678470 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf-istio-token\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vbgx5\" (UID: \"1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vbgx5" Oct 31 00:39:14.678571 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:14.678530 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf-istiod-ca-cert\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vbgx5\" (UID: \"1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vbgx5" Oct 31 00:39:14.678749 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:14.678571 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc854b8d-c4e0-4355-a059-09b118cf87b7-workload-certs\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-bcc9n\" (UID: \"bc854b8d-c4e0-4355-a059-09b118cf87b7\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-bcc9n" Oct 31 00:39:14.678749 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:14.678636 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc854b8d-c4e0-4355-a059-09b118cf87b7-istio-envoy\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-bcc9n\" (UID: \"bc854b8d-c4e0-4355-a059-09b118cf87b7\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-bcc9n" Oct 31 00:39:14.678749 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:14.678670 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc854b8d-c4e0-4355-a059-09b118cf87b7-istiod-ca-cert\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-bcc9n\" (UID: \"bc854b8d-c4e0-4355-a059-09b118cf87b7\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-bcc9n" Oct 31 00:39:14.678749 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:14.678721 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf-credential-socket\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vbgx5\" (UID: \"1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vbgx5" Oct 31 00:39:14.678941 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:14.678902 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc854b8d-c4e0-4355-a059-09b118cf87b7-istio-token\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-bcc9n\" (UID: \"bc854b8d-c4e0-4355-a059-09b118cf87b7\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-bcc9n" Oct 31 00:39:14.678991 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:14.678954 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc854b8d-c4e0-4355-a059-09b118cf87b7-istio-podinfo\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-bcc9n\" (UID: \"bc854b8d-c4e0-4355-a059-09b118cf87b7\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-bcc9n" Oct 31 00:39:14.679045 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:14.678999 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf-istio-podinfo\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vbgx5\" (UID: \"1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vbgx5" Oct 31 00:39:14.679045 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:14.679022 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc854b8d-c4e0-4355-a059-09b118cf87b7-credential-socket\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-bcc9n\" (UID: \"bc854b8d-c4e0-4355-a059-09b118cf87b7\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-bcc9n" Oct 31 00:39:14.679138 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:14.679029 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf-istio-envoy\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vbgx5\" (UID: \"1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vbgx5" Oct 31 00:39:14.679634 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:14.679601 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc854b8d-c4e0-4355-a059-09b118cf87b7-workload-certs\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-bcc9n\" (UID: \"bc854b8d-c4e0-4355-a059-09b118cf87b7\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-bcc9n" Oct 31 00:39:14.679879 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:14.679859 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc854b8d-c4e0-4355-a059-09b118cf87b7-istiod-ca-cert\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-bcc9n\" (UID: \"bc854b8d-c4e0-4355-a059-09b118cf87b7\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-bcc9n" Oct 31 00:39:14.679975 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:14.679889 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf-istio-data\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vbgx5\" (UID: \"1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vbgx5" Oct 31 00:39:14.680036 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:14.680020 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc854b8d-c4e0-4355-a059-09b118cf87b7-workload-socket\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-bcc9n\" (UID: \"bc854b8d-c4e0-4355-a059-09b118cf87b7\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-bcc9n" Oct 31 00:39:14.680091 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:14.680073 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dz85\" (UniqueName: \"kubernetes.io/projected/bc854b8d-c4e0-4355-a059-09b118cf87b7-kube-api-access-6dz85\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-bcc9n\" (UID: \"bc854b8d-c4e0-4355-a059-09b118cf87b7\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-bcc9n" Oct 31 00:39:14.680417 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:14.680138 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf-workload-certs\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vbgx5\" (UID: \"1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vbgx5" Oct 31 00:39:14.680417 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:14.680223 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpp4l\" (UniqueName: \"kubernetes.io/projected/1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf-kube-api-access-qpp4l\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vbgx5\" (UID: \"1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vbgx5" Oct 31 00:39:14.680417 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:14.680256 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc854b8d-c4e0-4355-a059-09b118cf87b7-istio-data\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-bcc9n\" (UID: \"bc854b8d-c4e0-4355-a059-09b118cf87b7\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-bcc9n" Oct 31 00:39:14.680417 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:14.680276 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc854b8d-c4e0-4355-a059-09b118cf87b7-workload-socket\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-bcc9n\" (UID: \"bc854b8d-c4e0-4355-a059-09b118cf87b7\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-bcc9n" Oct 31 00:39:14.680624 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:14.680602 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc854b8d-c4e0-4355-a059-09b118cf87b7-istio-data\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-bcc9n\" (UID: \"bc854b8d-c4e0-4355-a059-09b118cf87b7\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-bcc9n" Oct 31 00:39:14.682988 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:14.682964 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc854b8d-c4e0-4355-a059-09b118cf87b7-istio-podinfo\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-bcc9n\" (UID: \"bc854b8d-c4e0-4355-a059-09b118cf87b7\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-bcc9n" Oct 31 00:39:14.683289 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:14.683268 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc854b8d-c4e0-4355-a059-09b118cf87b7-istio-envoy\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-bcc9n\" (UID: \"bc854b8d-c4e0-4355-a059-09b118cf87b7\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-bcc9n" Oct 31 00:39:14.768474 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:14.766337 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dz85\" (UniqueName: \"kubernetes.io/projected/bc854b8d-c4e0-4355-a059-09b118cf87b7-kube-api-access-6dz85\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-bcc9n\" (UID: \"bc854b8d-c4e0-4355-a059-09b118cf87b7\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-bcc9n" Oct 31 00:39:14.774182 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:14.774138 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc854b8d-c4e0-4355-a059-09b118cf87b7-istio-token\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-bcc9n\" (UID: \"bc854b8d-c4e0-4355-a059-09b118cf87b7\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-bcc9n" Oct 31 00:39:14.781766 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:14.781732 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf-istio-podinfo\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vbgx5\" (UID: \"1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vbgx5" Oct 31 00:39:14.781766 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:14.781767 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf-istio-envoy\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vbgx5\" (UID: \"1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vbgx5" Oct 31 00:39:14.781966 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:14.781827 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf-istio-data\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vbgx5\" (UID: \"1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vbgx5" Oct 31 00:39:14.781966 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:14.781917 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf-workload-certs\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vbgx5\" (UID: \"1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vbgx5" Oct 31 00:39:14.782089 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:14.781969 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpp4l\" (UniqueName: \"kubernetes.io/projected/1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf-kube-api-access-qpp4l\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vbgx5\" (UID: \"1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vbgx5" Oct 31 00:39:14.782089 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:14.782008 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf-workload-socket\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vbgx5\" (UID: \"1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vbgx5" Oct 31 00:39:14.782089 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:14.782052 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf-istio-token\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vbgx5\" (UID: \"1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vbgx5" Oct 31 00:39:14.782089 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:14.782081 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf-istiod-ca-cert\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vbgx5\" (UID: \"1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vbgx5" Oct 31 00:39:14.782302 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:14.782151 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf-credential-socket\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vbgx5\" (UID: \"1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vbgx5" Oct 31 00:39:14.782361 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:14.782308 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf-workload-certs\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vbgx5\" (UID: \"1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vbgx5" Oct 31 00:39:14.782656 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:14.782637 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf-istio-data\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vbgx5\" (UID: \"1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vbgx5" Oct 31 00:39:14.782724 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:14.782634 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf-workload-socket\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vbgx5\" (UID: \"1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vbgx5" Oct 31 00:39:14.782800 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:14.782779 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf-credential-socket\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vbgx5\" (UID: \"1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vbgx5" Oct 31 00:39:14.783035 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:14.783016 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf-istiod-ca-cert\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vbgx5\" (UID: \"1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vbgx5" Oct 31 00:39:14.784275 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:14.784251 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf-istio-envoy\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vbgx5\" (UID: \"1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vbgx5" Oct 31 00:39:14.785853 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:14.785835 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf-istio-podinfo\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vbgx5\" (UID: \"1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vbgx5" Oct 31 00:39:14.803897 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:14.803866 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-bcc9n" Oct 31 00:39:14.845843 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:14.845750 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf-istio-token\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vbgx5\" (UID: \"1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vbgx5" Oct 31 00:39:14.846499 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:14.846472 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpp4l\" (UniqueName: \"kubernetes.io/projected/1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf-kube-api-access-qpp4l\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vbgx5\" (UID: \"1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vbgx5" Oct 31 00:39:14.952111 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:14.952056 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vbgx5" Oct 31 00:39:15.273873 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:39:15.273843 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc854b8d_c4e0_4355_a059_09b118cf87b7.slice/crio-8488e4de7253cf5b53d2a7d5990df7ec2cff16805cca4b980d18fbde1bb53e98 WatchSource:0}: Error finding container 8488e4de7253cf5b53d2a7d5990df7ec2cff16805cca4b980d18fbde1bb53e98: Status 404 returned error can't find the container with id 8488e4de7253cf5b53d2a7d5990df7ec2cff16805cca4b980d18fbde1bb53e98 Oct 31 00:39:15.276357 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:15.276330 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-bcc9n"] Oct 31 00:39:15.277610 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:15.277572 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:39:15.277708 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:15.277653 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:39:15.277708 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:15.277692 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:39:15.314084 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:15.313741 5222 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-lkfxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Oct 31 00:39:15.314084 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:15.313794 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lkfxl" podUID="21900006-51c4-44ab-848d-68f885effa5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Oct 31 00:39:15.355269 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:39:15.355240 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a81dd45_ff89_4738_bf3d_0d3dc0d1feaf.slice/crio-f27a9f499160e5f6fbe8249ec1f098c339c392bcc0147c2dd3d042cb313c5877 WatchSource:0}: Error finding container f27a9f499160e5f6fbe8249ec1f098c339c392bcc0147c2dd3d042cb313c5877: Status 404 returned error can't find the container with id f27a9f499160e5f6fbe8249ec1f098c339c392bcc0147c2dd3d042cb313c5877 Oct 31 00:39:15.358522 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:15.358487 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:39:15.358634 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:15.358573 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:39:15.358634 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:15.358616 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:39:15.362598 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:15.362573 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vbgx5"] Oct 31 00:39:15.418149 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:15.418112 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-bcc9n" event={"ID":"bc854b8d-c4e0-4355-a059-09b118cf87b7","Type":"ContainerStarted","Data":"cf4387b7f512f38cc2ea4e1e3e8a585a653bee5dc49022d7a1a9b40bb40e3895"} Oct 31 00:39:15.418334 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:15.418165 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-bcc9n" event={"ID":"bc854b8d-c4e0-4355-a059-09b118cf87b7","Type":"ContainerStarted","Data":"8488e4de7253cf5b53d2a7d5990df7ec2cff16805cca4b980d18fbde1bb53e98"} Oct 31 00:39:15.421120 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:15.421089 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vbgx5" event={"ID":"1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf","Type":"ContainerStarted","Data":"f27a9f499160e5f6fbe8249ec1f098c339c392bcc0147c2dd3d042cb313c5877"} Oct 31 00:39:15.546498 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:15.546362 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-bcc9n" podStartSLOduration=1.546340132 podStartE2EDuration="1.546340132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:39:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:39:15.543564273 +0000 UTC m=+1448.014734035" watchObservedRunningTime="2026-10-31 00:39:15.546340132 +0000 UTC m=+1448.017509881" Oct 31 00:39:15.804771 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:15.804657 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-bcc9n" Oct 31 00:39:16.320395 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:16.318288 5222 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-lkfxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Oct 31 00:39:16.320395 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:16.318347 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lkfxl" podUID="21900006-51c4-44ab-848d-68f885effa5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Oct 31 00:39:16.432758 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:16.432719 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vbgx5" event={"ID":"1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf","Type":"ContainerStarted","Data":"727faaa62ad980a28fcacfec163bf748ad91159ed70e9b234c57594c0ac17834"} Oct 31 00:39:16.597506 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:16.589207 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vbgx5" podStartSLOduration=2.589183967 podStartE2EDuration="2.589183967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:39:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:39:16.581141039 +0000 UTC m=+1449.052310796" watchObservedRunningTime="2026-10-31 00:39:16.589183967 +0000 UTC m=+1449.060353720" Oct 31 00:39:16.809222 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:16.809181 5222 patch_prober.go:29] interesting pod/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-bcc9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 31 00:39:16.809459 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:16.809268 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-bcc9n" podUID="bc854b8d-c4e0-4355-a059-09b118cf87b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 31 00:39:16.957274 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:16.956715 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vbgx5" Oct 31 00:39:16.957274 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:16.956858 5222 patch_prober.go:29] interesting pod/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vbgx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Oct 31 00:39:16.957274 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:16.956912 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vbgx5" podUID="1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Oct 31 00:39:17.806397 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:17.804799 5222 patch_prober.go:29] interesting pod/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-bcc9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 31 00:39:17.806397 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:17.804864 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-bcc9n" podUID="bc854b8d-c4e0-4355-a059-09b118cf87b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 31 00:39:17.954911 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:17.954874 5222 patch_prober.go:29] interesting pod/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vbgx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Oct 31 00:39:17.955202 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:17.955159 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vbgx5" podUID="1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Oct 31 00:39:18.021693 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:18.021641 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lkfxl" Oct 31 00:39:18.022948 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:18.022879 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lkfxl" Oct 31 00:39:18.023754 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:18.023721 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lkfxl" Oct 31 00:39:18.804729 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:18.804688 5222 patch_prober.go:29] interesting pod/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-bcc9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 31 00:39:18.804990 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:18.804744 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-bcc9n" podUID="bc854b8d-c4e0-4355-a059-09b118cf87b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 31 00:39:18.959394 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:18.958816 5222 patch_prober.go:29] interesting pod/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vbgx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Oct 31 00:39:18.959394 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:18.958883 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vbgx5" podUID="1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Oct 31 00:39:19.535532 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:19.535499 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-79tk8"] Oct 31 00:39:19.804867 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:19.804771 5222 patch_prober.go:29] interesting pod/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-bcc9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 31 00:39:19.805164 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:19.805124 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-bcc9n" podUID="bc854b8d-c4e0-4355-a059-09b118cf87b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 31 00:39:19.956830 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:19.956788 5222 patch_prober.go:29] interesting pod/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vbgx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Oct 31 00:39:19.957108 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:19.957079 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vbgx5" podUID="1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Oct 31 00:39:20.805901 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:20.805866 5222 patch_prober.go:29] interesting pod/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-bcc9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 31 00:39:20.806586 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:20.806554 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-bcc9n" podUID="bc854b8d-c4e0-4355-a059-09b118cf87b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 31 00:39:20.954838 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:20.954796 5222 patch_prober.go:29] interesting pod/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vbgx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Oct 31 00:39:20.955136 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:20.955109 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vbgx5" podUID="1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Oct 31 00:39:21.804350 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:21.804322 5222 patch_prober.go:29] interesting pod/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-bcc9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 31 00:39:21.804647 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:21.804602 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-bcc9n" podUID="bc854b8d-c4e0-4355-a059-09b118cf87b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 31 00:39:21.953871 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:21.953829 5222 patch_prober.go:29] interesting pod/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vbgx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Oct 31 00:39:21.954954 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:21.954514 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vbgx5" podUID="1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Oct 31 00:39:22.805422 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:22.805357 5222 patch_prober.go:29] interesting pod/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-bcc9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 31 00:39:22.805615 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:22.805455 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-bcc9n" podUID="bc854b8d-c4e0-4355-a059-09b118cf87b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 31 00:39:22.952641 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:22.952599 5222 patch_prober.go:29] interesting pod/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vbgx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Oct 31 00:39:22.952828 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:22.952684 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vbgx5" podUID="1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Oct 31 00:39:23.809457 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:23.809421 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-bcc9n" Oct 31 00:39:23.809943 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:23.809927 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-bcc9n" Oct 31 00:39:23.810791 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:23.810773 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-bcc9n" Oct 31 00:39:23.957884 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:23.957852 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vbgx5" Oct 31 00:39:23.958267 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:23.958247 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vbgx5" Oct 31 00:39:23.958921 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:23.958902 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vbgx5" Oct 31 00:39:32.507848 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:32.507812 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-26fb7237-hcqhj"] Oct 31 00:39:32.512669 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:32.512647 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-26fb7237-hcqhj" Oct 31 00:39:32.566270 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:32.566239 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-26fb7237-hcqhj"] Oct 31 00:39:32.566850 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:32.566820 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbmdm\" (UniqueName: \"kubernetes.io/projected/65ee49e4-6fa0-4e7a-991f-ea96cd0f9ae4-kube-api-access-rbmdm\") pod \"maas-api-revoke-keys-e2e-migrate-26fb7237-hcqhj\" (UID: \"65ee49e4-6fa0-4e7a-991f-ea96cd0f9ae4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-26fb7237-hcqhj" Oct 31 00:39:32.566986 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:32.566863 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65ee49e4-6fa0-4e7a-991f-ea96cd0f9ae4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-26fb7237-hcqhj\" (UID: \"65ee49e4-6fa0-4e7a-991f-ea96cd0f9ae4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-26fb7237-hcqhj" Oct 31 00:39:32.567051 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:32.567032 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65ee49e4-6fa0-4e7a-991f-ea96cd0f9ae4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-26fb7237-hcqhj\" (UID: \"65ee49e4-6fa0-4e7a-991f-ea96cd0f9ae4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-26fb7237-hcqhj" Oct 31 00:39:32.611687 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:32.611641 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:39:32.611687 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:32.611665 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:39:32.611687 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:32.611650 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:39:32.611687 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:32.611679 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfz9z\"" Oct 31 00:39:32.668266 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:32.668233 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbmdm\" (UniqueName: \"kubernetes.io/projected/65ee49e4-6fa0-4e7a-991f-ea96cd0f9ae4-kube-api-access-rbmdm\") pod \"maas-api-revoke-keys-e2e-migrate-26fb7237-hcqhj\" (UID: \"65ee49e4-6fa0-4e7a-991f-ea96cd0f9ae4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-26fb7237-hcqhj" Oct 31 00:39:32.668495 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:32.668271 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65ee49e4-6fa0-4e7a-991f-ea96cd0f9ae4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-26fb7237-hcqhj\" (UID: \"65ee49e4-6fa0-4e7a-991f-ea96cd0f9ae4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-26fb7237-hcqhj" Oct 31 00:39:32.668495 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:32.668441 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65ee49e4-6fa0-4e7a-991f-ea96cd0f9ae4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-26fb7237-hcqhj\" (UID: \"65ee49e4-6fa0-4e7a-991f-ea96cd0f9ae4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-26fb7237-hcqhj" Oct 31 00:39:32.669030 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:32.669004 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65ee49e4-6fa0-4e7a-991f-ea96cd0f9ae4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-26fb7237-hcqhj\" (UID: \"65ee49e4-6fa0-4e7a-991f-ea96cd0f9ae4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-26fb7237-hcqhj" Oct 31 00:39:32.672087 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:32.672059 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65ee49e4-6fa0-4e7a-991f-ea96cd0f9ae4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-26fb7237-hcqhj\" (UID: \"65ee49e4-6fa0-4e7a-991f-ea96cd0f9ae4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-26fb7237-hcqhj" Oct 31 00:39:32.731408 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:32.731329 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbmdm\" (UniqueName: \"kubernetes.io/projected/65ee49e4-6fa0-4e7a-991f-ea96cd0f9ae4-kube-api-access-rbmdm\") pod \"maas-api-revoke-keys-e2e-migrate-26fb7237-hcqhj\" (UID: \"65ee49e4-6fa0-4e7a-991f-ea96cd0f9ae4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-26fb7237-hcqhj" Oct 31 00:39:32.827950 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:32.827911 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-26fb7237-hcqhj" Oct 31 00:39:33.208588 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:39:33.208555 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65ee49e4_6fa0_4e7a_991f_ea96cd0f9ae4.slice/crio-3276f51052fc189f95f02cbc628d1ccc9c346d545b5c5dc2d593b2abf1b33d91 WatchSource:0}: Error finding container 3276f51052fc189f95f02cbc628d1ccc9c346d545b5c5dc2d593b2abf1b33d91: Status 404 returned error can't find the container with id 3276f51052fc189f95f02cbc628d1ccc9c346d545b5c5dc2d593b2abf1b33d91 Oct 31 00:39:33.211760 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:33.211682 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-26fb7237-hcqhj"] Oct 31 00:39:33.686409 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:33.686377 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-26fb7237-hcqhj" event={"ID":"65ee49e4-6fa0-4e7a-991f-ea96cd0f9ae4","Type":"ContainerStarted","Data":"61a57c631923b74d92012939a8abafd73a0b78f18152017b0ecc53b5e4ea03b3"} Oct 31 00:39:33.686784 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:33.686428 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-26fb7237-hcqhj" event={"ID":"65ee49e4-6fa0-4e7a-991f-ea96cd0f9ae4","Type":"ContainerStarted","Data":"3276f51052fc189f95f02cbc628d1ccc9c346d545b5c5dc2d593b2abf1b33d91"} Oct 31 00:39:33.808653 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:33.808544 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-26fb7237-hcqhj" podStartSLOduration=1.80852848 podStartE2EDuration="1.80852848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:39:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:39:33.804841311 +0000 UTC m=+1466.276011062" watchObservedRunningTime="2026-10-31 00:39:33.80852848 +0000 UTC m=+1466.279698229" Oct 31 00:39:34.054986 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:34.054954 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e72197c4-vmwck"] Oct 31 00:39:34.059526 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:34.059450 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e72197c4-vmwck"] Oct 31 00:39:34.120867 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:39:34.120830 5222 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65ee49e4_6fa0_4e7a_991f_ea96cd0f9ae4.slice/crio-61a57c631923b74d92012939a8abafd73a0b78f18152017b0ecc53b5e4ea03b3.scope\": RecentStats: unable to find data in memory cache]" Oct 31 00:39:34.695860 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:34.695823 5222 generic.go:356] "Generic (PLEG): container finished" podID="65ee49e4-6fa0-4e7a-991f-ea96cd0f9ae4" containerID="61a57c631923b74d92012939a8abafd73a0b78f18152017b0ecc53b5e4ea03b3" exitCode=0 Oct 31 00:39:34.696269 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:34.695912 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-26fb7237-hcqhj" event={"ID":"65ee49e4-6fa0-4e7a-991f-ea96cd0f9ae4","Type":"ContainerDied","Data":"61a57c631923b74d92012939a8abafd73a0b78f18152017b0ecc53b5e4ea03b3"} Oct 31 00:39:35.658009 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:35.657977 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="731b409e-e01b-42e0-b20b-ce7cfc1edd67" path="/var/lib/kubelet/pods/731b409e-e01b-42e0-b20b-ce7cfc1edd67/volumes" Oct 31 00:39:35.901969 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:39:35.901928 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 00:39:35.963606 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:35.963577 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-26fb7237-hcqhj" Oct 31 00:39:36.064355 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:36.064319 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65ee49e4-6fa0-4e7a-991f-ea96cd0f9ae4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65ee49e4-6fa0-4e7a-991f-ea96cd0f9ae4-maas-api-xks-ca\") pod \"65ee49e4-6fa0-4e7a-991f-ea96cd0f9ae4\" (UID: \"65ee49e4-6fa0-4e7a-991f-ea96cd0f9ae4\") " Oct 31 00:39:36.064567 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:36.064546 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65ee49e4-6fa0-4e7a-991f-ea96cd0f9ae4-kube-api-access-rbmdm\" (UniqueName: \"kubernetes.io/projected/65ee49e4-6fa0-4e7a-991f-ea96cd0f9ae4-kube-api-access-rbmdm\") pod \"65ee49e4-6fa0-4e7a-991f-ea96cd0f9ae4\" (UID: \"65ee49e4-6fa0-4e7a-991f-ea96cd0f9ae4\") " Oct 31 00:39:36.064692 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:36.064677 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65ee49e4-6fa0-4e7a-991f-ea96cd0f9ae4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65ee49e4-6fa0-4e7a-991f-ea96cd0f9ae4-maas-api-openshift-ca\") pod \"65ee49e4-6fa0-4e7a-991f-ea96cd0f9ae4\" (UID: \"65ee49e4-6fa0-4e7a-991f-ea96cd0f9ae4\") " Oct 31 00:39:36.065134 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:36.065065 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65ee49e4-6fa0-4e7a-991f-ea96cd0f9ae4-maas-api-openshift-ca" pod "65ee49e4-6fa0-4e7a-991f-ea96cd0f9ae4" (UID: "65ee49e4-6fa0-4e7a-991f-ea96cd0f9ae4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:39:36.065405 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:36.065283 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65ee49e4-6fa0-4e7a-991f-ea96cd0f9ae4-maas-api-openshift-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:39:36.066749 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:36.066727 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65ee49e4-6fa0-4e7a-991f-ea96cd0f9ae4-maas-api-xks-ca" pod "65ee49e4-6fa0-4e7a-991f-ea96cd0f9ae4" (UID: "65ee49e4-6fa0-4e7a-991f-ea96cd0f9ae4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:39:36.066876 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:36.066856 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65ee49e4-6fa0-4e7a-991f-ea96cd0f9ae4-kube-api-access-rbmdm" pod "65ee49e4-6fa0-4e7a-991f-ea96cd0f9ae4" (UID: "65ee49e4-6fa0-4e7a-991f-ea96cd0f9ae4"). InnerVolumeSpecName "kube-api-access-rbmdm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:39:36.166946 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:36.166903 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65ee49e4-6fa0-4e7a-991f-ea96cd0f9ae4-maas-api-xks-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:39:36.166946 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:36.166938 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbmdm\" (UniqueName: \"kubernetes.io/projected/65ee49e4-6fa0-4e7a-991f-ea96cd0f9ae4-kube-api-access-rbmdm\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:39:36.712973 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:36.712940 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-26fb7237-hcqhj" Oct 31 00:39:36.712973 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:36.712968 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-26fb7237-hcqhj" event={"ID":"65ee49e4-6fa0-4e7a-991f-ea96cd0f9ae4","Type":"ContainerDied","Data":"3276f51052fc189f95f02cbc628d1ccc9c346d545b5c5dc2d593b2abf1b33d91"} Oct 31 00:39:36.713186 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:36.713020 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3276f51052fc189f95f02cbc628d1ccc9c346d545b5c5dc2d593b2abf1b33d91" Oct 31 00:39:42.439646 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:42.439610 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b32e23d-mvndb"] Oct 31 00:39:42.440311 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:42.440292 5222 cpu_manager.go:414] "containerMap: removing container" podUID="65ee49e4-6fa0-4e7a-991f-ea96cd0f9ae4" containerName="revoke-keys" Oct 31 00:39:42.440311 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:42.440312 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65ee49e4-6fa0-4e7a-991f-ea96cd0f9ae4" containerName="revoke-keys" Oct 31 00:39:42.440490 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:42.440479 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="65ee49e4-6fa0-4e7a-991f-ea96cd0f9ae4" containerName="revoke-keys" Oct 31 00:39:42.444383 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:42.444346 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b32e23d-mvndb" Oct 31 00:39:42.496841 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:42.496808 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b32e23d-mvndb"] Oct 31 00:39:42.542766 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:42.542731 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:39:42.542933 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:42.542808 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfz9z\"" Oct 31 00:39:42.549413 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:42.549388 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:39:42.549596 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:42.549579 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:39:42.581124 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:42.581079 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zh72\" (UniqueName: \"kubernetes.io/projected/e232ca5e-2013-4024-8118-5ecdb3088579-kube-api-access-8zh72\") pod \"maas-api-revoke-sub-0b32e23d-mvndb\" (UID: \"e232ca5e-2013-4024-8118-5ecdb3088579\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b32e23d-mvndb" Oct 31 00:39:42.581306 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:42.581199 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e232ca5e-2013-4024-8118-5ecdb3088579-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b32e23d-mvndb\" (UID: \"e232ca5e-2013-4024-8118-5ecdb3088579\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b32e23d-mvndb" Oct 31 00:39:42.581306 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:42.581234 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e232ca5e-2013-4024-8118-5ecdb3088579-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b32e23d-mvndb\" (UID: \"e232ca5e-2013-4024-8118-5ecdb3088579\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b32e23d-mvndb" Oct 31 00:39:42.683092 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:42.683052 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zh72\" (UniqueName: \"kubernetes.io/projected/e232ca5e-2013-4024-8118-5ecdb3088579-kube-api-access-8zh72\") pod \"maas-api-revoke-sub-0b32e23d-mvndb\" (UID: \"e232ca5e-2013-4024-8118-5ecdb3088579\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b32e23d-mvndb" Oct 31 00:39:42.683265 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:42.683130 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e232ca5e-2013-4024-8118-5ecdb3088579-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b32e23d-mvndb\" (UID: \"e232ca5e-2013-4024-8118-5ecdb3088579\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b32e23d-mvndb" Oct 31 00:39:42.683265 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:42.683165 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e232ca5e-2013-4024-8118-5ecdb3088579-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b32e23d-mvndb\" (UID: \"e232ca5e-2013-4024-8118-5ecdb3088579\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b32e23d-mvndb" Oct 31 00:39:42.683840 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:42.683820 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e232ca5e-2013-4024-8118-5ecdb3088579-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b32e23d-mvndb\" (UID: \"e232ca5e-2013-4024-8118-5ecdb3088579\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b32e23d-mvndb" Oct 31 00:39:42.685773 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:42.685749 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e232ca5e-2013-4024-8118-5ecdb3088579-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b32e23d-mvndb\" (UID: \"e232ca5e-2013-4024-8118-5ecdb3088579\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b32e23d-mvndb" Oct 31 00:39:42.745925 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:42.745852 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zh72\" (UniqueName: \"kubernetes.io/projected/e232ca5e-2013-4024-8118-5ecdb3088579-kube-api-access-8zh72\") pod \"maas-api-revoke-sub-0b32e23d-mvndb\" (UID: \"e232ca5e-2013-4024-8118-5ecdb3088579\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b32e23d-mvndb" Oct 31 00:39:42.759040 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:42.759016 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b32e23d-mvndb" Oct 31 00:39:43.142531 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:39:43.142499 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode232ca5e_2013_4024_8118_5ecdb3088579.slice/crio-62f3bdd47ba7b030c361a9618927652be2af01149f647f7802473b3ebdbd2a16 WatchSource:0}: Error finding container 62f3bdd47ba7b030c361a9618927652be2af01149f647f7802473b3ebdbd2a16: Status 404 returned error can't find the container with id 62f3bdd47ba7b030c361a9618927652be2af01149f647f7802473b3ebdbd2a16 Oct 31 00:39:43.147744 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:43.147720 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b32e23d-mvndb"] Oct 31 00:39:43.777421 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:43.777385 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b32e23d-mvndb" event={"ID":"e232ca5e-2013-4024-8118-5ecdb3088579","Type":"ContainerStarted","Data":"ae9d30ee54c56260c54d9c0645579e5892dfa5a01eb9562a9ce2ef4cd3a66422"} Oct 31 00:39:43.777865 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:43.777442 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b32e23d-mvndb" event={"ID":"e232ca5e-2013-4024-8118-5ecdb3088579","Type":"ContainerStarted","Data":"62f3bdd47ba7b030c361a9618927652be2af01149f647f7802473b3ebdbd2a16"} Oct 31 00:39:43.899512 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:43.899453 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b32e23d-mvndb" podStartSLOduration=1.899436062 podStartE2EDuration="1.899436062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:39:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:39:43.895499624 +0000 UTC m=+1476.366669375" watchObservedRunningTime="2026-10-31 00:39:43.899436062 +0000 UTC m=+1476.370605811" Oct 31 00:39:44.789412 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:44.789379 5222 generic.go:356] "Generic (PLEG): container finished" podID="e232ca5e-2013-4024-8118-5ecdb3088579" containerID="ae9d30ee54c56260c54d9c0645579e5892dfa5a01eb9562a9ce2ef4cd3a66422" exitCode=0 Oct 31 00:39:44.789844 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:44.789460 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b32e23d-mvndb" event={"ID":"e232ca5e-2013-4024-8118-5ecdb3088579","Type":"ContainerDied","Data":"ae9d30ee54c56260c54d9c0645579e5892dfa5a01eb9562a9ce2ef4cd3a66422"} Oct 31 00:39:45.008436 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:45.008397 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-26fb7237-75b5fbc687-xdh58"] Oct 31 00:39:45.125662 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:45.125572 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-26fb7237-7878f946bc-j9xgk"] Oct 31 00:39:45.968450 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:45.968405 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b32e23d-mvndb" Oct 31 00:39:46.041613 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:46.041558 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e232ca5e-2013-4024-8118-5ecdb3088579-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e232ca5e-2013-4024-8118-5ecdb3088579-maas-api-xks-ca\") pod \"e232ca5e-2013-4024-8118-5ecdb3088579\" (UID: \"e232ca5e-2013-4024-8118-5ecdb3088579\") " Oct 31 00:39:46.041828 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:46.041813 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e232ca5e-2013-4024-8118-5ecdb3088579-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e232ca5e-2013-4024-8118-5ecdb3088579-maas-api-openshift-ca\") pod \"e232ca5e-2013-4024-8118-5ecdb3088579\" (UID: \"e232ca5e-2013-4024-8118-5ecdb3088579\") " Oct 31 00:39:46.041878 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:46.041849 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e232ca5e-2013-4024-8118-5ecdb3088579-kube-api-access-8zh72\" (UniqueName: \"kubernetes.io/projected/e232ca5e-2013-4024-8118-5ecdb3088579-kube-api-access-8zh72\") pod \"e232ca5e-2013-4024-8118-5ecdb3088579\" (UID: \"e232ca5e-2013-4024-8118-5ecdb3088579\") " Oct 31 00:39:46.042164 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:46.042138 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e232ca5e-2013-4024-8118-5ecdb3088579-maas-api-openshift-ca" pod "e232ca5e-2013-4024-8118-5ecdb3088579" (UID: "e232ca5e-2013-4024-8118-5ecdb3088579"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:39:46.042445 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:46.042401 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e232ca5e-2013-4024-8118-5ecdb3088579-maas-api-openshift-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:39:46.043847 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:46.043820 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e232ca5e-2013-4024-8118-5ecdb3088579-maas-api-xks-ca" pod "e232ca5e-2013-4024-8118-5ecdb3088579" (UID: "e232ca5e-2013-4024-8118-5ecdb3088579"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:39:46.044015 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:46.043998 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e232ca5e-2013-4024-8118-5ecdb3088579-kube-api-access-8zh72" pod "e232ca5e-2013-4024-8118-5ecdb3088579" (UID: "e232ca5e-2013-4024-8118-5ecdb3088579"). InnerVolumeSpecName "kube-api-access-8zh72". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:39:46.144350 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:46.144314 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zh72\" (UniqueName: \"kubernetes.io/projected/e232ca5e-2013-4024-8118-5ecdb3088579-kube-api-access-8zh72\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:39:46.144350 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:46.144348 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e232ca5e-2013-4024-8118-5ecdb3088579-maas-api-xks-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:39:46.822514 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:46.820270 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b32e23d-mvndb" event={"ID":"e232ca5e-2013-4024-8118-5ecdb3088579","Type":"ContainerDied","Data":"62f3bdd47ba7b030c361a9618927652be2af01149f647f7802473b3ebdbd2a16"} Oct 31 00:39:46.822514 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:46.820337 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="62f3bdd47ba7b030c361a9618927652be2af01149f647f7802473b3ebdbd2a16" Oct 31 00:39:46.822514 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:46.820443 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b32e23d-mvndb" Oct 31 00:39:47.684097 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:47.684062 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e7b593fa-s667c"] Oct 31 00:39:47.684755 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:47.684739 5222 cpu_manager.go:414] "containerMap: removing container" podUID="e232ca5e-2013-4024-8118-5ecdb3088579" containerName="revoke-keys" Oct 31 00:39:47.684805 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:47.684756 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e232ca5e-2013-4024-8118-5ecdb3088579" containerName="revoke-keys" Oct 31 00:39:47.684906 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:47.684897 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="e232ca5e-2013-4024-8118-5ecdb3088579" containerName="revoke-keys" Oct 31 00:39:47.688702 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:47.688685 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e7b593fa-s667c" Oct 31 00:39:47.743993 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:47.743949 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e7b593fa-s667c"] Oct 31 00:39:47.778380 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:47.778324 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7277c6d2-52f6-4109-b39d-786d6b41713d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-e7b593fa-s667c\" (UID: \"7277c6d2-52f6-4109-b39d-786d6b41713d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e7b593fa-s667c" Oct 31 00:39:47.778614 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:47.778593 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4t6n\" (UniqueName: \"kubernetes.io/projected/7277c6d2-52f6-4109-b39d-786d6b41713d-kube-api-access-q4t6n\") pod \"maas-api-revoke-keys-e2e-ait-e7b593fa-s667c\" (UID: \"7277c6d2-52f6-4109-b39d-786d6b41713d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e7b593fa-s667c" Oct 31 00:39:47.778668 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:47.778654 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7277c6d2-52f6-4109-b39d-786d6b41713d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-e7b593fa-s667c\" (UID: \"7277c6d2-52f6-4109-b39d-786d6b41713d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e7b593fa-s667c" Oct 31 00:39:47.787222 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:47.787190 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:39:47.787222 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:47.787202 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:39:47.787490 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:47.787276 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:39:47.795420 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:47.795396 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfz9z\"" Oct 31 00:39:47.880407 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:47.880337 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7277c6d2-52f6-4109-b39d-786d6b41713d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-e7b593fa-s667c\" (UID: \"7277c6d2-52f6-4109-b39d-786d6b41713d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e7b593fa-s667c" Oct 31 00:39:47.880592 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:47.880503 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4t6n\" (UniqueName: \"kubernetes.io/projected/7277c6d2-52f6-4109-b39d-786d6b41713d-kube-api-access-q4t6n\") pod \"maas-api-revoke-keys-e2e-ait-e7b593fa-s667c\" (UID: \"7277c6d2-52f6-4109-b39d-786d6b41713d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e7b593fa-s667c" Oct 31 00:39:47.880592 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:47.880540 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7277c6d2-52f6-4109-b39d-786d6b41713d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-e7b593fa-s667c\" (UID: \"7277c6d2-52f6-4109-b39d-786d6b41713d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e7b593fa-s667c" Oct 31 00:39:47.881199 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:47.881178 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7277c6d2-52f6-4109-b39d-786d6b41713d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-e7b593fa-s667c\" (UID: \"7277c6d2-52f6-4109-b39d-786d6b41713d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e7b593fa-s667c" Oct 31 00:39:47.882972 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:47.882947 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7277c6d2-52f6-4109-b39d-786d6b41713d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-e7b593fa-s667c\" (UID: \"7277c6d2-52f6-4109-b39d-786d6b41713d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e7b593fa-s667c" Oct 31 00:39:47.943236 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:47.943155 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4t6n\" (UniqueName: \"kubernetes.io/projected/7277c6d2-52f6-4109-b39d-786d6b41713d-kube-api-access-q4t6n\") pod \"maas-api-revoke-keys-e2e-ait-e7b593fa-s667c\" (UID: \"7277c6d2-52f6-4109-b39d-786d6b41713d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e7b593fa-s667c" Oct 31 00:39:48.002468 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:48.002430 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e7b593fa-s667c" Oct 31 00:39:48.382918 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:39:48.382885 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7277c6d2_52f6_4109_b39d_786d6b41713d.slice/crio-3b10e05d0be1b109e186364d3a787185b0fdd46c10d7e2a7714055f3fa3b20fb WatchSource:0}: Error finding container 3b10e05d0be1b109e186364d3a787185b0fdd46c10d7e2a7714055f3fa3b20fb: Status 404 returned error can't find the container with id 3b10e05d0be1b109e186364d3a787185b0fdd46c10d7e2a7714055f3fa3b20fb Oct 31 00:39:48.387587 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:48.387564 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e7b593fa-s667c"] Oct 31 00:39:48.838831 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:48.838793 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e7b593fa-s667c" event={"ID":"7277c6d2-52f6-4109-b39d-786d6b41713d","Type":"ContainerStarted","Data":"aa89b850223d8f994b3df7b38393faf69ca84e7ddc442d3be0476310687963b4"} Oct 31 00:39:48.839271 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:48.838860 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e7b593fa-s667c" event={"ID":"7277c6d2-52f6-4109-b39d-786d6b41713d","Type":"ContainerStarted","Data":"3b10e05d0be1b109e186364d3a787185b0fdd46c10d7e2a7714055f3fa3b20fb"} Oct 31 00:39:48.960663 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:48.960555 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e7b593fa-s667c" podStartSLOduration=1.960541133 podStartE2EDuration="1.960541133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:39:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:39:48.957381757 +0000 UTC m=+1481.428551499" watchObservedRunningTime="2026-10-31 00:39:48.960541133 +0000 UTC m=+1481.431710882" Oct 31 00:39:49.056514 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:49.056484 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e6ec7b6-g6jfm"] Oct 31 00:39:49.063696 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:49.063656 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e6ec7b6-g6jfm"] Oct 31 00:39:49.657216 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:49.657138 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34925a08-5519-4b30-b8b7-30c4681d709b" path="/var/lib/kubelet/pods/34925a08-5519-4b30-b8b7-30c4681d709b/volumes" Oct 31 00:39:49.848706 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:49.848666 5222 generic.go:356] "Generic (PLEG): container finished" podID="7277c6d2-52f6-4109-b39d-786d6b41713d" containerID="aa89b850223d8f994b3df7b38393faf69ca84e7ddc442d3be0476310687963b4" exitCode=0 Oct 31 00:39:49.849107 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:49.848732 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e7b593fa-s667c" event={"ID":"7277c6d2-52f6-4109-b39d-786d6b41713d","Type":"ContainerDied","Data":"aa89b850223d8f994b3df7b38393faf69ca84e7ddc442d3be0476310687963b4"} Oct 31 00:39:50.009444 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:50.009341 5222 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-26fb7237-75b5fbc687-xdh58" podUID="9d26f843-46bb-4fd0-a33b-81b8b6e67d99" containerName="payload-processing" containerID="cri-o://540bb11f8f5c877f63eafc69d7c2d84063fdc25f48a88c38d3dbc368184dea07" gracePeriod=25 Oct 31 00:39:50.126908 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:50.126870 5222 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-26fb7237-7878f946bc-j9xgk" podUID="c507c4c8-d49b-4d8f-9c71-bb8779483055" containerName="payload-pre-processing" containerID="cri-o://abb82a8d7d1b09bf03b5b5d2ed11a89edcc1bc5bc01c7a5050c91e9d6df34387" gracePeriod=25 Oct 31 00:39:50.298472 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:50.298440 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-26fb7237-75b5fbc687-xdh58" Oct 31 00:39:50.415257 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:50.415227 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d26f843-46bb-4fd0-a33b-81b8b6e67d99-kube-api-access-nj4np\" (UniqueName: \"kubernetes.io/projected/9d26f843-46bb-4fd0-a33b-81b8b6e67d99-kube-api-access-nj4np\") pod \"9d26f843-46bb-4fd0-a33b-81b8b6e67d99\" (UID: \"9d26f843-46bb-4fd0-a33b-81b8b6e67d99\") " Oct 31 00:39:50.415473 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:50.415362 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d26f843-46bb-4fd0-a33b-81b8b6e67d99-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d26f843-46bb-4fd0-a33b-81b8b6e67d99-plugins-config-volume\") pod \"9d26f843-46bb-4fd0-a33b-81b8b6e67d99\" (UID: \"9d26f843-46bb-4fd0-a33b-81b8b6e67d99\") " Oct 31 00:39:50.415769 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:50.415741 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d26f843-46bb-4fd0-a33b-81b8b6e67d99-plugins-config-volume" pod "9d26f843-46bb-4fd0-a33b-81b8b6e67d99" (UID: "9d26f843-46bb-4fd0-a33b-81b8b6e67d99"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:39:50.415989 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:50.415967 5222 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d26f843-46bb-4fd0-a33b-81b8b6e67d99-plugins-config-volume\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:39:50.418337 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:50.418308 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d26f843-46bb-4fd0-a33b-81b8b6e67d99-kube-api-access-nj4np" pod "9d26f843-46bb-4fd0-a33b-81b8b6e67d99" (UID: "9d26f843-46bb-4fd0-a33b-81b8b6e67d99"). InnerVolumeSpecName "kube-api-access-nj4np". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:39:50.419402 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:50.419355 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-26fb7237-7878f946bc-j9xgk" Oct 31 00:39:50.517355 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:50.517322 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c507c4c8-d49b-4d8f-9c71-bb8779483055-kube-api-access-lt9g6\" (UniqueName: \"kubernetes.io/projected/c507c4c8-d49b-4d8f-9c71-bb8779483055-kube-api-access-lt9g6\") pod \"c507c4c8-d49b-4d8f-9c71-bb8779483055\" (UID: \"c507c4c8-d49b-4d8f-9c71-bb8779483055\") " Oct 31 00:39:50.517583 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:50.517419 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c507c4c8-d49b-4d8f-9c71-bb8779483055-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c507c4c8-d49b-4d8f-9c71-bb8779483055-plugins-config-volume\") pod \"c507c4c8-d49b-4d8f-9c71-bb8779483055\" (UID: \"c507c4c8-d49b-4d8f-9c71-bb8779483055\") " Oct 31 00:39:50.517773 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:50.517750 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c507c4c8-d49b-4d8f-9c71-bb8779483055-plugins-config-volume" pod "c507c4c8-d49b-4d8f-9c71-bb8779483055" (UID: "c507c4c8-d49b-4d8f-9c71-bb8779483055"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:39:50.517840 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:50.517811 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nj4np\" (UniqueName: \"kubernetes.io/projected/9d26f843-46bb-4fd0-a33b-81b8b6e67d99-kube-api-access-nj4np\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:39:50.519620 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:50.519599 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c507c4c8-d49b-4d8f-9c71-bb8779483055-kube-api-access-lt9g6" pod "c507c4c8-d49b-4d8f-9c71-bb8779483055" (UID: "c507c4c8-d49b-4d8f-9c71-bb8779483055"). InnerVolumeSpecName "kube-api-access-lt9g6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:39:50.618853 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:50.618816 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lt9g6\" (UniqueName: \"kubernetes.io/projected/c507c4c8-d49b-4d8f-9c71-bb8779483055-kube-api-access-lt9g6\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:39:50.618853 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:50.618847 5222 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c507c4c8-d49b-4d8f-9c71-bb8779483055-plugins-config-volume\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:39:50.858622 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:50.858538 5222 generic.go:356] "Generic (PLEG): container finished" podID="9d26f843-46bb-4fd0-a33b-81b8b6e67d99" containerID="540bb11f8f5c877f63eafc69d7c2d84063fdc25f48a88c38d3dbc368184dea07" exitCode=0 Oct 31 00:39:50.858622 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:50.858587 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-26fb7237-75b5fbc687-xdh58" Oct 31 00:39:50.859141 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:50.858624 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-26fb7237-75b5fbc687-xdh58" event={"ID":"9d26f843-46bb-4fd0-a33b-81b8b6e67d99","Type":"ContainerDied","Data":"540bb11f8f5c877f63eafc69d7c2d84063fdc25f48a88c38d3dbc368184dea07"} Oct 31 00:39:50.859141 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:50.858694 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-26fb7237-75b5fbc687-xdh58" event={"ID":"9d26f843-46bb-4fd0-a33b-81b8b6e67d99","Type":"ContainerDied","Data":"e23b284246844b09dfc501419e5f2a5ce3389f62532311b160d682aa49e4487f"} Oct 31 00:39:50.859141 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:50.858722 5222 scope.go:122] "RemoveContainer" containerID="540bb11f8f5c877f63eafc69d7c2d84063fdc25f48a88c38d3dbc368184dea07" Oct 31 00:39:50.860284 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:50.860255 5222 generic.go:356] "Generic (PLEG): container finished" podID="c507c4c8-d49b-4d8f-9c71-bb8779483055" containerID="abb82a8d7d1b09bf03b5b5d2ed11a89edcc1bc5bc01c7a5050c91e9d6df34387" exitCode=0 Oct 31 00:39:50.860467 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:50.860307 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-26fb7237-7878f946bc-j9xgk" Oct 31 00:39:50.860467 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:50.860336 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-26fb7237-7878f946bc-j9xgk" event={"ID":"c507c4c8-d49b-4d8f-9c71-bb8779483055","Type":"ContainerDied","Data":"abb82a8d7d1b09bf03b5b5d2ed11a89edcc1bc5bc01c7a5050c91e9d6df34387"} Oct 31 00:39:50.860467 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:50.860455 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-26fb7237-7878f946bc-j9xgk" event={"ID":"c507c4c8-d49b-4d8f-9c71-bb8779483055","Type":"ContainerDied","Data":"8ac5c4a4c3480886ee4eb45e6ef11a96b8b4bf81868171b220c74d77fc430db6"} Oct 31 00:39:50.871949 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:50.871930 5222 scope.go:122] "RemoveContainer" containerID="540bb11f8f5c877f63eafc69d7c2d84063fdc25f48a88c38d3dbc368184dea07" Oct 31 00:39:50.872312 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:39:50.872292 5222 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"540bb11f8f5c877f63eafc69d7c2d84063fdc25f48a88c38d3dbc368184dea07\": container with ID starting with 540bb11f8f5c877f63eafc69d7c2d84063fdc25f48a88c38d3dbc368184dea07 not found: ID does not exist" containerID="540bb11f8f5c877f63eafc69d7c2d84063fdc25f48a88c38d3dbc368184dea07" Oct 31 00:39:50.872428 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:50.872322 5222 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"540bb11f8f5c877f63eafc69d7c2d84063fdc25f48a88c38d3dbc368184dea07"} err="failed to get container status \"540bb11f8f5c877f63eafc69d7c2d84063fdc25f48a88c38d3dbc368184dea07\": rpc error: code = NotFound desc = could not find container \"540bb11f8f5c877f63eafc69d7c2d84063fdc25f48a88c38d3dbc368184dea07\": container with ID starting with 540bb11f8f5c877f63eafc69d7c2d84063fdc25f48a88c38d3dbc368184dea07 not found: ID does not exist" Oct 31 00:39:50.872428 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:50.872342 5222 scope.go:122] "RemoveContainer" containerID="abb82a8d7d1b09bf03b5b5d2ed11a89edcc1bc5bc01c7a5050c91e9d6df34387" Oct 31 00:39:50.885493 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:50.885467 5222 scope.go:122] "RemoveContainer" containerID="abb82a8d7d1b09bf03b5b5d2ed11a89edcc1bc5bc01c7a5050c91e9d6df34387" Oct 31 00:39:50.885858 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:39:50.885834 5222 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abb82a8d7d1b09bf03b5b5d2ed11a89edcc1bc5bc01c7a5050c91e9d6df34387\": container with ID starting with abb82a8d7d1b09bf03b5b5d2ed11a89edcc1bc5bc01c7a5050c91e9d6df34387 not found: ID does not exist" containerID="abb82a8d7d1b09bf03b5b5d2ed11a89edcc1bc5bc01c7a5050c91e9d6df34387" Oct 31 00:39:50.885936 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:50.885867 5222 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abb82a8d7d1b09bf03b5b5d2ed11a89edcc1bc5bc01c7a5050c91e9d6df34387"} err="failed to get container status \"abb82a8d7d1b09bf03b5b5d2ed11a89edcc1bc5bc01c7a5050c91e9d6df34387\": rpc error: code = NotFound desc = could not find container \"abb82a8d7d1b09bf03b5b5d2ed11a89edcc1bc5bc01c7a5050c91e9d6df34387\": container with ID starting with abb82a8d7d1b09bf03b5b5d2ed11a89edcc1bc5bc01c7a5050c91e9d6df34387 not found: ID does not exist" Oct 31 00:39:51.016766 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:51.016735 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e7b593fa-s667c" Oct 31 00:39:51.025606 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:51.025579 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7277c6d2-52f6-4109-b39d-786d6b41713d-kube-api-access-q4t6n\" (UniqueName: \"kubernetes.io/projected/7277c6d2-52f6-4109-b39d-786d6b41713d-kube-api-access-q4t6n\") pod \"7277c6d2-52f6-4109-b39d-786d6b41713d\" (UID: \"7277c6d2-52f6-4109-b39d-786d6b41713d\") " Oct 31 00:39:51.025791 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:51.025637 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7277c6d2-52f6-4109-b39d-786d6b41713d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7277c6d2-52f6-4109-b39d-786d6b41713d-maas-api-openshift-ca\") pod \"7277c6d2-52f6-4109-b39d-786d6b41713d\" (UID: \"7277c6d2-52f6-4109-b39d-786d6b41713d\") " Oct 31 00:39:51.025919 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:51.025842 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7277c6d2-52f6-4109-b39d-786d6b41713d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7277c6d2-52f6-4109-b39d-786d6b41713d-maas-api-xks-ca\") pod \"7277c6d2-52f6-4109-b39d-786d6b41713d\" (UID: \"7277c6d2-52f6-4109-b39d-786d6b41713d\") " Oct 31 00:39:51.026108 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:51.026072 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7277c6d2-52f6-4109-b39d-786d6b41713d-maas-api-openshift-ca" pod "7277c6d2-52f6-4109-b39d-786d6b41713d" (UID: "7277c6d2-52f6-4109-b39d-786d6b41713d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:39:51.026416 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:51.026397 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7277c6d2-52f6-4109-b39d-786d6b41713d-maas-api-openshift-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:39:51.028179 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:51.028136 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7277c6d2-52f6-4109-b39d-786d6b41713d-kube-api-access-q4t6n" pod "7277c6d2-52f6-4109-b39d-786d6b41713d" (UID: "7277c6d2-52f6-4109-b39d-786d6b41713d"). InnerVolumeSpecName "kube-api-access-q4t6n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:39:51.028262 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:51.028215 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7277c6d2-52f6-4109-b39d-786d6b41713d-maas-api-xks-ca" pod "7277c6d2-52f6-4109-b39d-786d6b41713d" (UID: "7277c6d2-52f6-4109-b39d-786d6b41713d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:39:51.035629 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:51.035601 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-26fb7237-75b5fbc687-xdh58"] Oct 31 00:39:51.040094 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:51.040067 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-26fb7237-75b5fbc687-xdh58"] Oct 31 00:39:51.128246 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:51.128164 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7277c6d2-52f6-4109-b39d-786d6b41713d-maas-api-xks-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:39:51.128246 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:51.128193 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4t6n\" (UniqueName: \"kubernetes.io/projected/7277c6d2-52f6-4109-b39d-786d6b41713d-kube-api-access-q4t6n\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:39:51.152572 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:51.152540 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-26fb7237-7878f946bc-j9xgk"] Oct 31 00:39:51.156922 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:51.156902 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-26fb7237-7878f946bc-j9xgk"] Oct 31 00:39:51.657342 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:51.657307 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d26f843-46bb-4fd0-a33b-81b8b6e67d99" path="/var/lib/kubelet/pods/9d26f843-46bb-4fd0-a33b-81b8b6e67d99/volumes" Oct 31 00:39:51.657776 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:51.657756 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c507c4c8-d49b-4d8f-9c71-bb8779483055" path="/var/lib/kubelet/pods/c507c4c8-d49b-4d8f-9c71-bb8779483055/volumes" Oct 31 00:39:51.872821 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:51.872796 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e7b593fa-s667c" Oct 31 00:39:51.872821 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:51.872816 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e7b593fa-s667c" event={"ID":"7277c6d2-52f6-4109-b39d-786d6b41713d","Type":"ContainerDied","Data":"3b10e05d0be1b109e186364d3a787185b0fdd46c10d7e2a7714055f3fa3b20fb"} Oct 31 00:39:51.873247 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:51.872867 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b10e05d0be1b109e186364d3a787185b0fdd46c10d7e2a7714055f3fa3b20fb" Oct 31 00:39:56.840805 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:56.840762 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-49bb6b1a-gjbsr"] Oct 31 00:39:56.841435 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:56.841419 5222 cpu_manager.go:414] "containerMap: removing container" podUID="9d26f843-46bb-4fd0-a33b-81b8b6e67d99" containerName="payload-processing" Oct 31 00:39:56.841529 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:56.841435 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d26f843-46bb-4fd0-a33b-81b8b6e67d99" containerName="payload-processing" Oct 31 00:39:56.841529 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:56.841453 5222 cpu_manager.go:414] "containerMap: removing container" podUID="c507c4c8-d49b-4d8f-9c71-bb8779483055" containerName="payload-pre-processing" Oct 31 00:39:56.841529 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:56.841458 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c507c4c8-d49b-4d8f-9c71-bb8779483055" containerName="payload-pre-processing" Oct 31 00:39:56.841529 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:56.841502 5222 cpu_manager.go:414] "containerMap: removing container" podUID="7277c6d2-52f6-4109-b39d-786d6b41713d" containerName="revoke-keys" Oct 31 00:39:56.841529 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:56.841510 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7277c6d2-52f6-4109-b39d-786d6b41713d" containerName="revoke-keys" Oct 31 00:39:56.841790 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:56.841685 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="c507c4c8-d49b-4d8f-9c71-bb8779483055" containerName="payload-pre-processing" Oct 31 00:39:56.841790 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:56.841703 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="7277c6d2-52f6-4109-b39d-786d6b41713d" containerName="revoke-keys" Oct 31 00:39:56.841790 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:56.841720 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d26f843-46bb-4fd0-a33b-81b8b6e67d99" containerName="payload-processing" Oct 31 00:39:56.846032 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:56.846007 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-49bb6b1a-gjbsr" Oct 31 00:39:56.903979 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:56.903725 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:39:56.907988 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:56.907953 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwgnq\" (UniqueName: \"kubernetes.io/projected/e141d579-fcbb-40f1-aa2c-16713029c97b-kube-api-access-fwgnq\") pod \"maas-api-revoke-keys-e2e-ait-49bb6b1a-gjbsr\" (UID: \"e141d579-fcbb-40f1-aa2c-16713029c97b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-49bb6b1a-gjbsr" Oct 31 00:39:56.908336 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:56.908314 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e141d579-fcbb-40f1-aa2c-16713029c97b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-49bb6b1a-gjbsr\" (UID: \"e141d579-fcbb-40f1-aa2c-16713029c97b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-49bb6b1a-gjbsr" Oct 31 00:39:56.908519 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:56.908494 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e141d579-fcbb-40f1-aa2c-16713029c97b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-49bb6b1a-gjbsr\" (UID: \"e141d579-fcbb-40f1-aa2c-16713029c97b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-49bb6b1a-gjbsr" Oct 31 00:39:56.955294 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:56.955256 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:39:56.955492 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:56.955447 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfz9z\"" Oct 31 00:39:56.955587 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:56.955494 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:39:56.966321 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:56.966285 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-49bb6b1a-gjbsr"] Oct 31 00:39:57.009618 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:57.009584 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e141d579-fcbb-40f1-aa2c-16713029c97b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-49bb6b1a-gjbsr\" (UID: \"e141d579-fcbb-40f1-aa2c-16713029c97b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-49bb6b1a-gjbsr" Oct 31 00:39:57.009830 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:57.009634 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e141d579-fcbb-40f1-aa2c-16713029c97b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-49bb6b1a-gjbsr\" (UID: \"e141d579-fcbb-40f1-aa2c-16713029c97b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-49bb6b1a-gjbsr" Oct 31 00:39:57.009830 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:57.009792 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwgnq\" (UniqueName: \"kubernetes.io/projected/e141d579-fcbb-40f1-aa2c-16713029c97b-kube-api-access-fwgnq\") pod \"maas-api-revoke-keys-e2e-ait-49bb6b1a-gjbsr\" (UID: \"e141d579-fcbb-40f1-aa2c-16713029c97b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-49bb6b1a-gjbsr" Oct 31 00:39:57.010278 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:57.010255 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e141d579-fcbb-40f1-aa2c-16713029c97b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-49bb6b1a-gjbsr\" (UID: \"e141d579-fcbb-40f1-aa2c-16713029c97b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-49bb6b1a-gjbsr" Oct 31 00:39:57.012302 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:57.012273 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e141d579-fcbb-40f1-aa2c-16713029c97b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-49bb6b1a-gjbsr\" (UID: \"e141d579-fcbb-40f1-aa2c-16713029c97b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-49bb6b1a-gjbsr" Oct 31 00:39:57.076269 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:57.076232 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwgnq\" (UniqueName: \"kubernetes.io/projected/e141d579-fcbb-40f1-aa2c-16713029c97b-kube-api-access-fwgnq\") pod \"maas-api-revoke-keys-e2e-ait-49bb6b1a-gjbsr\" (UID: \"e141d579-fcbb-40f1-aa2c-16713029c97b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-49bb6b1a-gjbsr" Oct 31 00:39:57.161741 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:57.161663 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-49bb6b1a-gjbsr" Oct 31 00:39:57.565739 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:39:57.565710 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode141d579_fcbb_40f1_aa2c_16713029c97b.slice/crio-1c0d45f6f24ade42443ac7bb8e95acd58a938600d0e05ef85143c659b2b062ea WatchSource:0}: Error finding container 1c0d45f6f24ade42443ac7bb8e95acd58a938600d0e05ef85143c659b2b062ea: Status 404 returned error can't find the container with id 1c0d45f6f24ade42443ac7bb8e95acd58a938600d0e05ef85143c659b2b062ea Oct 31 00:39:57.573039 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:57.573010 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-49bb6b1a-gjbsr"] Oct 31 00:39:57.926603 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:57.926519 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-49bb6b1a-gjbsr" event={"ID":"e141d579-fcbb-40f1-aa2c-16713029c97b","Type":"ContainerStarted","Data":"1c0d45f6f24ade42443ac7bb8e95acd58a938600d0e05ef85143c659b2b062ea"} Oct 31 00:39:58.937427 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:58.937357 5222 generic.go:356] "Generic (PLEG): container finished" podID="e141d579-fcbb-40f1-aa2c-16713029c97b" containerID="15257effc61ebaf3d1fcabcb48bfa63dce43e6d96d2e9019579ade74e583b4e6" exitCode=0 Oct 31 00:39:58.937427 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:39:58.937402 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-49bb6b1a-gjbsr" event={"ID":"e141d579-fcbb-40f1-aa2c-16713029c97b","Type":"ContainerDied","Data":"15257effc61ebaf3d1fcabcb48bfa63dce43e6d96d2e9019579ade74e583b4e6"} Oct 31 00:40:00.115834 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:00.115808 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-49bb6b1a-gjbsr" Oct 31 00:40:00.162349 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:00.162312 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e141d579-fcbb-40f1-aa2c-16713029c97b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e141d579-fcbb-40f1-aa2c-16713029c97b-maas-api-xks-ca\") pod \"e141d579-fcbb-40f1-aa2c-16713029c97b\" (UID: \"e141d579-fcbb-40f1-aa2c-16713029c97b\") " Oct 31 00:40:00.162667 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:00.162645 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e141d579-fcbb-40f1-aa2c-16713029c97b-kube-api-access-fwgnq\" (UniqueName: \"kubernetes.io/projected/e141d579-fcbb-40f1-aa2c-16713029c97b-kube-api-access-fwgnq\") pod \"e141d579-fcbb-40f1-aa2c-16713029c97b\" (UID: \"e141d579-fcbb-40f1-aa2c-16713029c97b\") " Oct 31 00:40:00.162797 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:00.162781 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e141d579-fcbb-40f1-aa2c-16713029c97b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e141d579-fcbb-40f1-aa2c-16713029c97b-maas-api-openshift-ca\") pod \"e141d579-fcbb-40f1-aa2c-16713029c97b\" (UID: \"e141d579-fcbb-40f1-aa2c-16713029c97b\") " Oct 31 00:40:00.163158 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:00.163137 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e141d579-fcbb-40f1-aa2c-16713029c97b-maas-api-openshift-ca" pod "e141d579-fcbb-40f1-aa2c-16713029c97b" (UID: "e141d579-fcbb-40f1-aa2c-16713029c97b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:40:00.163361 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:00.163347 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e141d579-fcbb-40f1-aa2c-16713029c97b-maas-api-openshift-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:40:00.164648 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:00.164627 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e141d579-fcbb-40f1-aa2c-16713029c97b-maas-api-xks-ca" pod "e141d579-fcbb-40f1-aa2c-16713029c97b" (UID: "e141d579-fcbb-40f1-aa2c-16713029c97b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:40:00.164806 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:00.164787 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e141d579-fcbb-40f1-aa2c-16713029c97b-kube-api-access-fwgnq" pod "e141d579-fcbb-40f1-aa2c-16713029c97b" (UID: "e141d579-fcbb-40f1-aa2c-16713029c97b"). InnerVolumeSpecName "kube-api-access-fwgnq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:40:00.265304 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:00.265269 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwgnq\" (UniqueName: \"kubernetes.io/projected/e141d579-fcbb-40f1-aa2c-16713029c97b-kube-api-access-fwgnq\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:40:00.265493 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:00.265311 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e141d579-fcbb-40f1-aa2c-16713029c97b-maas-api-xks-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:40:00.967842 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:00.967801 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-49bb6b1a-gjbsr" event={"ID":"e141d579-fcbb-40f1-aa2c-16713029c97b","Type":"ContainerDied","Data":"1c0d45f6f24ade42443ac7bb8e95acd58a938600d0e05ef85143c659b2b062ea"} Oct 31 00:40:00.968113 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:00.968096 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1c0d45f6f24ade42443ac7bb8e95acd58a938600d0e05ef85143c659b2b062ea" Oct 31 00:40:00.968281 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:00.968268 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-49bb6b1a-gjbsr" Oct 31 00:40:05.891602 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:40:05.891559 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 00:40:07.864406 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:07.864350 5222 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h9hsf_0f8c3c03-a1fc-4376-98c6-feeeefa08068/ovn-acl-logging/0.log" Oct 31 00:40:07.869574 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:07.869548 5222 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h9hsf_0f8c3c03-a1fc-4376-98c6-feeeefa08068/ovn-acl-logging/0.log" Oct 31 00:40:08.471243 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:08.471207 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-49bb6b1a-775dfc4dd5-7jxf7"] Oct 31 00:40:08.568797 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:08.568766 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-49bb6b1a-6d969799c5-6lkcs"] Oct 31 00:40:13.471839 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:13.471793 5222 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-49bb6b1a-775dfc4dd5-7jxf7" podUID="2f791ab3-6734-4655-afb6-3267d91c961a" containerName="payload-processing" containerID="cri-o://ad353db7d100dd0bcfa6bdc6755aa8e35d949be3807c4fd1b9bf05a0ca7e3651" gracePeriod=25 Oct 31 00:40:13.569912 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:13.569856 5222 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-49bb6b1a-6d969799c5-6lkcs" podUID="0e8bd878-f701-4547-bc62-6ea4de9bb5e0" containerName="payload-pre-processing" containerID="cri-o://5ece13bb5b292401dac8c5ed51dcc1dd95bbe45f04cbd3dafe3fc25a89c923e6" gracePeriod=25 Oct 31 00:40:13.764328 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:13.764301 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-49bb6b1a-775dfc4dd5-7jxf7" Oct 31 00:40:13.848471 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:13.848439 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f791ab3-6734-4655-afb6-3267d91c961a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f791ab3-6734-4655-afb6-3267d91c961a-plugins-config-volume\") pod \"2f791ab3-6734-4655-afb6-3267d91c961a\" (UID: \"2f791ab3-6734-4655-afb6-3267d91c961a\") " Oct 31 00:40:13.848646 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:13.848527 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f791ab3-6734-4655-afb6-3267d91c961a-kube-api-access-ltklt\" (UniqueName: \"kubernetes.io/projected/2f791ab3-6734-4655-afb6-3267d91c961a-kube-api-access-ltklt\") pod \"2f791ab3-6734-4655-afb6-3267d91c961a\" (UID: \"2f791ab3-6734-4655-afb6-3267d91c961a\") " Oct 31 00:40:13.848881 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:13.848837 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f791ab3-6734-4655-afb6-3267d91c961a-plugins-config-volume" pod "2f791ab3-6734-4655-afb6-3267d91c961a" (UID: "2f791ab3-6734-4655-afb6-3267d91c961a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:40:13.849277 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:13.849251 5222 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f791ab3-6734-4655-afb6-3267d91c961a-plugins-config-volume\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:40:13.850895 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:13.850867 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f791ab3-6734-4655-afb6-3267d91c961a-kube-api-access-ltklt" pod "2f791ab3-6734-4655-afb6-3267d91c961a" (UID: "2f791ab3-6734-4655-afb6-3267d91c961a"). InnerVolumeSpecName "kube-api-access-ltklt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:40:13.873678 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:13.873651 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-49bb6b1a-6d969799c5-6lkcs" Oct 31 00:40:13.949956 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:13.949922 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e8bd878-f701-4547-bc62-6ea4de9bb5e0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e8bd878-f701-4547-bc62-6ea4de9bb5e0-plugins-config-volume\") pod \"0e8bd878-f701-4547-bc62-6ea4de9bb5e0\" (UID: \"0e8bd878-f701-4547-bc62-6ea4de9bb5e0\") " Oct 31 00:40:13.950133 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:13.949993 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e8bd878-f701-4547-bc62-6ea4de9bb5e0-kube-api-access-rb95k\" (UniqueName: \"kubernetes.io/projected/0e8bd878-f701-4547-bc62-6ea4de9bb5e0-kube-api-access-rb95k\") pod \"0e8bd878-f701-4547-bc62-6ea4de9bb5e0\" (UID: \"0e8bd878-f701-4547-bc62-6ea4de9bb5e0\") " Oct 31 00:40:13.950338 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:13.950317 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e8bd878-f701-4547-bc62-6ea4de9bb5e0-plugins-config-volume" pod "0e8bd878-f701-4547-bc62-6ea4de9bb5e0" (UID: "0e8bd878-f701-4547-bc62-6ea4de9bb5e0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:40:13.950615 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:13.950593 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltklt\" (UniqueName: \"kubernetes.io/projected/2f791ab3-6734-4655-afb6-3267d91c961a-kube-api-access-ltklt\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:40:13.950661 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:13.950623 5222 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e8bd878-f701-4547-bc62-6ea4de9bb5e0-plugins-config-volume\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:40:13.952251 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:13.952232 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e8bd878-f701-4547-bc62-6ea4de9bb5e0-kube-api-access-rb95k" pod "0e8bd878-f701-4547-bc62-6ea4de9bb5e0" (UID: "0e8bd878-f701-4547-bc62-6ea4de9bb5e0"). InnerVolumeSpecName "kube-api-access-rb95k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:40:14.052551 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:14.052519 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rb95k\" (UniqueName: \"kubernetes.io/projected/0e8bd878-f701-4547-bc62-6ea4de9bb5e0-kube-api-access-rb95k\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:40:14.128185 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:14.128152 5222 generic.go:356] "Generic (PLEG): container finished" podID="0e8bd878-f701-4547-bc62-6ea4de9bb5e0" containerID="5ece13bb5b292401dac8c5ed51dcc1dd95bbe45f04cbd3dafe3fc25a89c923e6" exitCode=0 Oct 31 00:40:14.128356 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:14.128197 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-49bb6b1a-6d969799c5-6lkcs" Oct 31 00:40:14.128356 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:14.128235 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-49bb6b1a-6d969799c5-6lkcs" event={"ID":"0e8bd878-f701-4547-bc62-6ea4de9bb5e0","Type":"ContainerDied","Data":"5ece13bb5b292401dac8c5ed51dcc1dd95bbe45f04cbd3dafe3fc25a89c923e6"} Oct 31 00:40:14.128356 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:14.128291 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-49bb6b1a-6d969799c5-6lkcs" event={"ID":"0e8bd878-f701-4547-bc62-6ea4de9bb5e0","Type":"ContainerDied","Data":"5eae1632d6eefcdba16574cc647a89b04b2377d9a683e57862f8f24f1306c962"} Oct 31 00:40:14.128356 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:14.128316 5222 scope.go:122] "RemoveContainer" containerID="5ece13bb5b292401dac8c5ed51dcc1dd95bbe45f04cbd3dafe3fc25a89c923e6" Oct 31 00:40:14.130163 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:14.130132 5222 generic.go:356] "Generic (PLEG): container finished" podID="2f791ab3-6734-4655-afb6-3267d91c961a" containerID="ad353db7d100dd0bcfa6bdc6755aa8e35d949be3807c4fd1b9bf05a0ca7e3651" exitCode=0 Oct 31 00:40:14.130292 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:14.130177 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-49bb6b1a-775dfc4dd5-7jxf7" event={"ID":"2f791ab3-6734-4655-afb6-3267d91c961a","Type":"ContainerDied","Data":"ad353db7d100dd0bcfa6bdc6755aa8e35d949be3807c4fd1b9bf05a0ca7e3651"} Oct 31 00:40:14.130292 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:14.130191 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-49bb6b1a-775dfc4dd5-7jxf7" Oct 31 00:40:14.130292 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:14.130218 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-49bb6b1a-775dfc4dd5-7jxf7" event={"ID":"2f791ab3-6734-4655-afb6-3267d91c961a","Type":"ContainerDied","Data":"7be7f03571e3404bb32270fd39236cc647ee860bd878deff85f136cf31949af4"} Oct 31 00:40:14.140527 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:14.140508 5222 scope.go:122] "RemoveContainer" containerID="5ece13bb5b292401dac8c5ed51dcc1dd95bbe45f04cbd3dafe3fc25a89c923e6" Oct 31 00:40:14.140919 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:40:14.140833 5222 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ece13bb5b292401dac8c5ed51dcc1dd95bbe45f04cbd3dafe3fc25a89c923e6\": container with ID starting with 5ece13bb5b292401dac8c5ed51dcc1dd95bbe45f04cbd3dafe3fc25a89c923e6 not found: ID does not exist" containerID="5ece13bb5b292401dac8c5ed51dcc1dd95bbe45f04cbd3dafe3fc25a89c923e6" Oct 31 00:40:14.140919 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:14.140867 5222 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ece13bb5b292401dac8c5ed51dcc1dd95bbe45f04cbd3dafe3fc25a89c923e6"} err="failed to get container status \"5ece13bb5b292401dac8c5ed51dcc1dd95bbe45f04cbd3dafe3fc25a89c923e6\": rpc error: code = NotFound desc = could not find container \"5ece13bb5b292401dac8c5ed51dcc1dd95bbe45f04cbd3dafe3fc25a89c923e6\": container with ID starting with 5ece13bb5b292401dac8c5ed51dcc1dd95bbe45f04cbd3dafe3fc25a89c923e6 not found: ID does not exist" Oct 31 00:40:14.140919 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:14.140893 5222 scope.go:122] "RemoveContainer" containerID="ad353db7d100dd0bcfa6bdc6755aa8e35d949be3807c4fd1b9bf05a0ca7e3651" Oct 31 00:40:14.151418 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:14.151360 5222 scope.go:122] "RemoveContainer" containerID="ad353db7d100dd0bcfa6bdc6755aa8e35d949be3807c4fd1b9bf05a0ca7e3651" Oct 31 00:40:14.151716 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:40:14.151694 5222 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad353db7d100dd0bcfa6bdc6755aa8e35d949be3807c4fd1b9bf05a0ca7e3651\": container with ID starting with ad353db7d100dd0bcfa6bdc6755aa8e35d949be3807c4fd1b9bf05a0ca7e3651 not found: ID does not exist" containerID="ad353db7d100dd0bcfa6bdc6755aa8e35d949be3807c4fd1b9bf05a0ca7e3651" Oct 31 00:40:14.151789 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:14.151725 5222 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad353db7d100dd0bcfa6bdc6755aa8e35d949be3807c4fd1b9bf05a0ca7e3651"} err="failed to get container status \"ad353db7d100dd0bcfa6bdc6755aa8e35d949be3807c4fd1b9bf05a0ca7e3651\": rpc error: code = NotFound desc = could not find container \"ad353db7d100dd0bcfa6bdc6755aa8e35d949be3807c4fd1b9bf05a0ca7e3651\": container with ID starting with ad353db7d100dd0bcfa6bdc6755aa8e35d949be3807c4fd1b9bf05a0ca7e3651 not found: ID does not exist" Oct 31 00:40:14.300077 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:14.300048 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-49bb6b1a-775dfc4dd5-7jxf7"] Oct 31 00:40:14.305333 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:14.305252 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-49bb6b1a-775dfc4dd5-7jxf7"] Oct 31 00:40:14.416741 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:14.416709 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-49bb6b1a-6d969799c5-6lkcs"] Oct 31 00:40:14.421147 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:14.421122 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-49bb6b1a-6d969799c5-6lkcs"] Oct 31 00:40:15.657249 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:15.657217 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e8bd878-f701-4547-bc62-6ea4de9bb5e0" path="/var/lib/kubelet/pods/0e8bd878-f701-4547-bc62-6ea4de9bb5e0/volumes" Oct 31 00:40:15.657703 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:15.657688 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f791ab3-6734-4655-afb6-3267d91c961a" path="/var/lib/kubelet/pods/2f791ab3-6734-4655-afb6-3267d91c961a/volumes" Oct 31 00:40:16.053057 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:16.053023 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b8bd811-82hkb"] Oct 31 00:40:16.104114 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:16.104079 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b8bd811-82hkb"] Oct 31 00:40:17.657438 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:17.657404 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4348f9bc-3109-48a5-a918-4511cf6d0fec" path="/var/lib/kubelet/pods/4348f9bc-3109-48a5-a918-4511cf6d0fec/volumes" Oct 31 00:40:24.631648 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:24.631609 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-26fb7237-hcqhj"] Oct 31 00:40:24.684358 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:24.684220 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-26fb7237-hcqhj"] Oct 31 00:40:25.657296 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:25.657254 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65ee49e4-6fa0-4e7a-991f-ea96cd0f9ae4" path="/var/lib/kubelet/pods/65ee49e4-6fa0-4e7a-991f-ea96cd0f9ae4/volumes" Oct 31 00:40:26.087396 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:26.086660 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-dd862feb-openshift-default-67b7c76895-zrjbk"] Oct 31 00:40:26.091325 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:26.087616 5222 cpu_manager.go:414] "containerMap: removing container" podUID="e141d579-fcbb-40f1-aa2c-16713029c97b" containerName="revoke-keys" Oct 31 00:40:26.091325 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:26.087648 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e141d579-fcbb-40f1-aa2c-16713029c97b" containerName="revoke-keys" Oct 31 00:40:26.091325 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:26.087679 5222 cpu_manager.go:414] "containerMap: removing container" podUID="2f791ab3-6734-4655-afb6-3267d91c961a" containerName="payload-processing" Oct 31 00:40:26.091325 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:26.087688 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f791ab3-6734-4655-afb6-3267d91c961a" containerName="payload-processing" Oct 31 00:40:26.091325 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:26.087748 5222 cpu_manager.go:414] "containerMap: removing container" podUID="0e8bd878-f701-4547-bc62-6ea4de9bb5e0" containerName="payload-pre-processing" Oct 31 00:40:26.091325 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:26.087762 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e8bd878-f701-4547-bc62-6ea4de9bb5e0" containerName="payload-pre-processing" Oct 31 00:40:26.091325 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:26.087927 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e8bd878-f701-4547-bc62-6ea4de9bb5e0" containerName="payload-pre-processing" Oct 31 00:40:26.091325 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:26.087951 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="e141d579-fcbb-40f1-aa2c-16713029c97b" containerName="revoke-keys" Oct 31 00:40:26.091325 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:26.087963 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f791ab3-6734-4655-afb6-3267d91c961a" containerName="payload-processing" Oct 31 00:40:26.093343 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:26.093319 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-dd862feb-openshift-default-67b7c76895-zrjbk" Oct 31 00:40:26.205328 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:26.172716 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-dd862feb-openshift-default-dockercfg-d9qvp\"" Oct 31 00:40:26.205328 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:26.178284 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7525d83-432f-4c20-aa47-6d51a4c88030-istio-token\") pod \"e2e-ait-dd862feb-openshift-default-67b7c76895-zrjbk\" (UID: \"d7525d83-432f-4c20-aa47-6d51a4c88030\") " pod="openshift-ingress/e2e-ait-dd862feb-openshift-default-67b7c76895-zrjbk" Oct 31 00:40:26.205328 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:26.178355 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7525d83-432f-4c20-aa47-6d51a4c88030-istio-podinfo\") pod \"e2e-ait-dd862feb-openshift-default-67b7c76895-zrjbk\" (UID: \"d7525d83-432f-4c20-aa47-6d51a4c88030\") " pod="openshift-ingress/e2e-ait-dd862feb-openshift-default-67b7c76895-zrjbk" Oct 31 00:40:26.205328 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:26.178419 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7525d83-432f-4c20-aa47-6d51a4c88030-istiod-ca-cert\") pod \"e2e-ait-dd862feb-openshift-default-67b7c76895-zrjbk\" (UID: \"d7525d83-432f-4c20-aa47-6d51a4c88030\") " pod="openshift-ingress/e2e-ait-dd862feb-openshift-default-67b7c76895-zrjbk" Oct 31 00:40:26.205328 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:26.178506 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7525d83-432f-4c20-aa47-6d51a4c88030-credential-socket\") pod \"e2e-ait-dd862feb-openshift-default-67b7c76895-zrjbk\" (UID: \"d7525d83-432f-4c20-aa47-6d51a4c88030\") " pod="openshift-ingress/e2e-ait-dd862feb-openshift-default-67b7c76895-zrjbk" Oct 31 00:40:26.205328 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:26.178543 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7525d83-432f-4c20-aa47-6d51a4c88030-istio-data\") pod \"e2e-ait-dd862feb-openshift-default-67b7c76895-zrjbk\" (UID: \"d7525d83-432f-4c20-aa47-6d51a4c88030\") " pod="openshift-ingress/e2e-ait-dd862feb-openshift-default-67b7c76895-zrjbk" Oct 31 00:40:26.205328 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:26.178569 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7525d83-432f-4c20-aa47-6d51a4c88030-workload-certs\") pod \"e2e-ait-dd862feb-openshift-default-67b7c76895-zrjbk\" (UID: \"d7525d83-432f-4c20-aa47-6d51a4c88030\") " pod="openshift-ingress/e2e-ait-dd862feb-openshift-default-67b7c76895-zrjbk" Oct 31 00:40:26.205328 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:26.178749 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzz5v\" (UniqueName: \"kubernetes.io/projected/d7525d83-432f-4c20-aa47-6d51a4c88030-kube-api-access-rzz5v\") pod \"e2e-ait-dd862feb-openshift-default-67b7c76895-zrjbk\" (UID: \"d7525d83-432f-4c20-aa47-6d51a4c88030\") " pod="openshift-ingress/e2e-ait-dd862feb-openshift-default-67b7c76895-zrjbk" Oct 31 00:40:26.205328 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:26.178830 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7525d83-432f-4c20-aa47-6d51a4c88030-istio-envoy\") pod \"e2e-ait-dd862feb-openshift-default-67b7c76895-zrjbk\" (UID: \"d7525d83-432f-4c20-aa47-6d51a4c88030\") " pod="openshift-ingress/e2e-ait-dd862feb-openshift-default-67b7c76895-zrjbk" Oct 31 00:40:26.205328 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:26.178871 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7525d83-432f-4c20-aa47-6d51a4c88030-workload-socket\") pod \"e2e-ait-dd862feb-openshift-default-67b7c76895-zrjbk\" (UID: \"d7525d83-432f-4c20-aa47-6d51a4c88030\") " pod="openshift-ingress/e2e-ait-dd862feb-openshift-default-67b7c76895-zrjbk" Oct 31 00:40:26.302197 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:26.302143 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzz5v\" (UniqueName: \"kubernetes.io/projected/d7525d83-432f-4c20-aa47-6d51a4c88030-kube-api-access-rzz5v\") pod \"e2e-ait-dd862feb-openshift-default-67b7c76895-zrjbk\" (UID: \"d7525d83-432f-4c20-aa47-6d51a4c88030\") " pod="openshift-ingress/e2e-ait-dd862feb-openshift-default-67b7c76895-zrjbk" Oct 31 00:40:26.303039 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:26.303007 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7525d83-432f-4c20-aa47-6d51a4c88030-istio-envoy\") pod \"e2e-ait-dd862feb-openshift-default-67b7c76895-zrjbk\" (UID: \"d7525d83-432f-4c20-aa47-6d51a4c88030\") " pod="openshift-ingress/e2e-ait-dd862feb-openshift-default-67b7c76895-zrjbk" Oct 31 00:40:26.303282 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:26.303263 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7525d83-432f-4c20-aa47-6d51a4c88030-workload-socket\") pod \"e2e-ait-dd862feb-openshift-default-67b7c76895-zrjbk\" (UID: \"d7525d83-432f-4c20-aa47-6d51a4c88030\") " pod="openshift-ingress/e2e-ait-dd862feb-openshift-default-67b7c76895-zrjbk" Oct 31 00:40:26.304498 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:26.304476 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7525d83-432f-4c20-aa47-6d51a4c88030-istio-token\") pod \"e2e-ait-dd862feb-openshift-default-67b7c76895-zrjbk\" (UID: \"d7525d83-432f-4c20-aa47-6d51a4c88030\") " pod="openshift-ingress/e2e-ait-dd862feb-openshift-default-67b7c76895-zrjbk" Oct 31 00:40:26.304685 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:26.304669 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7525d83-432f-4c20-aa47-6d51a4c88030-istio-podinfo\") pod \"e2e-ait-dd862feb-openshift-default-67b7c76895-zrjbk\" (UID: \"d7525d83-432f-4c20-aa47-6d51a4c88030\") " pod="openshift-ingress/e2e-ait-dd862feb-openshift-default-67b7c76895-zrjbk" Oct 31 00:40:26.304818 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:26.304803 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7525d83-432f-4c20-aa47-6d51a4c88030-istiod-ca-cert\") pod \"e2e-ait-dd862feb-openshift-default-67b7c76895-zrjbk\" (UID: \"d7525d83-432f-4c20-aa47-6d51a4c88030\") " pod="openshift-ingress/e2e-ait-dd862feb-openshift-default-67b7c76895-zrjbk" Oct 31 00:40:26.305042 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:26.304983 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7525d83-432f-4c20-aa47-6d51a4c88030-credential-socket\") pod \"e2e-ait-dd862feb-openshift-default-67b7c76895-zrjbk\" (UID: \"d7525d83-432f-4c20-aa47-6d51a4c88030\") " pod="openshift-ingress/e2e-ait-dd862feb-openshift-default-67b7c76895-zrjbk" Oct 31 00:40:26.305161 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:26.305148 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7525d83-432f-4c20-aa47-6d51a4c88030-istio-data\") pod \"e2e-ait-dd862feb-openshift-default-67b7c76895-zrjbk\" (UID: \"d7525d83-432f-4c20-aa47-6d51a4c88030\") " pod="openshift-ingress/e2e-ait-dd862feb-openshift-default-67b7c76895-zrjbk" Oct 31 00:40:26.305266 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:26.305251 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7525d83-432f-4c20-aa47-6d51a4c88030-workload-certs\") pod \"e2e-ait-dd862feb-openshift-default-67b7c76895-zrjbk\" (UID: \"d7525d83-432f-4c20-aa47-6d51a4c88030\") " pod="openshift-ingress/e2e-ait-dd862feb-openshift-default-67b7c76895-zrjbk" Oct 31 00:40:26.306000 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:26.305979 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7525d83-432f-4c20-aa47-6d51a4c88030-workload-certs\") pod \"e2e-ait-dd862feb-openshift-default-67b7c76895-zrjbk\" (UID: \"d7525d83-432f-4c20-aa47-6d51a4c88030\") " pod="openshift-ingress/e2e-ait-dd862feb-openshift-default-67b7c76895-zrjbk" Oct 31 00:40:26.308346 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:26.308322 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-dd862feb-openshift-default-67b7c76895-zrjbk"] Oct 31 00:40:26.308930 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:26.308912 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7525d83-432f-4c20-aa47-6d51a4c88030-workload-socket\") pod \"e2e-ait-dd862feb-openshift-default-67b7c76895-zrjbk\" (UID: \"d7525d83-432f-4c20-aa47-6d51a4c88030\") " pod="openshift-ingress/e2e-ait-dd862feb-openshift-default-67b7c76895-zrjbk" Oct 31 00:40:26.309324 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:26.309305 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7525d83-432f-4c20-aa47-6d51a4c88030-credential-socket\") pod \"e2e-ait-dd862feb-openshift-default-67b7c76895-zrjbk\" (UID: \"d7525d83-432f-4c20-aa47-6d51a4c88030\") " pod="openshift-ingress/e2e-ait-dd862feb-openshift-default-67b7c76895-zrjbk" Oct 31 00:40:26.310218 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:26.310196 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7525d83-432f-4c20-aa47-6d51a4c88030-istiod-ca-cert\") pod \"e2e-ait-dd862feb-openshift-default-67b7c76895-zrjbk\" (UID: \"d7525d83-432f-4c20-aa47-6d51a4c88030\") " pod="openshift-ingress/e2e-ait-dd862feb-openshift-default-67b7c76895-zrjbk" Oct 31 00:40:26.310682 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:26.310665 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7525d83-432f-4c20-aa47-6d51a4c88030-istio-data\") pod \"e2e-ait-dd862feb-openshift-default-67b7c76895-zrjbk\" (UID: \"d7525d83-432f-4c20-aa47-6d51a4c88030\") " pod="openshift-ingress/e2e-ait-dd862feb-openshift-default-67b7c76895-zrjbk" Oct 31 00:40:26.396067 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:26.395981 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7525d83-432f-4c20-aa47-6d51a4c88030-istio-token\") pod \"e2e-ait-dd862feb-openshift-default-67b7c76895-zrjbk\" (UID: \"d7525d83-432f-4c20-aa47-6d51a4c88030\") " pod="openshift-ingress/e2e-ait-dd862feb-openshift-default-67b7c76895-zrjbk" Oct 31 00:40:26.403296 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:26.403029 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzz5v\" (UniqueName: \"kubernetes.io/projected/d7525d83-432f-4c20-aa47-6d51a4c88030-kube-api-access-rzz5v\") pod \"e2e-ait-dd862feb-openshift-default-67b7c76895-zrjbk\" (UID: \"d7525d83-432f-4c20-aa47-6d51a4c88030\") " pod="openshift-ingress/e2e-ait-dd862feb-openshift-default-67b7c76895-zrjbk" Oct 31 00:40:26.409021 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:26.408986 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7525d83-432f-4c20-aa47-6d51a4c88030-istio-envoy\") pod \"e2e-ait-dd862feb-openshift-default-67b7c76895-zrjbk\" (UID: \"d7525d83-432f-4c20-aa47-6d51a4c88030\") " pod="openshift-ingress/e2e-ait-dd862feb-openshift-default-67b7c76895-zrjbk" Oct 31 00:40:26.413257 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:26.413170 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7525d83-432f-4c20-aa47-6d51a4c88030-istio-podinfo\") pod \"e2e-ait-dd862feb-openshift-default-67b7c76895-zrjbk\" (UID: \"d7525d83-432f-4c20-aa47-6d51a4c88030\") " pod="openshift-ingress/e2e-ait-dd862feb-openshift-default-67b7c76895-zrjbk" Oct 31 00:40:26.469392 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:26.459663 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-dd862feb-openshift-default-67b7c76895-zrjbk" Oct 31 00:40:26.948818 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:40:26.948776 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7525d83_432f_4c20_aa47_6d51a4c88030.slice/crio-dfa827e91a34fac84923099bb584ca887c2c13d56d117a252351bc6f16cd3826 WatchSource:0}: Error finding container dfa827e91a34fac84923099bb584ca887c2c13d56d117a252351bc6f16cd3826: Status 404 returned error can't find the container with id dfa827e91a34fac84923099bb584ca887c2c13d56d117a252351bc6f16cd3826 Oct 31 00:40:26.951481 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:26.951443 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:40:26.951590 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:26.951515 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:40:26.951590 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:26.951544 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:40:26.953706 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:26.953681 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-dd862feb-openshift-default-67b7c76895-zrjbk"] Oct 31 00:40:27.325872 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:27.325836 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-dd862feb-openshift-default-67b7c76895-zrjbk" event={"ID":"d7525d83-432f-4c20-aa47-6d51a4c88030","Type":"ContainerStarted","Data":"8832255eadbeeddacb5f8ede93066f4fe2e30ad169a899e6f6541120691329b8"} Oct 31 00:40:27.326149 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:27.325907 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-dd862feb-openshift-default-67b7c76895-zrjbk" event={"ID":"d7525d83-432f-4c20-aa47-6d51a4c88030","Type":"ContainerStarted","Data":"dfa827e91a34fac84923099bb584ca887c2c13d56d117a252351bc6f16cd3826"} Oct 31 00:40:27.454717 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:27.454657 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-dd862feb-openshift-default-67b7c76895-zrjbk" podStartSLOduration=2.4546394830000002 podStartE2EDuration="2.454639483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:40:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:40:27.4518369 +0000 UTC m=+1519.923006648" watchObservedRunningTime="2026-10-31 00:40:27.454639483 +0000 UTC m=+1519.925809232" Oct 31 00:40:27.460546 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:27.460505 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-dd862feb-openshift-default-67b7c76895-zrjbk" Oct 31 00:40:28.466830 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:28.466802 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-dd862feb-openshift-default-67b7c76895-zrjbk" Oct 31 00:40:29.484835 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:29.484804 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-dd862feb-openshift-default-67b7c76895-zrjbk" Oct 31 00:40:29.487237 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:29.487218 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-dd862feb-openshift-default-67b7c76895-zrjbk" Oct 31 00:40:31.058242 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:31.058202 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5cc9374-m57hz"] Oct 31 00:40:31.153400 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:31.142618 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5cc9374-m57hz"] Oct 31 00:40:31.379723 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:31.379639 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e7b593fa-s667c"] Oct 31 00:40:31.385813 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:31.385773 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e7b593fa-s667c"] Oct 31 00:40:31.659824 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:31.659745 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c55e12e-e99d-48c7-b2cd-d094309b4035" path="/var/lib/kubelet/pods/3c55e12e-e99d-48c7-b2cd-d094309b4035/volumes" Oct 31 00:40:31.660249 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:31.660235 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7277c6d2-52f6-4109-b39d-786d6b41713d" path="/var/lib/kubelet/pods/7277c6d2-52f6-4109-b39d-786d6b41713d/volumes" Oct 31 00:40:33.831947 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:33.831857 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-dd862feb-6c5d597495-9ftp6"] Oct 31 00:40:33.837766 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:33.837736 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-dd862feb-6c5d597495-9ftp6" Oct 31 00:40:33.897286 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:33.897253 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-dd862feb-6c5d597495-9ftp6"] Oct 31 00:40:33.932686 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:33.932652 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-dd862feb\"" Oct 31 00:40:34.031398 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:34.006143 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzjmp\" (UniqueName: \"kubernetes.io/projected/bdbc3343-c465-4898-a845-f005363dbd36-kube-api-access-fzjmp\") pod \"payload-pre-processing-e2e-ait-dd862feb-6c5d597495-9ftp6\" (UID: \"bdbc3343-c465-4898-a845-f005363dbd36\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-dd862feb-6c5d597495-9ftp6" Oct 31 00:40:34.050483 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:34.050444 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdbc3343-c465-4898-a845-f005363dbd36-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-dd862feb-6c5d597495-9ftp6\" (UID: \"bdbc3343-c465-4898-a845-f005363dbd36\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-dd862feb-6c5d597495-9ftp6" Oct 31 00:40:34.071475 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:34.071425 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-dd862feb-9c885f569-n957f"] Oct 31 00:40:34.101778 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:34.101694 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-dd862feb-9c885f569-n957f" Oct 31 00:40:34.145671 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:34.145637 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-dd862feb-9c885f569-n957f"] Oct 31 00:40:34.165395 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:34.161740 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-dd862feb-dockercfg-fw6s5\"" Oct 31 00:40:34.172460 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:34.169208 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39ec242a-8946-48a5-b29a-47a211e70144-plugins-config-volume\") pod \"payload-processing-e2e-ait-dd862feb-9c885f569-n957f\" (UID: \"39ec242a-8946-48a5-b29a-47a211e70144\") " pod="openshift-ingress/payload-processing-e2e-ait-dd862feb-9c885f569-n957f" Oct 31 00:40:34.193401 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:34.172839 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdbc3343-c465-4898-a845-f005363dbd36-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-dd862feb-6c5d597495-9ftp6\" (UID: \"bdbc3343-c465-4898-a845-f005363dbd36\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-dd862feb-6c5d597495-9ftp6" Oct 31 00:40:34.193864 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:34.193836 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzjmp\" (UniqueName: \"kubernetes.io/projected/bdbc3343-c465-4898-a845-f005363dbd36-kube-api-access-fzjmp\") pod \"payload-pre-processing-e2e-ait-dd862feb-6c5d597495-9ftp6\" (UID: \"bdbc3343-c465-4898-a845-f005363dbd36\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-dd862feb-6c5d597495-9ftp6" Oct 31 00:40:34.194137 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:34.194115 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmhtp\" (UniqueName: \"kubernetes.io/projected/39ec242a-8946-48a5-b29a-47a211e70144-kube-api-access-nmhtp\") pod \"payload-processing-e2e-ait-dd862feb-9c885f569-n957f\" (UID: \"39ec242a-8946-48a5-b29a-47a211e70144\") " pod="openshift-ingress/payload-processing-e2e-ait-dd862feb-9c885f569-n957f" Oct 31 00:40:34.211393 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:34.183312 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdbc3343-c465-4898-a845-f005363dbd36-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-dd862feb-6c5d597495-9ftp6\" (UID: \"bdbc3343-c465-4898-a845-f005363dbd36\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-dd862feb-6c5d597495-9ftp6" Oct 31 00:40:34.258804 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:34.258763 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzjmp\" (UniqueName: \"kubernetes.io/projected/bdbc3343-c465-4898-a845-f005363dbd36-kube-api-access-fzjmp\") pod \"payload-pre-processing-e2e-ait-dd862feb-6c5d597495-9ftp6\" (UID: \"bdbc3343-c465-4898-a845-f005363dbd36\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-dd862feb-6c5d597495-9ftp6" Oct 31 00:40:34.295625 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:34.295574 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmhtp\" (UniqueName: \"kubernetes.io/projected/39ec242a-8946-48a5-b29a-47a211e70144-kube-api-access-nmhtp\") pod \"payload-processing-e2e-ait-dd862feb-9c885f569-n957f\" (UID: \"39ec242a-8946-48a5-b29a-47a211e70144\") " pod="openshift-ingress/payload-processing-e2e-ait-dd862feb-9c885f569-n957f" Oct 31 00:40:34.295827 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:34.295687 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39ec242a-8946-48a5-b29a-47a211e70144-plugins-config-volume\") pod \"payload-processing-e2e-ait-dd862feb-9c885f569-n957f\" (UID: \"39ec242a-8946-48a5-b29a-47a211e70144\") " pod="openshift-ingress/payload-processing-e2e-ait-dd862feb-9c885f569-n957f" Oct 31 00:40:34.296455 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:34.296428 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39ec242a-8946-48a5-b29a-47a211e70144-plugins-config-volume\") pod \"payload-processing-e2e-ait-dd862feb-9c885f569-n957f\" (UID: \"39ec242a-8946-48a5-b29a-47a211e70144\") " pod="openshift-ingress/payload-processing-e2e-ait-dd862feb-9c885f569-n957f" Oct 31 00:40:34.357977 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:34.357897 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmhtp\" (UniqueName: \"kubernetes.io/projected/39ec242a-8946-48a5-b29a-47a211e70144-kube-api-access-nmhtp\") pod \"payload-processing-e2e-ait-dd862feb-9c885f569-n957f\" (UID: \"39ec242a-8946-48a5-b29a-47a211e70144\") " pod="openshift-ingress/payload-processing-e2e-ait-dd862feb-9c885f569-n957f" Oct 31 00:40:34.438801 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:34.438767 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-dd862feb-9c885f569-n957f" Oct 31 00:40:34.453340 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:34.453303 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-dd862feb-6c5d597495-9ftp6" Oct 31 00:40:34.835895 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:40:34.835863 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39ec242a_8946_48a5_b29a_47a211e70144.slice/crio-61898b83cfcdd1bd8e9e95a0ca0f5dadce46d9e15abb1d7934af127564b0b998 WatchSource:0}: Error finding container 61898b83cfcdd1bd8e9e95a0ca0f5dadce46d9e15abb1d7934af127564b0b998: Status 404 returned error can't find the container with id 61898b83cfcdd1bd8e9e95a0ca0f5dadce46d9e15abb1d7934af127564b0b998 Oct 31 00:40:34.840731 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:34.840705 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-dd862feb-9c885f569-n957f"] Oct 31 00:40:34.884499 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:40:34.884465 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdbc3343_c465_4898_a845_f005363dbd36.slice/crio-4cbe62f383c79753d55a7687f3ea3265a80e81f3b9bd3d8fcdb09b43be8bcd00 WatchSource:0}: Error finding container 4cbe62f383c79753d55a7687f3ea3265a80e81f3b9bd3d8fcdb09b43be8bcd00: Status 404 returned error can't find the container with id 4cbe62f383c79753d55a7687f3ea3265a80e81f3b9bd3d8fcdb09b43be8bcd00 Oct 31 00:40:34.888430 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:34.888397 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-dd862feb-6c5d597495-9ftp6"] Oct 31 00:40:35.253703 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:35.253613 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-79tk8"] Oct 31 00:40:35.722586 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:35.722547 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-dd862feb-6c5d597495-9ftp6" event={"ID":"bdbc3343-c465-4898-a845-f005363dbd36","Type":"ContainerStarted","Data":"a3e50288e2008ff08e5ee50ceed029dcba707e6fd0035d08ae21ed5e90aa2fe4"} Oct 31 00:40:35.722786 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:35.722613 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-dd862feb-6c5d597495-9ftp6" event={"ID":"bdbc3343-c465-4898-a845-f005363dbd36","Type":"ContainerStarted","Data":"4cbe62f383c79753d55a7687f3ea3265a80e81f3b9bd3d8fcdb09b43be8bcd00"} Oct 31 00:40:35.722786 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:35.722645 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-dd862feb-6c5d597495-9ftp6" Oct 31 00:40:35.724334 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:35.724307 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-dd862feb-9c885f569-n957f" event={"ID":"39ec242a-8946-48a5-b29a-47a211e70144","Type":"ContainerStarted","Data":"fb72a1f23513ed2b1d852788442913aea9d1f648af6066ea074a9a5b793398ec"} Oct 31 00:40:35.724514 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:35.724358 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-dd862feb-9c885f569-n957f" event={"ID":"39ec242a-8946-48a5-b29a-47a211e70144","Type":"ContainerStarted","Data":"61898b83cfcdd1bd8e9e95a0ca0f5dadce46d9e15abb1d7934af127564b0b998"} Oct 31 00:40:35.724514 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:35.724494 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-dd862feb-9c885f569-n957f" Oct 31 00:40:35.848398 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:35.848316 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-dd862feb-6c5d597495-9ftp6" podStartSLOduration=2.8482941139999998 podStartE2EDuration="2.848294114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:40:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:40:35.841314183 +0000 UTC m=+1528.312483932" watchObservedRunningTime="2026-10-31 00:40:35.848294114 +0000 UTC m=+1528.319463866" Oct 31 00:40:35.891789 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:40:35.891753 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 00:40:35.969108 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:35.969042 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-dd862feb-9c885f569-n957f" podStartSLOduration=2.969024272 podStartE2EDuration="2.969024272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:40:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:40:35.964079292 +0000 UTC m=+1528.435249045" watchObservedRunningTime="2026-10-31 00:40:35.969024272 +0000 UTC m=+1528.440194022" Oct 31 00:40:38.962573 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:38.962507 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-49bb6b1a-gjbsr"] Oct 31 00:40:39.016094 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:39.016061 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-49bb6b1a-gjbsr"] Oct 31 00:40:39.662233 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:39.662191 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e141d579-fcbb-40f1-aa2c-16713029c97b" path="/var/lib/kubelet/pods/e141d579-fcbb-40f1-aa2c-16713029c97b/volumes" Oct 31 00:40:46.736979 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:46.736773 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-dd862feb-6c5d597495-9ftp6" Oct 31 00:40:46.737459 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:46.737208 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-dd862feb-9c885f569-n957f" Oct 31 00:40:47.559108 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:47.559076 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b1262dc-znvmb"] Oct 31 00:40:47.564771 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:47.564751 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b1262dc-znvmb" Oct 31 00:40:47.618714 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:47.618680 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:40:47.619006 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:47.618695 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:40:47.619520 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:47.619487 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nml2h\" (UniqueName: \"kubernetes.io/projected/b6a3822f-439f-4f89-a225-49c93c09e17f-kube-api-access-nml2h\") pod \"maas-api-revoke-sub-7b1262dc-znvmb\" (UID: \"b6a3822f-439f-4f89-a225-49c93c09e17f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b1262dc-znvmb" Oct 31 00:40:47.619639 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:47.619605 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b1262dc-znvmb"] Oct 31 00:40:47.619725 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:47.619703 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6a3822f-439f-4f89-a225-49c93c09e17f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b1262dc-znvmb\" (UID: \"b6a3822f-439f-4f89-a225-49c93c09e17f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b1262dc-znvmb" Oct 31 00:40:47.619827 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:47.619802 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6a3822f-439f-4f89-a225-49c93c09e17f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b1262dc-znvmb\" (UID: \"b6a3822f-439f-4f89-a225-49c93c09e17f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b1262dc-znvmb" Oct 31 00:40:47.663250 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:47.663225 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfz9z\"" Oct 31 00:40:47.669926 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:47.669903 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:40:47.720923 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:47.720891 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6a3822f-439f-4f89-a225-49c93c09e17f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b1262dc-znvmb\" (UID: \"b6a3822f-439f-4f89-a225-49c93c09e17f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b1262dc-znvmb" Oct 31 00:40:47.721102 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:47.720931 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6a3822f-439f-4f89-a225-49c93c09e17f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b1262dc-znvmb\" (UID: \"b6a3822f-439f-4f89-a225-49c93c09e17f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b1262dc-znvmb" Oct 31 00:40:47.721102 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:47.721075 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nml2h\" (UniqueName: \"kubernetes.io/projected/b6a3822f-439f-4f89-a225-49c93c09e17f-kube-api-access-nml2h\") pod \"maas-api-revoke-sub-7b1262dc-znvmb\" (UID: \"b6a3822f-439f-4f89-a225-49c93c09e17f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b1262dc-znvmb" Oct 31 00:40:47.721747 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:47.721720 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6a3822f-439f-4f89-a225-49c93c09e17f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b1262dc-znvmb\" (UID: \"b6a3822f-439f-4f89-a225-49c93c09e17f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b1262dc-znvmb" Oct 31 00:40:47.723804 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:47.723780 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6a3822f-439f-4f89-a225-49c93c09e17f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b1262dc-znvmb\" (UID: \"b6a3822f-439f-4f89-a225-49c93c09e17f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b1262dc-znvmb" Oct 31 00:40:47.784041 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:47.784009 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nml2h\" (UniqueName: \"kubernetes.io/projected/b6a3822f-439f-4f89-a225-49c93c09e17f-kube-api-access-nml2h\") pod \"maas-api-revoke-sub-7b1262dc-znvmb\" (UID: \"b6a3822f-439f-4f89-a225-49c93c09e17f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b1262dc-znvmb" Oct 31 00:40:47.880148 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:47.880060 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b1262dc-znvmb" Oct 31 00:40:48.265801 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:40:48.265769 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6a3822f_439f_4f89_a225_49c93c09e17f.slice/crio-571afad6a027132b26cd8917b53bd6e32c4c81ec2ef5bebb6e7a70b0b6467c71 WatchSource:0}: Error finding container 571afad6a027132b26cd8917b53bd6e32c4c81ec2ef5bebb6e7a70b0b6467c71: Status 404 returned error can't find the container with id 571afad6a027132b26cd8917b53bd6e32c4c81ec2ef5bebb6e7a70b0b6467c71 Oct 31 00:40:48.269975 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:48.269950 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b1262dc-znvmb"] Oct 31 00:40:49.009549 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:49.008140 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b1262dc-znvmb" event={"ID":"b6a3822f-439f-4f89-a225-49c93c09e17f","Type":"ContainerStarted","Data":"9f6dd793ff8d0d0f8c0af3501a89585eb64eedaadc9abf3f077c7765644cdb41"} Oct 31 00:40:49.009549 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:49.008215 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b1262dc-znvmb" event={"ID":"b6a3822f-439f-4f89-a225-49c93c09e17f","Type":"ContainerStarted","Data":"571afad6a027132b26cd8917b53bd6e32c4c81ec2ef5bebb6e7a70b0b6467c71"} Oct 31 00:40:49.141731 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:49.141679 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b1262dc-znvmb" podStartSLOduration=2.141663675 podStartE2EDuration="2.141663675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:40:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:40:49.138744039 +0000 UTC m=+1541.609913789" watchObservedRunningTime="2026-10-31 00:40:49.141663675 +0000 UTC m=+1541.612833423" Oct 31 00:40:50.018790 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:50.018750 5222 generic.go:356] "Generic (PLEG): container finished" podID="b6a3822f-439f-4f89-a225-49c93c09e17f" containerID="9f6dd793ff8d0d0f8c0af3501a89585eb64eedaadc9abf3f077c7765644cdb41" exitCode=0 Oct 31 00:40:50.019216 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:50.018845 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b1262dc-znvmb" event={"ID":"b6a3822f-439f-4f89-a225-49c93c09e17f","Type":"ContainerDied","Data":"9f6dd793ff8d0d0f8c0af3501a89585eb64eedaadc9abf3f077c7765644cdb41"} Oct 31 00:40:51.203217 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:51.203191 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b1262dc-znvmb" Oct 31 00:40:51.276587 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:51.276537 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6a3822f-439f-4f89-a225-49c93c09e17f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6a3822f-439f-4f89-a225-49c93c09e17f-maas-api-xks-ca\") pod \"b6a3822f-439f-4f89-a225-49c93c09e17f\" (UID: \"b6a3822f-439f-4f89-a225-49c93c09e17f\") " Oct 31 00:40:51.276804 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:51.276660 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6a3822f-439f-4f89-a225-49c93c09e17f-kube-api-access-nml2h\" (UniqueName: \"kubernetes.io/projected/b6a3822f-439f-4f89-a225-49c93c09e17f-kube-api-access-nml2h\") pod \"b6a3822f-439f-4f89-a225-49c93c09e17f\" (UID: \"b6a3822f-439f-4f89-a225-49c93c09e17f\") " Oct 31 00:40:51.276804 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:51.276707 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6a3822f-439f-4f89-a225-49c93c09e17f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6a3822f-439f-4f89-a225-49c93c09e17f-maas-api-openshift-ca\") pod \"b6a3822f-439f-4f89-a225-49c93c09e17f\" (UID: \"b6a3822f-439f-4f89-a225-49c93c09e17f\") " Oct 31 00:40:51.277094 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:51.277072 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6a3822f-439f-4f89-a225-49c93c09e17f-maas-api-openshift-ca" pod "b6a3822f-439f-4f89-a225-49c93c09e17f" (UID: "b6a3822f-439f-4f89-a225-49c93c09e17f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:40:51.277500 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:51.277469 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6a3822f-439f-4f89-a225-49c93c09e17f-maas-api-openshift-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:40:51.278897 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:51.278874 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6a3822f-439f-4f89-a225-49c93c09e17f-maas-api-xks-ca" pod "b6a3822f-439f-4f89-a225-49c93c09e17f" (UID: "b6a3822f-439f-4f89-a225-49c93c09e17f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:40:51.279296 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:51.279271 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6a3822f-439f-4f89-a225-49c93c09e17f-kube-api-access-nml2h" pod "b6a3822f-439f-4f89-a225-49c93c09e17f" (UID: "b6a3822f-439f-4f89-a225-49c93c09e17f"). InnerVolumeSpecName "kube-api-access-nml2h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:40:51.379263 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:51.379222 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6a3822f-439f-4f89-a225-49c93c09e17f-maas-api-xks-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:40:51.379263 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:51.379259 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nml2h\" (UniqueName: \"kubernetes.io/projected/b6a3822f-439f-4f89-a225-49c93c09e17f-kube-api-access-nml2h\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:40:52.035779 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:52.035749 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b1262dc-znvmb" Oct 31 00:40:52.035994 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:52.035780 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b1262dc-znvmb" event={"ID":"b6a3822f-439f-4f89-a225-49c93c09e17f","Type":"ContainerDied","Data":"571afad6a027132b26cd8917b53bd6e32c4c81ec2ef5bebb6e7a70b0b6467c71"} Oct 31 00:40:52.035994 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:52.035827 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="571afad6a027132b26cd8917b53bd6e32c4c81ec2ef5bebb6e7a70b0b6467c71" Oct 31 00:40:53.866535 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:53.866490 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-r57tc"] Oct 31 00:40:53.867319 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:53.867293 5222 cpu_manager.go:414] "containerMap: removing container" podUID="b6a3822f-439f-4f89-a225-49c93c09e17f" containerName="revoke-keys" Oct 31 00:40:53.867448 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:53.867322 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6a3822f-439f-4f89-a225-49c93c09e17f" containerName="revoke-keys" Oct 31 00:40:53.867492 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:53.867474 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6a3822f-439f-4f89-a225-49c93c09e17f" containerName="revoke-keys" Oct 31 00:40:53.871203 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:53.871176 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-r57tc" Oct 31 00:40:53.987235 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:53.987013 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mjxmq"] Oct 31 00:40:53.995212 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:53.994658 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mjxmq" Oct 31 00:40:53.998031 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:53.998002 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-r57tc"] Oct 31 00:40:54.016481 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:54.016447 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mjxmq"] Oct 31 00:40:54.043417 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:54.043354 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33d0dda2-34fd-409e-9165-5861f2373db9-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-r57tc\" (UID: \"33d0dda2-34fd-409e-9165-5861f2373db9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-r57tc" Oct 31 00:40:54.043599 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:54.043450 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbvqj\" (UniqueName: \"kubernetes.io/projected/33d0dda2-34fd-409e-9165-5861f2373db9-kube-api-access-vbvqj\") pod \"maas-default-gateway-openshift-default-6886b96f9-r57tc\" (UID: \"33d0dda2-34fd-409e-9165-5861f2373db9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-r57tc" Oct 31 00:40:54.043599 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:54.043528 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33d0dda2-34fd-409e-9165-5861f2373db9-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-r57tc\" (UID: \"33d0dda2-34fd-409e-9165-5861f2373db9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-r57tc" Oct 31 00:40:54.043599 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:54.043574 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33d0dda2-34fd-409e-9165-5861f2373db9-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-r57tc\" (UID: \"33d0dda2-34fd-409e-9165-5861f2373db9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-r57tc" Oct 31 00:40:54.043746 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:54.043655 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33d0dda2-34fd-409e-9165-5861f2373db9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-r57tc\" (UID: \"33d0dda2-34fd-409e-9165-5861f2373db9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-r57tc" Oct 31 00:40:54.043801 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:54.043753 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33d0dda2-34fd-409e-9165-5861f2373db9-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-r57tc\" (UID: \"33d0dda2-34fd-409e-9165-5861f2373db9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-r57tc" Oct 31 00:40:54.043918 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:54.043892 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33d0dda2-34fd-409e-9165-5861f2373db9-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-r57tc\" (UID: \"33d0dda2-34fd-409e-9165-5861f2373db9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-r57tc" Oct 31 00:40:54.044074 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:54.043933 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33d0dda2-34fd-409e-9165-5861f2373db9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-r57tc\" (UID: \"33d0dda2-34fd-409e-9165-5861f2373db9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-r57tc" Oct 31 00:40:54.044074 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:54.043971 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33d0dda2-34fd-409e-9165-5861f2373db9-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-r57tc\" (UID: \"33d0dda2-34fd-409e-9165-5861f2373db9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-r57tc" Oct 31 00:40:54.146031 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:54.145931 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33d0dda2-34fd-409e-9165-5861f2373db9-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-r57tc\" (UID: \"33d0dda2-34fd-409e-9165-5861f2373db9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-r57tc" Oct 31 00:40:54.146031 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:54.145980 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b9592e3-41e5-4ada-9e42-ea705c3eb1f3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-mjxmq\" (UID: \"0b9592e3-41e5-4ada-9e42-ea705c3eb1f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mjxmq" Oct 31 00:40:54.146031 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:54.146001 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbvqj\" (UniqueName: \"kubernetes.io/projected/33d0dda2-34fd-409e-9165-5861f2373db9-kube-api-access-vbvqj\") pod \"maas-default-gateway-openshift-default-6886b96f9-r57tc\" (UID: \"33d0dda2-34fd-409e-9165-5861f2373db9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-r57tc" Oct 31 00:40:54.146320 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:54.146039 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b9592e3-41e5-4ada-9e42-ea705c3eb1f3-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-mjxmq\" (UID: \"0b9592e3-41e5-4ada-9e42-ea705c3eb1f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mjxmq" Oct 31 00:40:54.146320 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:54.146168 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33d0dda2-34fd-409e-9165-5861f2373db9-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-r57tc\" (UID: \"33d0dda2-34fd-409e-9165-5861f2373db9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-r57tc" Oct 31 00:40:54.146320 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:54.146208 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b9592e3-41e5-4ada-9e42-ea705c3eb1f3-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-mjxmq\" (UID: \"0b9592e3-41e5-4ada-9e42-ea705c3eb1f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mjxmq" Oct 31 00:40:54.146320 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:54.146262 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33d0dda2-34fd-409e-9165-5861f2373db9-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-r57tc\" (UID: \"33d0dda2-34fd-409e-9165-5861f2373db9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-r57tc" Oct 31 00:40:54.146541 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:54.146480 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b9592e3-41e5-4ada-9e42-ea705c3eb1f3-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-mjxmq\" (UID: \"0b9592e3-41e5-4ada-9e42-ea705c3eb1f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mjxmq" Oct 31 00:40:54.146603 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:54.146581 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b9592e3-41e5-4ada-9e42-ea705c3eb1f3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-mjxmq\" (UID: \"0b9592e3-41e5-4ada-9e42-ea705c3eb1f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mjxmq" Oct 31 00:40:54.146651 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:54.146617 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33d0dda2-34fd-409e-9165-5861f2373db9-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-r57tc\" (UID: \"33d0dda2-34fd-409e-9165-5861f2373db9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-r57tc" Oct 31 00:40:54.146651 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:54.146618 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33d0dda2-34fd-409e-9165-5861f2373db9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-r57tc\" (UID: \"33d0dda2-34fd-409e-9165-5861f2373db9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-r57tc" Oct 31 00:40:54.146651 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:54.146637 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33d0dda2-34fd-409e-9165-5861f2373db9-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-r57tc\" (UID: \"33d0dda2-34fd-409e-9165-5861f2373db9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-r57tc" Oct 31 00:40:54.146773 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:54.146676 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbtc5\" (UniqueName: \"kubernetes.io/projected/0b9592e3-41e5-4ada-9e42-ea705c3eb1f3-kube-api-access-qbtc5\") pod \"maas-default-gateway-openshift-default-6886b96f9-mjxmq\" (UID: \"0b9592e3-41e5-4ada-9e42-ea705c3eb1f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mjxmq" Oct 31 00:40:54.146773 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:54.146732 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33d0dda2-34fd-409e-9165-5861f2373db9-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-r57tc\" (UID: \"33d0dda2-34fd-409e-9165-5861f2373db9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-r57tc" Oct 31 00:40:54.146920 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:54.146900 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33d0dda2-34fd-409e-9165-5861f2373db9-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-r57tc\" (UID: \"33d0dda2-34fd-409e-9165-5861f2373db9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-r57tc" Oct 31 00:40:54.146969 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:54.146942 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33d0dda2-34fd-409e-9165-5861f2373db9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-r57tc\" (UID: \"33d0dda2-34fd-409e-9165-5861f2373db9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-r57tc" Oct 31 00:40:54.147012 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:54.146979 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33d0dda2-34fd-409e-9165-5861f2373db9-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-r57tc\" (UID: \"33d0dda2-34fd-409e-9165-5861f2373db9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-r57tc" Oct 31 00:40:54.147063 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:54.147010 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33d0dda2-34fd-409e-9165-5861f2373db9-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-r57tc\" (UID: \"33d0dda2-34fd-409e-9165-5861f2373db9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-r57tc" Oct 31 00:40:54.147117 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:54.147082 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b9592e3-41e5-4ada-9e42-ea705c3eb1f3-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-mjxmq\" (UID: \"0b9592e3-41e5-4ada-9e42-ea705c3eb1f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mjxmq" Oct 31 00:40:54.147168 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:54.147113 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b9592e3-41e5-4ada-9e42-ea705c3eb1f3-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-mjxmq\" (UID: \"0b9592e3-41e5-4ada-9e42-ea705c3eb1f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mjxmq" Oct 31 00:40:54.147168 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:54.147159 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b9592e3-41e5-4ada-9e42-ea705c3eb1f3-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-mjxmq\" (UID: \"0b9592e3-41e5-4ada-9e42-ea705c3eb1f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mjxmq" Oct 31 00:40:54.147277 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:54.147202 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33d0dda2-34fd-409e-9165-5861f2373db9-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-r57tc\" (UID: \"33d0dda2-34fd-409e-9165-5861f2373db9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-r57tc" Oct 31 00:40:54.147688 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:54.147668 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33d0dda2-34fd-409e-9165-5861f2373db9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-r57tc\" (UID: \"33d0dda2-34fd-409e-9165-5861f2373db9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-r57tc" Oct 31 00:40:54.148728 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:54.148672 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33d0dda2-34fd-409e-9165-5861f2373db9-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-r57tc\" (UID: \"33d0dda2-34fd-409e-9165-5861f2373db9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-r57tc" Oct 31 00:40:54.149656 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:54.149632 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33d0dda2-34fd-409e-9165-5861f2373db9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-r57tc\" (UID: \"33d0dda2-34fd-409e-9165-5861f2373db9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-r57tc" Oct 31 00:40:54.214072 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:54.214043 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbvqj\" (UniqueName: \"kubernetes.io/projected/33d0dda2-34fd-409e-9165-5861f2373db9-kube-api-access-vbvqj\") pod \"maas-default-gateway-openshift-default-6886b96f9-r57tc\" (UID: \"33d0dda2-34fd-409e-9165-5861f2373db9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-r57tc" Oct 31 00:40:54.214722 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:54.214697 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33d0dda2-34fd-409e-9165-5861f2373db9-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-r57tc\" (UID: \"33d0dda2-34fd-409e-9165-5861f2373db9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-r57tc" Oct 31 00:40:54.248153 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:54.248115 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b9592e3-41e5-4ada-9e42-ea705c3eb1f3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-mjxmq\" (UID: \"0b9592e3-41e5-4ada-9e42-ea705c3eb1f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mjxmq" Oct 31 00:40:54.248341 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:54.248175 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b9592e3-41e5-4ada-9e42-ea705c3eb1f3-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-mjxmq\" (UID: \"0b9592e3-41e5-4ada-9e42-ea705c3eb1f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mjxmq" Oct 31 00:40:54.248457 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:54.248335 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b9592e3-41e5-4ada-9e42-ea705c3eb1f3-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-mjxmq\" (UID: \"0b9592e3-41e5-4ada-9e42-ea705c3eb1f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mjxmq" Oct 31 00:40:54.248536 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:54.248462 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b9592e3-41e5-4ada-9e42-ea705c3eb1f3-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-mjxmq\" (UID: \"0b9592e3-41e5-4ada-9e42-ea705c3eb1f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mjxmq" Oct 31 00:40:54.248536 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:54.248503 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b9592e3-41e5-4ada-9e42-ea705c3eb1f3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-mjxmq\" (UID: \"0b9592e3-41e5-4ada-9e42-ea705c3eb1f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mjxmq" Oct 31 00:40:54.248536 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:54.248522 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbtc5\" (UniqueName: \"kubernetes.io/projected/0b9592e3-41e5-4ada-9e42-ea705c3eb1f3-kube-api-access-qbtc5\") pod \"maas-default-gateway-openshift-default-6886b96f9-mjxmq\" (UID: \"0b9592e3-41e5-4ada-9e42-ea705c3eb1f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mjxmq" Oct 31 00:40:54.248696 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:54.248632 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b9592e3-41e5-4ada-9e42-ea705c3eb1f3-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-mjxmq\" (UID: \"0b9592e3-41e5-4ada-9e42-ea705c3eb1f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mjxmq" Oct 31 00:40:54.248748 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:54.248701 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b9592e3-41e5-4ada-9e42-ea705c3eb1f3-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-mjxmq\" (UID: \"0b9592e3-41e5-4ada-9e42-ea705c3eb1f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mjxmq" Oct 31 00:40:54.248748 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:54.248726 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b9592e3-41e5-4ada-9e42-ea705c3eb1f3-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-mjxmq\" (UID: \"0b9592e3-41e5-4ada-9e42-ea705c3eb1f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mjxmq" Oct 31 00:40:54.248843 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:54.248781 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b9592e3-41e5-4ada-9e42-ea705c3eb1f3-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-mjxmq\" (UID: \"0b9592e3-41e5-4ada-9e42-ea705c3eb1f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mjxmq" Oct 31 00:40:54.248843 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:54.248790 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b9592e3-41e5-4ada-9e42-ea705c3eb1f3-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-mjxmq\" (UID: \"0b9592e3-41e5-4ada-9e42-ea705c3eb1f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mjxmq" Oct 31 00:40:54.248939 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:54.248876 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b9592e3-41e5-4ada-9e42-ea705c3eb1f3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-mjxmq\" (UID: \"0b9592e3-41e5-4ada-9e42-ea705c3eb1f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mjxmq" Oct 31 00:40:54.249169 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:54.249145 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b9592e3-41e5-4ada-9e42-ea705c3eb1f3-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-mjxmq\" (UID: \"0b9592e3-41e5-4ada-9e42-ea705c3eb1f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mjxmq" Oct 31 00:40:54.249329 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:54.249308 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b9592e3-41e5-4ada-9e42-ea705c3eb1f3-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-mjxmq\" (UID: \"0b9592e3-41e5-4ada-9e42-ea705c3eb1f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mjxmq" Oct 31 00:40:54.250863 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:54.250845 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b9592e3-41e5-4ada-9e42-ea705c3eb1f3-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-mjxmq\" (UID: \"0b9592e3-41e5-4ada-9e42-ea705c3eb1f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mjxmq" Oct 31 00:40:54.251063 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:54.251046 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b9592e3-41e5-4ada-9e42-ea705c3eb1f3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-mjxmq\" (UID: \"0b9592e3-41e5-4ada-9e42-ea705c3eb1f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mjxmq" Oct 31 00:40:54.311357 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:54.311330 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b9592e3-41e5-4ada-9e42-ea705c3eb1f3-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-mjxmq\" (UID: \"0b9592e3-41e5-4ada-9e42-ea705c3eb1f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mjxmq" Oct 31 00:40:54.311615 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:54.311592 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbtc5\" (UniqueName: \"kubernetes.io/projected/0b9592e3-41e5-4ada-9e42-ea705c3eb1f3-kube-api-access-qbtc5\") pod \"maas-default-gateway-openshift-default-6886b96f9-mjxmq\" (UID: \"0b9592e3-41e5-4ada-9e42-ea705c3eb1f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mjxmq" Oct 31 00:40:54.322677 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:54.322650 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mjxmq" Oct 31 00:40:54.504352 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:54.504245 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-r57tc" Oct 31 00:40:54.753606 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:40:54.753576 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b9592e3_41e5_4ada_9e42_ea705c3eb1f3.slice/crio-9af5e3d2c7c8f2cacab9eba6dada9ae783638bb7da85a56184293ec0d6c30ecb WatchSource:0}: Error finding container 9af5e3d2c7c8f2cacab9eba6dada9ae783638bb7da85a56184293ec0d6c30ecb: Status 404 returned error can't find the container with id 9af5e3d2c7c8f2cacab9eba6dada9ae783638bb7da85a56184293ec0d6c30ecb Oct 31 00:40:54.756535 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:54.756489 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:40:54.756639 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:54.756578 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:40:54.756639 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:54.756623 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:40:54.760251 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:54.760212 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mjxmq"] Oct 31 00:40:54.896497 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:40:54.896460 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33d0dda2_34fd_409e_9165_5861f2373db9.slice/crio-2de1c8c1d775aa9edb0cba202523dfd8957aafd806c342e499ffe222d108f748 WatchSource:0}: Error finding container 2de1c8c1d775aa9edb0cba202523dfd8957aafd806c342e499ffe222d108f748: Status 404 returned error can't find the container with id 2de1c8c1d775aa9edb0cba202523dfd8957aafd806c342e499ffe222d108f748 Oct 31 00:40:54.900066 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:54.900020 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:40:54.900195 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:54.900119 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:40:54.900195 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:54.900162 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:40:54.902105 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:54.902083 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-r57tc"] Oct 31 00:40:55.066810 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:55.066768 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-r57tc" event={"ID":"33d0dda2-34fd-409e-9165-5861f2373db9","Type":"ContainerStarted","Data":"daba08f2af2f279c5bc195222337caad8b9790336f58e37cdfd843d3b391ca69"} Oct 31 00:40:55.067023 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:55.066838 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-r57tc" event={"ID":"33d0dda2-34fd-409e-9165-5861f2373db9","Type":"ContainerStarted","Data":"2de1c8c1d775aa9edb0cba202523dfd8957aafd806c342e499ffe222d108f748"} Oct 31 00:40:55.069151 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:55.069123 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mjxmq" event={"ID":"0b9592e3-41e5-4ada-9e42-ea705c3eb1f3","Type":"ContainerStarted","Data":"a1e2c5bf2768aaf6e70d1488262ba6fc79c1d52e426ab722ba9126e949dcd73d"} Oct 31 00:40:55.069290 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:55.069176 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mjxmq" event={"ID":"0b9592e3-41e5-4ada-9e42-ea705c3eb1f3","Type":"ContainerStarted","Data":"9af5e3d2c7c8f2cacab9eba6dada9ae783638bb7da85a56184293ec0d6c30ecb"} Oct 31 00:40:55.195662 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:55.195604 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-r57tc" podStartSLOduration=2.195588526 podStartE2EDuration="2.195588526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:40:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:40:55.191856156 +0000 UTC m=+1547.663025907" watchObservedRunningTime="2026-10-31 00:40:55.195588526 +0000 UTC m=+1547.666758274" Oct 31 00:40:55.323749 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:55.323667 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mjxmq" Oct 31 00:40:55.325532 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:55.325502 5222 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-mjxmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Oct 31 00:40:55.325662 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:55.325564 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mjxmq" podUID="0b9592e3-41e5-4ada-9e42-ea705c3eb1f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Oct 31 00:40:55.374813 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:55.374754 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mjxmq" podStartSLOduration=2.374733889 podStartE2EDuration="2.374733889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:40:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:40:55.368672032 +0000 UTC m=+1547.839841783" watchObservedRunningTime="2026-10-31 00:40:55.374733889 +0000 UTC m=+1547.845903633" Oct 31 00:40:55.505391 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:55.505341 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-r57tc" Oct 31 00:40:55.509781 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:55.508149 5222 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-r57tc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Oct 31 00:40:55.509781 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:55.508228 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-r57tc" podUID="33d0dda2-34fd-409e-9165-5861f2373db9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Oct 31 00:40:56.107466 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:56.107428 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ddabde0d-46kf9"] Oct 31 00:40:56.111784 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:56.111755 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ddabde0d-46kf9"] Oct 31 00:40:56.323125 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:56.323085 5222 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-mjxmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Oct 31 00:40:56.323309 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:56.323166 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mjxmq" podUID="0b9592e3-41e5-4ada-9e42-ea705c3eb1f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Oct 31 00:40:56.504961 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:56.504873 5222 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-r57tc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Oct 31 00:40:56.504961 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:56.504934 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-r57tc" podUID="33d0dda2-34fd-409e-9165-5861f2373db9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Oct 31 00:40:57.323748 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:57.323705 5222 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-mjxmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Oct 31 00:40:57.324175 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:57.323788 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mjxmq" podUID="0b9592e3-41e5-4ada-9e42-ea705c3eb1f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Oct 31 00:40:57.504834 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:57.504683 5222 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-r57tc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Oct 31 00:40:57.504834 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:57.504747 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-r57tc" podUID="33d0dda2-34fd-409e-9165-5861f2373db9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Oct 31 00:40:57.696576 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:57.696491 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a346f5b5-f696-43cd-a709-be781e0a5d18" path="/var/lib/kubelet/pods/a346f5b5-f696-43cd-a709-be781e0a5d18/volumes" Oct 31 00:40:58.323424 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:58.323384 5222 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-mjxmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Oct 31 00:40:58.323609 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:58.323480 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mjxmq" podUID="0b9592e3-41e5-4ada-9e42-ea705c3eb1f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Oct 31 00:40:58.504816 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:58.504784 5222 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-r57tc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Oct 31 00:40:58.505220 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:58.504836 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-r57tc" podUID="33d0dda2-34fd-409e-9165-5861f2373db9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Oct 31 00:40:59.339396 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:59.333803 5222 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-mjxmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Oct 31 00:40:59.339396 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:59.333877 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mjxmq" podUID="0b9592e3-41e5-4ada-9e42-ea705c3eb1f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Oct 31 00:40:59.509038 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:59.505245 5222 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-r57tc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Oct 31 00:40:59.509038 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:40:59.505302 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-r57tc" podUID="33d0dda2-34fd-409e-9165-5861f2373db9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Oct 31 00:41:00.323118 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:00.323063 5222 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-mjxmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Oct 31 00:41:00.323306 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:00.323123 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mjxmq" podUID="0b9592e3-41e5-4ada-9e42-ea705c3eb1f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Oct 31 00:41:00.505157 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:00.505125 5222 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-r57tc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Oct 31 00:41:00.505347 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:00.505187 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-r57tc" podUID="33d0dda2-34fd-409e-9165-5861f2373db9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Oct 31 00:41:01.323246 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:01.323206 5222 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-mjxmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Oct 31 00:41:01.323663 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:01.323260 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mjxmq" podUID="0b9592e3-41e5-4ada-9e42-ea705c3eb1f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Oct 31 00:41:01.505718 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:01.505681 5222 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-r57tc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Oct 31 00:41:01.505923 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:01.505740 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-r57tc" podUID="33d0dda2-34fd-409e-9165-5861f2373db9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Oct 31 00:41:02.323460 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:02.323420 5222 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-mjxmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Oct 31 00:41:02.323897 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:02.323473 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mjxmq" podUID="0b9592e3-41e5-4ada-9e42-ea705c3eb1f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Oct 31 00:41:02.505600 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:02.505554 5222 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-r57tc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Oct 31 00:41:02.505812 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:02.505645 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-r57tc" podUID="33d0dda2-34fd-409e-9165-5861f2373db9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Oct 31 00:41:03.835568 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:03.835536 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mjxmq" Oct 31 00:41:03.836541 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:03.836514 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mjxmq" Oct 31 00:41:03.837659 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:03.837638 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mjxmq" Oct 31 00:41:04.247108 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:04.247012 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-r57tc" Oct 31 00:41:04.247486 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:04.247465 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-r57tc" Oct 31 00:41:04.248282 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:04.248265 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-r57tc" Oct 31 00:41:05.891195 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:41:05.891145 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 00:41:08.406906 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:08.406871 5222 scope.go:122] "RemoveContainer" containerID="cbb96a653344fea37fcb6d94c557504565a05b542eb2794fa546cd7ccf7a7cb2" Oct 31 00:41:08.419844 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:08.419784 5222 scope.go:122] "RemoveContainer" containerID="1b39813a2154ebf791ea20527256467f4c439e06f3cf38a44e2696c9238c2bc2" Oct 31 00:41:12.054698 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:12.054662 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f7d1a52-j7nfm"] Oct 31 00:41:12.105499 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:12.105456 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f7d1a52-j7nfm"] Oct 31 00:41:13.657604 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:13.657574 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c5fe715-e790-4c03-8cb9-7c3c726e75d0" path="/var/lib/kubelet/pods/3c5fe715-e790-4c03-8cb9-7c3c726e75d0/volumes" Oct 31 00:41:28.061943 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:28.061906 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66356b86-fbsl2"] Oct 31 00:41:28.112388 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:28.112325 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ea3a02b-6lg28"] Oct 31 00:41:28.116849 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:28.116812 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ea3a02b-6lg28"] Oct 31 00:41:28.120918 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:28.120892 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66356b86-fbsl2"] Oct 31 00:41:29.663388 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:29.663335 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="472f67dd-2129-4a33-8054-4d3bdb164f9c" path="/var/lib/kubelet/pods/472f67dd-2129-4a33-8054-4d3bdb164f9c/volumes" Oct 31 00:41:29.663896 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:29.663879 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66c3d766-2417-4ab1-87ae-4d678dfa3f7f" path="/var/lib/kubelet/pods/66c3d766-2417-4ab1-87ae-4d678dfa3f7f/volumes" Oct 31 00:41:30.666554 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:30.665716 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-4b86e4-6c67bcf5f-8tb44"] Oct 31 00:41:30.686752 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:30.686721 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-4b86e4-6c67bcf5f-8tb44" Oct 31 00:41:30.748409 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:30.742942 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-4b86e4-6c67bcf5f-8tb44"] Oct 31 00:41:30.805677 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:30.805650 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-4b86e4-dockercfg-r5bgj\"" Oct 31 00:41:30.806135 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:30.806116 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-4b86e4\"" Oct 31 00:41:30.873469 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:30.809729 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c94d94ac-1fb6-45d2-a286-5a0d8542ad55-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-4b86e4-6c67bcf5f-8tb44\" (UID: \"c94d94ac-1fb6-45d2-a286-5a0d8542ad55\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-4b86e4-6c67bcf5f-8tb44" Oct 31 00:41:30.911923 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:30.911889 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bb6mk\" (UniqueName: \"kubernetes.io/projected/c94d94ac-1fb6-45d2-a286-5a0d8542ad55-kube-api-access-bb6mk\") pod \"payload-pre-processing-e2e-shared-a-w4-4b86e4-6c67bcf5f-8tb44\" (UID: \"c94d94ac-1fb6-45d2-a286-5a0d8542ad55\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-4b86e4-6c67bcf5f-8tb44" Oct 31 00:41:31.021140 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:31.021076 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c94d94ac-1fb6-45d2-a286-5a0d8542ad55-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-4b86e4-6c67bcf5f-8tb44\" (UID: \"c94d94ac-1fb6-45d2-a286-5a0d8542ad55\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-4b86e4-6c67bcf5f-8tb44" Oct 31 00:41:31.022793 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:31.022766 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bb6mk\" (UniqueName: \"kubernetes.io/projected/c94d94ac-1fb6-45d2-a286-5a0d8542ad55-kube-api-access-bb6mk\") pod \"payload-pre-processing-e2e-shared-a-w4-4b86e4-6c67bcf5f-8tb44\" (UID: \"c94d94ac-1fb6-45d2-a286-5a0d8542ad55\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-4b86e4-6c67bcf5f-8tb44" Oct 31 00:41:31.025063 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:31.022658 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c94d94ac-1fb6-45d2-a286-5a0d8542ad55-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-4b86e4-6c67bcf5f-8tb44\" (UID: \"c94d94ac-1fb6-45d2-a286-5a0d8542ad55\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-4b86e4-6c67bcf5f-8tb44" Oct 31 00:41:31.027463 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:31.027433 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-4b86e4-5f8755b487-kp5qc"] Oct 31 00:41:31.040905 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:31.040874 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-4b86e4-5f8755b487-kp5qc" Oct 31 00:41:31.052096 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:31.052028 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-4b86e4-5f8755b487-kp5qc"] Oct 31 00:41:31.105651 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:31.105575 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bb6mk\" (UniqueName: \"kubernetes.io/projected/c94d94ac-1fb6-45d2-a286-5a0d8542ad55-kube-api-access-bb6mk\") pod \"payload-pre-processing-e2e-shared-a-w4-4b86e4-6c67bcf5f-8tb44\" (UID: \"c94d94ac-1fb6-45d2-a286-5a0d8542ad55\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-4b86e4-6c67bcf5f-8tb44" Oct 31 00:41:31.229083 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:31.229033 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b269e17a-5cf4-4d8a-9668-2587522d25da-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-4b86e4-5f8755b487-kp5qc\" (UID: \"b269e17a-5cf4-4d8a-9668-2587522d25da\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-4b86e4-5f8755b487-kp5qc" Oct 31 00:41:31.229272 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:31.229236 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdj8p\" (UniqueName: \"kubernetes.io/projected/b269e17a-5cf4-4d8a-9668-2587522d25da-kube-api-access-bdj8p\") pod \"payload-processing-e2e-shared-a-w4-4b86e4-5f8755b487-kp5qc\" (UID: \"b269e17a-5cf4-4d8a-9668-2587522d25da\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-4b86e4-5f8755b487-kp5qc" Oct 31 00:41:31.330608 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:31.330520 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b269e17a-5cf4-4d8a-9668-2587522d25da-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-4b86e4-5f8755b487-kp5qc\" (UID: \"b269e17a-5cf4-4d8a-9668-2587522d25da\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-4b86e4-5f8755b487-kp5qc" Oct 31 00:41:31.330773 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:31.330634 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdj8p\" (UniqueName: \"kubernetes.io/projected/b269e17a-5cf4-4d8a-9668-2587522d25da-kube-api-access-bdj8p\") pod \"payload-processing-e2e-shared-a-w4-4b86e4-5f8755b487-kp5qc\" (UID: \"b269e17a-5cf4-4d8a-9668-2587522d25da\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-4b86e4-5f8755b487-kp5qc" Oct 31 00:41:31.331260 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:31.331240 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b269e17a-5cf4-4d8a-9668-2587522d25da-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-4b86e4-5f8755b487-kp5qc\" (UID: \"b269e17a-5cf4-4d8a-9668-2587522d25da\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-4b86e4-5f8755b487-kp5qc" Oct 31 00:41:31.363002 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:31.362962 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-4b86e4-6c67bcf5f-8tb44" Oct 31 00:41:31.394126 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:31.394097 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdj8p\" (UniqueName: \"kubernetes.io/projected/b269e17a-5cf4-4d8a-9668-2587522d25da-kube-api-access-bdj8p\") pod \"payload-processing-e2e-shared-a-w4-4b86e4-5f8755b487-kp5qc\" (UID: \"b269e17a-5cf4-4d8a-9668-2587522d25da\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-4b86e4-5f8755b487-kp5qc" Oct 31 00:41:31.441426 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:31.432287 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-4b86e4-5f8755b487-kp5qc" Oct 31 00:41:31.749759 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:41:31.749727 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc94d94ac_1fb6_45d2_a286_5a0d8542ad55.slice/crio-e6965452b1c0c81c8f7e808f9624f30594e9d7784dcfb8a53c86bca91dece724 WatchSource:0}: Error finding container e6965452b1c0c81c8f7e808f9624f30594e9d7784dcfb8a53c86bca91dece724: Status 404 returned error can't find the container with id e6965452b1c0c81c8f7e808f9624f30594e9d7784dcfb8a53c86bca91dece724 Oct 31 00:41:31.754068 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:31.754041 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-4b86e4-6c67bcf5f-8tb44"] Oct 31 00:41:31.842885 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:41:31.842848 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb269e17a_5cf4_4d8a_9668_2587522d25da.slice/crio-5e8654d2b8ba5adebed2102ec387f6b08e247c0dde94607bbd6ba5e6d5575711 WatchSource:0}: Error finding container 5e8654d2b8ba5adebed2102ec387f6b08e247c0dde94607bbd6ba5e6d5575711: Status 404 returned error can't find the container with id 5e8654d2b8ba5adebed2102ec387f6b08e247c0dde94607bbd6ba5e6d5575711 Oct 31 00:41:31.847352 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:31.847326 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-4b86e4-5f8755b487-kp5qc"] Oct 31 00:41:32.632461 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:32.632412 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-4b86e4-5f8755b487-kp5qc" event={"ID":"b269e17a-5cf4-4d8a-9668-2587522d25da","Type":"ContainerStarted","Data":"e03dbf0baafa9097030f3ac9822f5878a21be45385643db121b15dbc138f3882"} Oct 31 00:41:32.632661 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:32.632484 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-4b86e4-5f8755b487-kp5qc" event={"ID":"b269e17a-5cf4-4d8a-9668-2587522d25da","Type":"ContainerStarted","Data":"5e8654d2b8ba5adebed2102ec387f6b08e247c0dde94607bbd6ba5e6d5575711"} Oct 31 00:41:32.632661 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:32.632509 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-4b86e4-5f8755b487-kp5qc" Oct 31 00:41:32.634234 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:32.634209 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-4b86e4-6c67bcf5f-8tb44" event={"ID":"c94d94ac-1fb6-45d2-a286-5a0d8542ad55","Type":"ContainerStarted","Data":"bc5e088a3c680693be801212e9d5c2b3fdcc00beafdef804bb044789d839ac20"} Oct 31 00:41:32.634382 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:32.634249 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-4b86e4-6c67bcf5f-8tb44" event={"ID":"c94d94ac-1fb6-45d2-a286-5a0d8542ad55","Type":"ContainerStarted","Data":"e6965452b1c0c81c8f7e808f9624f30594e9d7784dcfb8a53c86bca91dece724"} Oct 31 00:41:32.634382 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:32.634296 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-4b86e4-6c67bcf5f-8tb44" Oct 31 00:41:32.896821 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:32.891256 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-4b86e4-5f8755b487-kp5qc" podStartSLOduration=2.891234238 podStartE2EDuration="2.891234238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:41:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:41:32.76405868 +0000 UTC m=+1585.235228429" watchObservedRunningTime="2026-10-31 00:41:32.891234238 +0000 UTC m=+1585.362403988" Oct 31 00:41:32.913181 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:32.907567 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-4b86e4-6c67bcf5f-8tb44" podStartSLOduration=2.907543726 podStartE2EDuration="2.907543726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:41:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:41:32.892731472 +0000 UTC m=+1585.363901231" watchObservedRunningTime="2026-10-31 00:41:32.907543726 +0000 UTC m=+1585.378713476" Oct 31 00:41:35.891470 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:41:35.891429 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 00:41:37.184172 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:37.184126 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-79tk8"] Oct 31 00:41:40.309396 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:40.308783 5222 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nvtsz" podUID="c357cb32-7e30-442c-b612-f46e4a7dbc5a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": context deadline exceeded" Oct 31 00:41:40.347276 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:40.347230 5222 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-101b9e/premium-w0-101b9e-kserve-dc65f694b-m4tsk" podUID="fbf98c81-6f05-41b8-ba3b-c10ecbbff5d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 31 00:41:40.411647 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:40.411568 5222 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-1e5e75/premium-w3-1e5e75-kserve-7865bd6955-55dbx" podUID="8f72e496-0ec8-44f4-a295-4d985d1afb92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:41:40.412020 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:40.411991 5222 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6f4a5/simulator-w6-a6f4a5-kserve-7769689d48-bpkc6" podUID="24079bcb-658a-4109-9b14-a167a38cfac7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:41:40.443591 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:40.443526 5222 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1e5e75/distinct-w3-1e5e75-kserve-7b55d58bbb-hchmb" podUID="79ffb21b-10c3-4174-89be-05ba9e852a98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:41:40.486545 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:40.486489 5222 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-412743/distinct-2-w5-412743-kserve-5b679b6b66-59prz" podUID="2f6cbca5-d586-4bc2-9114-7212e9cc8a01" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:41:40.531392 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:40.530496 5222 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1e5e75/premium-w3-1e5e75-kserve-7865bd6955-55dbx" podUID="8f72e496-0ec8-44f4-a295-4d985d1afb92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 31 00:41:40.560527 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:40.559504 5222 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-412743/distinct-2-w5-412743-kserve-5b679b6b66-59prz" podUID="2f6cbca5-d586-4bc2-9114-7212e9cc8a01" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:41:40.605601 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:40.605547 5222 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da9d07/simulator-w4-da9d07-kserve-7c7588b894-snqq4" podUID="3d37ba9a-a648-44b2-aa04-4acf2b503a08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 31 00:41:40.741396 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:40.737547 5222 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-412743/simulator-w5-412743-kserve-6f897dd967-5bkq5" podUID="68d7fd60-bcad-45f7-84c0-11424e26f26d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Oct 31 00:41:40.794630 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:40.794541 5222 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-412743/distinct-w5-412743-kserve-64d65f7d8f-c4tfs" podUID="d0e8d5c5-aa08-41f6-badf-792648a20a68" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:41:40.918455 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:40.914714 5222 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-982ae3/premium-w1-982ae3-kserve-5c8755f445-qm9bh" podUID="005a2416-8e4f-4a72-8f1a-169a25e0f18c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:41:40.971393 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:40.970496 5222 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-101b9e/distinct-2-w0-101b9e-kserve-74548f6d6-v5phc" podUID="b8049531-bcf6-4f19-84d2-b974b9236223" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:41:41.277548 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:41.277492 5222 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1e5e75/simulator-w3-1e5e75-kserve-6cd984c46b-tjz8p" podUID="a6158bb1-fa2f-4d00-8159-d1620958c7c0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:41:41.320619 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:41.320559 5222 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nvtsz" podUID="c357cb32-7e30-442c-b612-f46e4a7dbc5a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": context deadline exceeded" Oct 31 00:41:41.349397 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:41.321399 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nvtsz" Oct 31 00:41:41.353568 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:41.353523 5222 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-101b9e/premium-w0-101b9e-kserve-dc65f694b-m4tsk" podUID="fbf98c81-6f05-41b8-ba3b-c10ecbbff5d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:41:41.377059 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:41.359955 5222 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-101b9e/distinct-w0-101b9e-kserve-6dcc4cc94f-kwv9w" podUID="73487e30-1bc6-49b2-b07b-1b004137232e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:41:41.385062 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:41.360019 5222 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-412743/simulator-w5-412743-kserve-6f897dd967-5bkq5" podUID="68d7fd60-bcad-45f7-84c0-11424e26f26d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:41:41.385062 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:41.383515 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-101b9e/premium-w0-101b9e-kserve-dc65f694b-m4tsk" Oct 31 00:41:41.425008 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:41.424959 5222 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6f4a5/simulator-w6-a6f4a5-kserve-7769689d48-bpkc6" podUID="24079bcb-658a-4109-9b14-a167a38cfac7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:41:41.426786 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:41.426751 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a6f4a5/simulator-w6-a6f4a5-kserve-7769689d48-bpkc6" Oct 31 00:41:41.458525 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:41.458478 5222 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1e5e75/distinct-w3-1e5e75-kserve-7b55d58bbb-hchmb" podUID="79ffb21b-10c3-4174-89be-05ba9e852a98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Oct 31 00:41:41.459210 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:41.459183 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-1e5e75/distinct-w3-1e5e75-kserve-7b55d58bbb-hchmb" Oct 31 00:41:41.460916 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:41.460881 5222 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nvtsz" podUID="c357cb32-7e30-442c-b612-f46e4a7dbc5a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:41:41.609773 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:41.609685 5222 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da9d07/simulator-w4-da9d07-kserve-7c7588b894-snqq4" podUID="3d37ba9a-a648-44b2-aa04-4acf2b503a08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:41:41.866275 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:41.612461 5222 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-412743/distinct-2-w5-412743-kserve-5b679b6b66-59prz" podUID="2f6cbca5-d586-4bc2-9114-7212e9cc8a01" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": context deadline exceeded" Oct 31 00:41:41.866275 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:41.675656 5222 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1e5e75/premium-w3-1e5e75-kserve-7865bd6955-55dbx" podUID="8f72e496-0ec8-44f4-a295-4d985d1afb92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:41:41.866275 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:41.737911 5222 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-412743/simulator-w5-412743-kserve-6f897dd967-5bkq5" podUID="68d7fd60-bcad-45f7-84c0-11424e26f26d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:41:41.866275 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:41.796484 5222 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-412743/distinct-w5-412743-kserve-64d65f7d8f-c4tfs" podUID="d0e8d5c5-aa08-41f6-badf-792648a20a68" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:41:41.960768 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:41.960330 5222 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-982ae3/premium-w1-982ae3-kserve-5c8755f445-qm9bh" podUID="005a2416-8e4f-4a72-8f1a-169a25e0f18c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:41:41.997796 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:41.997750 5222 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-101b9e/distinct-2-w0-101b9e-kserve-74548f6d6-v5phc" podUID="b8049531-bcf6-4f19-84d2-b974b9236223" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:41:42.157450 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:42.157130 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-412743/distinct-2-w5-412743-kserve-5b679b6b66-59prz" Oct 31 00:41:42.157450 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:42.157204 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-1e5e75/premium-w3-1e5e75-kserve-7865bd6955-55dbx" Oct 31 00:41:42.157450 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:42.157262 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-412743/simulator-w5-412743-kserve-6f897dd967-5bkq5" Oct 31 00:41:42.157450 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:42.157296 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-412743/distinct-w5-412743-kserve-64d65f7d8f-c4tfs" Oct 31 00:41:42.157450 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:42.157332 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-982ae3/premium-w1-982ae3-kserve-5c8755f445-qm9bh" Oct 31 00:41:42.157450 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:42.157386 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-101b9e/distinct-2-w0-101b9e-kserve-74548f6d6-v5phc" Oct 31 00:41:42.157450 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:42.157418 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-da9d07/simulator-w4-da9d07-kserve-7c7588b894-snqq4" Oct 31 00:41:42.284394 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:42.284198 5222 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1e5e75/simulator-w3-1e5e75-kserve-6cd984c46b-tjz8p" podUID="a6158bb1-fa2f-4d00-8159-d1620958c7c0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Oct 31 00:41:42.305833 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:42.304805 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-1e5e75/simulator-w3-1e5e75-kserve-6cd984c46b-tjz8p" Oct 31 00:41:42.327585 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:42.327520 5222 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nvtsz" podUID="c357cb32-7e30-442c-b612-f46e4a7dbc5a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:41:42.332412 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:42.331922 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a6f4a5/simulator-w6-a6f4a5-kserve-7769689d48-bpkc6" Oct 31 00:41:42.384423 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:42.377456 5222 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-101b9e/premium-w0-101b9e-kserve-dc65f694b-m4tsk" podUID="fbf98c81-6f05-41b8-ba3b-c10ecbbff5d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 31 00:41:42.389189 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:42.386876 5222 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-101b9e/distinct-w0-101b9e-kserve-6dcc4cc94f-kwv9w" podUID="73487e30-1bc6-49b2-b07b-1b004137232e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded" Oct 31 00:41:42.389189 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:42.388225 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-101b9e/distinct-w0-101b9e-kserve-6dcc4cc94f-kwv9w" Oct 31 00:41:42.416548 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:42.415150 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-1e5e75/simulator-w3-1e5e75-kserve-6cd984c46b-tjz8p" Oct 31 00:41:42.460613 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:42.460564 5222 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1e5e75/distinct-w3-1e5e75-kserve-7b55d58bbb-hchmb" podUID="79ffb21b-10c3-4174-89be-05ba9e852a98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:41:42.514504 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:42.514399 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-1e5e75/distinct-w3-1e5e75-kserve-7b55d58bbb-hchmb" Oct 31 00:41:42.534522 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:42.534349 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-101b9e/distinct-w0-101b9e-kserve-6dcc4cc94f-kwv9w" Oct 31 00:41:42.571456 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:42.556680 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-412743/distinct-2-w5-412743-kserve-5b679b6b66-59prz" Oct 31 00:41:42.589394 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:42.588849 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-101b9e/distinct-2-w0-101b9e-kserve-74548f6d6-v5phc" Oct 31 00:41:42.615282 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:42.615255 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-412743/distinct-w5-412743-kserve-64d65f7d8f-c4tfs" Oct 31 00:41:42.622763 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:42.622696 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-1e5e75/premium-w3-1e5e75-kserve-7865bd6955-55dbx" Oct 31 00:41:42.636646 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:42.636623 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nvtsz" Oct 31 00:41:42.683906 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:42.682917 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-da9d07/simulator-w4-da9d07-kserve-7c7588b894-snqq4" Oct 31 00:41:42.683906 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:42.683773 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-412743/simulator-w5-412743-kserve-6f897dd967-5bkq5" Oct 31 00:41:42.692152 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:42.692127 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-982ae3/premium-w1-982ae3-kserve-5c8755f445-qm9bh" Oct 31 00:41:42.696550 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:42.695702 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-101b9e/premium-w0-101b9e-kserve-dc65f694b-m4tsk" Oct 31 00:41:43.689438 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:43.689407 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-4b86e4-5f8755b487-kp5qc" Oct 31 00:41:43.690062 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:43.690045 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-4b86e4-6c67bcf5f-8tb44" Oct 31 00:41:45.079496 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:45.079461 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-730b8259-d7mpx"] Oct 31 00:41:45.086273 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:45.086246 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ad093e0-nfqcb"] Oct 31 00:41:45.090746 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:45.090715 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-730b8259-d7mpx"] Oct 31 00:41:45.094661 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:45.094639 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ad093e0-nfqcb"] Oct 31 00:41:45.659913 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:45.659870 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c366443-f988-4827-9d18-b985810050da" path="/var/lib/kubelet/pods/5c366443-f988-4827-9d18-b985810050da/volumes" Oct 31 00:41:45.660660 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:45.660640 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63148eb3-f1b9-49a9-be6f-13abac0aa8e9" path="/var/lib/kubelet/pods/63148eb3-f1b9-49a9-be6f-13abac0aa8e9/volumes" Oct 31 00:41:50.052890 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:50.052855 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-056c3b16-dbshj"] Oct 31 00:41:50.103916 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:50.103884 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-056c3b16-dbshj"] Oct 31 00:41:50.249731 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:50.249700 5222 generic.go:356] "Generic (PLEG): container finished" podID="d7525d83-432f-4c20-aa47-6d51a4c88030" containerID="8832255eadbeeddacb5f8ede93066f4fe2e30ad169a899e6f6541120691329b8" exitCode=0 Oct 31 00:41:50.249918 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:50.249784 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-dd862feb-openshift-default-67b7c76895-zrjbk" event={"ID":"d7525d83-432f-4c20-aa47-6d51a4c88030","Type":"ContainerDied","Data":"8832255eadbeeddacb5f8ede93066f4fe2e30ad169a899e6f6541120691329b8"} Oct 31 00:41:50.250291 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:50.250263 5222 scope.go:122] "RemoveContainer" containerID="8832255eadbeeddacb5f8ede93066f4fe2e30ad169a899e6f6541120691329b8" Oct 31 00:41:50.251938 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:50.251904 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:41:50.252061 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:50.251994 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:41:50.252061 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:50.252043 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:41:51.060676 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:51.060639 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6bf713ef-4p9bn"] Oct 31 00:41:51.065010 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:51.064984 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6bf713ef-4p9bn"] Oct 31 00:41:51.271159 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:51.271129 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-dd862feb-openshift-default-67b7c76895-zrjbk" event={"ID":"d7525d83-432f-4c20-aa47-6d51a4c88030","Type":"ContainerStarted","Data":"854ea1ef31b2f4fe88b1e02dd03bafd291079cb58f4de37c726f029fe3b143fa"} Oct 31 00:41:51.462675 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:51.462568 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-dd862feb-openshift-default-67b7c76895-zrjbk" Oct 31 00:41:51.462905 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:51.462873 5222 patch_prober.go:29] interesting pod/e2e-ait-dd862feb-openshift-default-67b7c76895-zrjbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Oct 31 00:41:51.463055 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:51.462950 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-dd862feb-openshift-default-67b7c76895-zrjbk" podUID="d7525d83-432f-4c20-aa47-6d51a4c88030" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Oct 31 00:41:51.669671 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:51.669640 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ca0cfaa-8fd4-473b-9d14-776c25ffdb05" path="/var/lib/kubelet/pods/4ca0cfaa-8fd4-473b-9d14-776c25ffdb05/volumes" Oct 31 00:41:51.670141 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:51.670127 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90e2a297-20cc-42a5-bf04-5df44c6bfecb" path="/var/lib/kubelet/pods/90e2a297-20cc-42a5-bf04-5df44c6bfecb/volumes" Oct 31 00:41:52.460862 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:52.460823 5222 patch_prober.go:29] interesting pod/e2e-ait-dd862feb-openshift-default-67b7c76895-zrjbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Oct 31 00:41:52.461507 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:52.461468 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-dd862feb-openshift-default-67b7c76895-zrjbk" podUID="d7525d83-432f-4c20-aa47-6d51a4c88030" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Oct 31 00:41:53.461395 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:53.460831 5222 patch_prober.go:29] interesting pod/e2e-ait-dd862feb-openshift-default-67b7c76895-zrjbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Oct 31 00:41:53.461395 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:53.460896 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-dd862feb-openshift-default-67b7c76895-zrjbk" podUID="d7525d83-432f-4c20-aa47-6d51a4c88030" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Oct 31 00:41:54.462399 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:54.461993 5222 patch_prober.go:29] interesting pod/e2e-ait-dd862feb-openshift-default-67b7c76895-zrjbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Oct 31 00:41:54.462399 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:54.462061 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-dd862feb-openshift-default-67b7c76895-zrjbk" podUID="d7525d83-432f-4c20-aa47-6d51a4c88030" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Oct 31 00:41:55.462911 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:55.462870 5222 patch_prober.go:29] interesting pod/e2e-ait-dd862feb-openshift-default-67b7c76895-zrjbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Oct 31 00:41:55.463340 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:55.462950 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-dd862feb-openshift-default-67b7c76895-zrjbk" podUID="d7525d83-432f-4c20-aa47-6d51a4c88030" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Oct 31 00:41:56.462400 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:56.461950 5222 patch_prober.go:29] interesting pod/e2e-ait-dd862feb-openshift-default-67b7c76895-zrjbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Oct 31 00:41:56.462400 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:56.462023 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-dd862feb-openshift-default-67b7c76895-zrjbk" podUID="d7525d83-432f-4c20-aa47-6d51a4c88030" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Oct 31 00:41:57.465069 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:57.465035 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-dd862feb-openshift-default-67b7c76895-zrjbk" Oct 31 00:41:57.465518 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:57.465498 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-dd862feb-openshift-default-67b7c76895-zrjbk" Oct 31 00:41:57.466234 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:57.466212 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-dd862feb-openshift-default-67b7c76895-zrjbk" Oct 31 00:41:59.910608 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:59.910575 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-56xs9"] Oct 31 00:41:59.911187 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:41:59.910820 5222 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-56xs9" podUID="15886257-5d46-4575-acb2-68a0da38b954" containerName="istio-proxy" containerID="cri-o://4a0a3ba14c7811082b0ca15f89859b3b5e0b541589b793cbd5a1dc8f628e61e6" gracePeriod=30 Oct 31 00:42:01.058619 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:01.058586 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b182c63-kgc4k"] Oct 31 00:42:01.109402 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:01.109341 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b182c63-kgc4k"] Oct 31 00:42:01.657675 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:01.657639 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e22b4bd7-7950-40e6-88c5-5bfba6415954" path="/var/lib/kubelet/pods/e22b4bd7-7950-40e6-88c5-5bfba6415954/volumes" Oct 31 00:42:02.055032 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:02.054983 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f65ac1f3-wtqvf"] Oct 31 00:42:02.078043 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:02.077998 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f65ac1f3-wtqvf"] Oct 31 00:42:03.657182 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:03.657139 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="701064e2-35dc-4b2d-9901-84c186af8f47" path="/var/lib/kubelet/pods/701064e2-35dc-4b2d-9901-84c186af8f47/volumes" Oct 31 00:42:05.196470 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:05.196438 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-56xs9" Oct 31 00:42:05.228650 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:05.228604 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15886257-5d46-4575-acb2-68a0da38b954-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15886257-5d46-4575-acb2-68a0da38b954-istio-envoy\") pod \"15886257-5d46-4575-acb2-68a0da38b954\" (UID: \"15886257-5d46-4575-acb2-68a0da38b954\") " Oct 31 00:42:05.228826 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:05.228729 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15886257-5d46-4575-acb2-68a0da38b954-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15886257-5d46-4575-acb2-68a0da38b954-istiod-ca-cert\") pod \"15886257-5d46-4575-acb2-68a0da38b954\" (UID: \"15886257-5d46-4575-acb2-68a0da38b954\") " Oct 31 00:42:05.228826 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:05.228799 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15886257-5d46-4575-acb2-68a0da38b954-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15886257-5d46-4575-acb2-68a0da38b954-istio-data\") pod \"15886257-5d46-4575-acb2-68a0da38b954\" (UID: \"15886257-5d46-4575-acb2-68a0da38b954\") " Oct 31 00:42:05.228926 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:05.228905 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15886257-5d46-4575-acb2-68a0da38b954-istio-token\" (UniqueName: \"kubernetes.io/projected/15886257-5d46-4575-acb2-68a0da38b954-istio-token\") pod \"15886257-5d46-4575-acb2-68a0da38b954\" (UID: \"15886257-5d46-4575-acb2-68a0da38b954\") " Oct 31 00:42:05.228962 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:05.228955 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/15886257-5d46-4575-acb2-68a0da38b954-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15886257-5d46-4575-acb2-68a0da38b954-istio-podinfo\") pod \"15886257-5d46-4575-acb2-68a0da38b954\" (UID: \"15886257-5d46-4575-acb2-68a0da38b954\") " Oct 31 00:42:05.229029 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:05.229013 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15886257-5d46-4575-acb2-68a0da38b954-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15886257-5d46-4575-acb2-68a0da38b954-workload-certs\") pod \"15886257-5d46-4575-acb2-68a0da38b954\" (UID: \"15886257-5d46-4575-acb2-68a0da38b954\") " Oct 31 00:42:05.229089 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:05.229067 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15886257-5d46-4575-acb2-68a0da38b954-istio-data" pod "15886257-5d46-4575-acb2-68a0da38b954" (UID: "15886257-5d46-4575-acb2-68a0da38b954"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:42:05.229145 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:05.229091 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15886257-5d46-4575-acb2-68a0da38b954-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15886257-5d46-4575-acb2-68a0da38b954-credential-socket\") pod \"15886257-5d46-4575-acb2-68a0da38b954\" (UID: \"15886257-5d46-4575-acb2-68a0da38b954\") " Oct 31 00:42:05.229145 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:05.229123 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15886257-5d46-4575-acb2-68a0da38b954-kube-api-access-8zmpp\" (UniqueName: \"kubernetes.io/projected/15886257-5d46-4575-acb2-68a0da38b954-kube-api-access-8zmpp\") pod \"15886257-5d46-4575-acb2-68a0da38b954\" (UID: \"15886257-5d46-4575-acb2-68a0da38b954\") " Oct 31 00:42:05.229255 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:05.229135 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15886257-5d46-4575-acb2-68a0da38b954-istiod-ca-cert" pod "15886257-5d46-4575-acb2-68a0da38b954" (UID: "15886257-5d46-4575-acb2-68a0da38b954"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:42:05.229255 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:05.229192 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15886257-5d46-4575-acb2-68a0da38b954-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15886257-5d46-4575-acb2-68a0da38b954-workload-socket\") pod \"15886257-5d46-4575-acb2-68a0da38b954\" (UID: \"15886257-5d46-4575-acb2-68a0da38b954\") " Oct 31 00:42:05.229415 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:05.229352 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15886257-5d46-4575-acb2-68a0da38b954-workload-certs" pod "15886257-5d46-4575-acb2-68a0da38b954" (UID: "15886257-5d46-4575-acb2-68a0da38b954"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:42:05.229415 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:05.229409 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15886257-5d46-4575-acb2-68a0da38b954-credential-socket" pod "15886257-5d46-4575-acb2-68a0da38b954" (UID: "15886257-5d46-4575-acb2-68a0da38b954"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:42:05.229682 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:05.229552 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15886257-5d46-4575-acb2-68a0da38b954-workload-socket" pod "15886257-5d46-4575-acb2-68a0da38b954" (UID: "15886257-5d46-4575-acb2-68a0da38b954"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:42:05.230206 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:05.230180 5222 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15886257-5d46-4575-acb2-68a0da38b954-credential-socket\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:05.230316 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:05.230211 5222 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15886257-5d46-4575-acb2-68a0da38b954-workload-socket\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:05.230316 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:05.230226 5222 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15886257-5d46-4575-acb2-68a0da38b954-istiod-ca-cert\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:05.230316 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:05.230245 5222 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15886257-5d46-4575-acb2-68a0da38b954-istio-data\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:05.230316 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:05.230259 5222 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15886257-5d46-4575-acb2-68a0da38b954-workload-certs\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:05.231164 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:05.231138 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15886257-5d46-4575-acb2-68a0da38b954-istio-envoy" pod "15886257-5d46-4575-acb2-68a0da38b954" (UID: "15886257-5d46-4575-acb2-68a0da38b954"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:42:05.231398 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:05.231360 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/15886257-5d46-4575-acb2-68a0da38b954-istio-podinfo" pod "15886257-5d46-4575-acb2-68a0da38b954" (UID: "15886257-5d46-4575-acb2-68a0da38b954"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:42:05.231496 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:05.231477 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15886257-5d46-4575-acb2-68a0da38b954-kube-api-access-8zmpp" pod "15886257-5d46-4575-acb2-68a0da38b954" (UID: "15886257-5d46-4575-acb2-68a0da38b954"). InnerVolumeSpecName "kube-api-access-8zmpp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:42:05.231561 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:05.231507 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15886257-5d46-4575-acb2-68a0da38b954-istio-token" pod "15886257-5d46-4575-acb2-68a0da38b954" (UID: "15886257-5d46-4575-acb2-68a0da38b954"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:42:05.331859 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:05.331822 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zmpp\" (UniqueName: \"kubernetes.io/projected/15886257-5d46-4575-acb2-68a0da38b954-kube-api-access-8zmpp\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:05.331859 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:05.331855 5222 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15886257-5d46-4575-acb2-68a0da38b954-istio-envoy\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:05.331859 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:05.331866 5222 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15886257-5d46-4575-acb2-68a0da38b954-istio-token\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:05.332111 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:05.331875 5222 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15886257-5d46-4575-acb2-68a0da38b954-istio-podinfo\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:05.400953 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:05.400912 5222 generic.go:356] "Generic (PLEG): container finished" podID="15886257-5d46-4575-acb2-68a0da38b954" containerID="4a0a3ba14c7811082b0ca15f89859b3b5e0b541589b793cbd5a1dc8f628e61e6" exitCode=0 Oct 31 00:42:05.401165 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:05.400959 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-56xs9" event={"ID":"15886257-5d46-4575-acb2-68a0da38b954","Type":"ContainerDied","Data":"4a0a3ba14c7811082b0ca15f89859b3b5e0b541589b793cbd5a1dc8f628e61e6"} Oct 31 00:42:05.401165 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:05.400983 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-56xs9" Oct 31 00:42:05.401165 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:05.401056 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-56xs9" event={"ID":"15886257-5d46-4575-acb2-68a0da38b954","Type":"ContainerDied","Data":"a0a5bb7813d105fda295597b13b79440180adf34a3b290955955151fae6396a7"} Oct 31 00:42:05.401165 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:05.401089 5222 scope.go:122] "RemoveContainer" containerID="4a0a3ba14c7811082b0ca15f89859b3b5e0b541589b793cbd5a1dc8f628e61e6" Oct 31 00:42:05.415032 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:05.415002 5222 scope.go:122] "RemoveContainer" containerID="4a0a3ba14c7811082b0ca15f89859b3b5e0b541589b793cbd5a1dc8f628e61e6" Oct 31 00:42:05.415542 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:42:05.415508 5222 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a0a3ba14c7811082b0ca15f89859b3b5e0b541589b793cbd5a1dc8f628e61e6\": container with ID starting with 4a0a3ba14c7811082b0ca15f89859b3b5e0b541589b793cbd5a1dc8f628e61e6 not found: ID does not exist" containerID="4a0a3ba14c7811082b0ca15f89859b3b5e0b541589b793cbd5a1dc8f628e61e6" Oct 31 00:42:05.415633 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:05.415556 5222 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a0a3ba14c7811082b0ca15f89859b3b5e0b541589b793cbd5a1dc8f628e61e6"} err="failed to get container status \"4a0a3ba14c7811082b0ca15f89859b3b5e0b541589b793cbd5a1dc8f628e61e6\": rpc error: code = NotFound desc = could not find container \"4a0a3ba14c7811082b0ca15f89859b3b5e0b541589b793cbd5a1dc8f628e61e6\": container with ID starting with 4a0a3ba14c7811082b0ca15f89859b3b5e0b541589b793cbd5a1dc8f628e61e6 not found: ID does not exist" Oct 31 00:42:05.575105 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:05.575076 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-56xs9"] Oct 31 00:42:05.579351 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:05.579325 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-56xs9"] Oct 31 00:42:05.658774 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:05.658743 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15886257-5d46-4575-acb2-68a0da38b954" path="/var/lib/kubelet/pods/15886257-5d46-4575-acb2-68a0da38b954/volumes" Oct 31 00:42:05.891206 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:42:05.891119 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 00:42:06.061298 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:06.061262 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef147e9e-tzz8g"] Oct 31 00:42:06.065405 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:06.065358 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef147e9e-tzz8g"] Oct 31 00:42:07.658932 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:07.658903 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a7a7936-2833-4bf8-80d6-dbb269954605" path="/var/lib/kubelet/pods/2a7a7936-2833-4bf8-80d6-dbb269954605/volumes" Oct 31 00:42:08.475501 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:08.475467 5222 scope.go:122] "RemoveContainer" containerID="2bbfba490a9fb4970d0d1601e91fe18a109870bf73ae03bc1cfd8a2ff6fbea22" Oct 31 00:42:08.497209 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:08.497185 5222 scope.go:122] "RemoveContainer" containerID="976764b4462b3bc02b6bc9df61bdecf06f095371451da245985fee27d13351bc" Oct 31 00:42:08.507457 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:08.507433 5222 scope.go:122] "RemoveContainer" containerID="9f08237735c945c768fa68436434d5ffb4da83b3e33022ab018dc1c205e0cef4" Oct 31 00:42:11.091649 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:11.091604 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c09e604-cxspn"] Oct 31 00:42:11.126279 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:11.126248 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c09e604-cxspn"] Oct 31 00:42:11.657346 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:11.657315 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96e468eb-c292-4249-b748-b36cb86a4538" path="/var/lib/kubelet/pods/96e468eb-c292-4249-b748-b36cb86a4538/volumes" Oct 31 00:42:15.053886 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:15.053809 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2fe61f62-xkstq"] Oct 31 00:42:15.104892 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:15.104859 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2fe61f62-xkstq"] Oct 31 00:42:15.658327 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:15.658282 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e08c9285-911f-4816-a7e4-b0c0f48269ac" path="/var/lib/kubelet/pods/e08c9285-911f-4816-a7e4-b0c0f48269ac/volumes" Oct 31 00:42:18.053217 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:18.053175 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dcba2d2b-ddk75"] Oct 31 00:42:18.104237 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:18.104194 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dcba2d2b-ddk75"] Oct 31 00:42:18.593812 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:18.593777 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dd862feb-jc9cw"] Oct 31 00:42:18.594554 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:18.594533 5222 cpu_manager.go:414] "containerMap: removing container" podUID="15886257-5d46-4575-acb2-68a0da38b954" containerName="istio-proxy" Oct 31 00:42:18.594554 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:18.594553 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15886257-5d46-4575-acb2-68a0da38b954" containerName="istio-proxy" Oct 31 00:42:18.594779 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:18.594765 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="15886257-5d46-4575-acb2-68a0da38b954" containerName="istio-proxy" Oct 31 00:42:18.600351 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:18.600328 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dd862feb-jc9cw" Oct 31 00:42:18.654174 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:18.654135 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dd862feb-jc9cw"] Oct 31 00:42:18.654504 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:18.654484 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:42:18.669144 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:18.669106 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsqtc\" (UniqueName: \"kubernetes.io/projected/84752b79-f569-4777-af91-fd3604122ba9-kube-api-access-nsqtc\") pod \"maas-api-revoke-keys-e2e-ait-dd862feb-jc9cw\" (UID: \"84752b79-f569-4777-af91-fd3604122ba9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dd862feb-jc9cw" Oct 31 00:42:18.669336 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:18.669218 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84752b79-f569-4777-af91-fd3604122ba9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-dd862feb-jc9cw\" (UID: \"84752b79-f569-4777-af91-fd3604122ba9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dd862feb-jc9cw" Oct 31 00:42:18.669440 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:18.669384 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84752b79-f569-4777-af91-fd3604122ba9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-dd862feb-jc9cw\" (UID: \"84752b79-f569-4777-af91-fd3604122ba9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dd862feb-jc9cw" Oct 31 00:42:18.698641 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:18.698613 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfz9z\"" Oct 31 00:42:18.698641 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:18.698612 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:42:18.698641 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:18.698612 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:42:18.771123 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:18.771087 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84752b79-f569-4777-af91-fd3604122ba9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-dd862feb-jc9cw\" (UID: \"84752b79-f569-4777-af91-fd3604122ba9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dd862feb-jc9cw" Oct 31 00:42:18.771324 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:18.771171 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84752b79-f569-4777-af91-fd3604122ba9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-dd862feb-jc9cw\" (UID: \"84752b79-f569-4777-af91-fd3604122ba9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dd862feb-jc9cw" Oct 31 00:42:18.771395 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:18.771357 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsqtc\" (UniqueName: \"kubernetes.io/projected/84752b79-f569-4777-af91-fd3604122ba9-kube-api-access-nsqtc\") pod \"maas-api-revoke-keys-e2e-ait-dd862feb-jc9cw\" (UID: \"84752b79-f569-4777-af91-fd3604122ba9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dd862feb-jc9cw" Oct 31 00:42:18.771912 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:18.771884 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84752b79-f569-4777-af91-fd3604122ba9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-dd862feb-jc9cw\" (UID: \"84752b79-f569-4777-af91-fd3604122ba9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dd862feb-jc9cw" Oct 31 00:42:18.773683 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:18.773661 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84752b79-f569-4777-af91-fd3604122ba9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-dd862feb-jc9cw\" (UID: \"84752b79-f569-4777-af91-fd3604122ba9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dd862feb-jc9cw" Oct 31 00:42:18.834491 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:18.834456 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsqtc\" (UniqueName: \"kubernetes.io/projected/84752b79-f569-4777-af91-fd3604122ba9-kube-api-access-nsqtc\") pod \"maas-api-revoke-keys-e2e-ait-dd862feb-jc9cw\" (UID: \"84752b79-f569-4777-af91-fd3604122ba9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dd862feb-jc9cw" Oct 31 00:42:18.914970 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:18.914882 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dd862feb-jc9cw" Oct 31 00:42:19.306809 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:19.306766 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dd862feb-jc9cw"] Oct 31 00:42:19.313840 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:42:19.313808 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84752b79_f569_4777_af91_fd3604122ba9.slice/crio-1be807f344053f6668215e8041ac7540895f1102f051ce16b1f3be25049f255d WatchSource:0}: Error finding container 1be807f344053f6668215e8041ac7540895f1102f051ce16b1f3be25049f255d: Status 404 returned error can't find the container with id 1be807f344053f6668215e8041ac7540895f1102f051ce16b1f3be25049f255d Oct 31 00:42:19.519846 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:19.519814 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dd862feb-jc9cw" event={"ID":"84752b79-f569-4777-af91-fd3604122ba9","Type":"ContainerStarted","Data":"1be807f344053f6668215e8041ac7540895f1102f051ce16b1f3be25049f255d"} Oct 31 00:42:19.659936 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:19.659896 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5189d24-61c9-4669-a66c-4856d33e3584" path="/var/lib/kubelet/pods/c5189d24-61c9-4669-a66c-4856d33e3584/volumes" Oct 31 00:42:20.531605 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:20.531566 5222 generic.go:356] "Generic (PLEG): container finished" podID="84752b79-f569-4777-af91-fd3604122ba9" containerID="aff2ed7c4ce564a2e82fe8b353a3dda60a100554b03193279a855b854c85675a" exitCode=0 Oct 31 00:42:20.532077 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:20.531652 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dd862feb-jc9cw" event={"ID":"84752b79-f569-4777-af91-fd3604122ba9","Type":"ContainerDied","Data":"aff2ed7c4ce564a2e82fe8b353a3dda60a100554b03193279a855b854c85675a"} Oct 31 00:42:21.719687 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:21.719657 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dd862feb-jc9cw" Oct 31 00:42:21.836065 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:21.835984 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84752b79-f569-4777-af91-fd3604122ba9-kube-api-access-nsqtc\" (UniqueName: \"kubernetes.io/projected/84752b79-f569-4777-af91-fd3604122ba9-kube-api-access-nsqtc\") pod \"84752b79-f569-4777-af91-fd3604122ba9\" (UID: \"84752b79-f569-4777-af91-fd3604122ba9\") " Oct 31 00:42:21.836244 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:21.836116 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84752b79-f569-4777-af91-fd3604122ba9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84752b79-f569-4777-af91-fd3604122ba9-maas-api-openshift-ca\") pod \"84752b79-f569-4777-af91-fd3604122ba9\" (UID: \"84752b79-f569-4777-af91-fd3604122ba9\") " Oct 31 00:42:21.836309 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:21.836238 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84752b79-f569-4777-af91-fd3604122ba9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84752b79-f569-4777-af91-fd3604122ba9-maas-api-xks-ca\") pod \"84752b79-f569-4777-af91-fd3604122ba9\" (UID: \"84752b79-f569-4777-af91-fd3604122ba9\") " Oct 31 00:42:21.836695 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:21.836673 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84752b79-f569-4777-af91-fd3604122ba9-maas-api-openshift-ca" pod "84752b79-f569-4777-af91-fd3604122ba9" (UID: "84752b79-f569-4777-af91-fd3604122ba9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:42:21.836862 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:21.836842 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84752b79-f569-4777-af91-fd3604122ba9-maas-api-openshift-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:21.838661 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:21.838630 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84752b79-f569-4777-af91-fd3604122ba9-kube-api-access-nsqtc" pod "84752b79-f569-4777-af91-fd3604122ba9" (UID: "84752b79-f569-4777-af91-fd3604122ba9"). InnerVolumeSpecName "kube-api-access-nsqtc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:42:21.838960 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:21.838940 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84752b79-f569-4777-af91-fd3604122ba9-maas-api-xks-ca" pod "84752b79-f569-4777-af91-fd3604122ba9" (UID: "84752b79-f569-4777-af91-fd3604122ba9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:42:21.937886 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:21.937840 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nsqtc\" (UniqueName: \"kubernetes.io/projected/84752b79-f569-4777-af91-fd3604122ba9-kube-api-access-nsqtc\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:21.937886 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:21.937877 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84752b79-f569-4777-af91-fd3604122ba9-maas-api-xks-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:22.555771 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:22.555739 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dd862feb-jc9cw" Oct 31 00:42:22.555771 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:22.555767 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dd862feb-jc9cw" event={"ID":"84752b79-f569-4777-af91-fd3604122ba9","Type":"ContainerDied","Data":"1be807f344053f6668215e8041ac7540895f1102f051ce16b1f3be25049f255d"} Oct 31 00:42:22.556030 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:22.555816 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1be807f344053f6668215e8041ac7540895f1102f051ce16b1f3be25049f255d" Oct 31 00:42:23.615590 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:23.615553 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-79tk8"] Oct 31 00:42:24.157878 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:24.156732 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18e8cedf-txqlt"] Oct 31 00:42:24.176342 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:24.176291 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18e8cedf-txqlt"] Oct 31 00:42:25.657254 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:25.657220 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47f9711a-f60e-4fec-accd-a065ab4eb48a" path="/var/lib/kubelet/pods/47f9711a-f60e-4fec-accd-a065ab4eb48a/volumes" Oct 31 00:42:26.055313 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:26.055276 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55867ce2-z8zk8"] Oct 31 00:42:26.106156 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:26.106123 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55867ce2-z8zk8"] Oct 31 00:42:27.062065 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:27.062031 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a92e31b-bnxkp"] Oct 31 00:42:27.113242 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:27.113201 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a92e31b-bnxkp"] Oct 31 00:42:27.659652 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:27.659621 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="852ea7c9-0231-499f-9fb2-1a3256ad77c3" path="/var/lib/kubelet/pods/852ea7c9-0231-499f-9fb2-1a3256ad77c3/volumes" Oct 31 00:42:27.660114 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:27.660100 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8a5550a-5d20-4ec9-8272-dc539a00e96d" path="/var/lib/kubelet/pods/e8a5550a-5d20-4ec9-8272-dc539a00e96d/volumes" Oct 31 00:42:28.073955 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:28.073913 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-d58cd3-openshift-default-d9c46776f-vklx7"] Oct 31 00:42:28.074882 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:28.074857 5222 cpu_manager.go:414] "containerMap: removing container" podUID="84752b79-f569-4777-af91-fd3604122ba9" containerName="revoke-keys" Oct 31 00:42:28.075023 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:28.074884 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84752b79-f569-4777-af91-fd3604122ba9" containerName="revoke-keys" Oct 31 00:42:28.075108 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:28.075095 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="84752b79-f569-4777-af91-fd3604122ba9" containerName="revoke-keys" Oct 31 00:42:28.080248 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:28.080223 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-d58cd3-openshift-default-d9c46776f-vklx7" Oct 31 00:42:28.140036 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:28.140000 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-d58cd3-openshift-default-d9c46776f-vklx7"] Oct 31 00:42:28.150922 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:28.150890 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/608e314b-554b-40a2-90f7-fbdefd2b27b6-istio-token\") pod \"e2e-shared-a-w2-d58cd3-openshift-default-d9c46776f-vklx7\" (UID: \"608e314b-554b-40a2-90f7-fbdefd2b27b6\") " pod="openshift-ingress/e2e-shared-a-w2-d58cd3-openshift-default-d9c46776f-vklx7" Oct 31 00:42:28.151119 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:28.150944 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/608e314b-554b-40a2-90f7-fbdefd2b27b6-istiod-ca-cert\") pod \"e2e-shared-a-w2-d58cd3-openshift-default-d9c46776f-vklx7\" (UID: \"608e314b-554b-40a2-90f7-fbdefd2b27b6\") " pod="openshift-ingress/e2e-shared-a-w2-d58cd3-openshift-default-d9c46776f-vklx7" Oct 31 00:42:28.151119 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:28.151055 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwhns\" (UniqueName: \"kubernetes.io/projected/608e314b-554b-40a2-90f7-fbdefd2b27b6-kube-api-access-nwhns\") pod \"e2e-shared-a-w2-d58cd3-openshift-default-d9c46776f-vklx7\" (UID: \"608e314b-554b-40a2-90f7-fbdefd2b27b6\") " pod="openshift-ingress/e2e-shared-a-w2-d58cd3-openshift-default-d9c46776f-vklx7" Oct 31 00:42:28.151119 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:28.151111 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/608e314b-554b-40a2-90f7-fbdefd2b27b6-istio-podinfo\") pod \"e2e-shared-a-w2-d58cd3-openshift-default-d9c46776f-vklx7\" (UID: \"608e314b-554b-40a2-90f7-fbdefd2b27b6\") " pod="openshift-ingress/e2e-shared-a-w2-d58cd3-openshift-default-d9c46776f-vklx7" Oct 31 00:42:28.151303 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:28.151216 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/608e314b-554b-40a2-90f7-fbdefd2b27b6-workload-certs\") pod \"e2e-shared-a-w2-d58cd3-openshift-default-d9c46776f-vklx7\" (UID: \"608e314b-554b-40a2-90f7-fbdefd2b27b6\") " pod="openshift-ingress/e2e-shared-a-w2-d58cd3-openshift-default-d9c46776f-vklx7" Oct 31 00:42:28.151303 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:28.151284 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/608e314b-554b-40a2-90f7-fbdefd2b27b6-workload-socket\") pod \"e2e-shared-a-w2-d58cd3-openshift-default-d9c46776f-vklx7\" (UID: \"608e314b-554b-40a2-90f7-fbdefd2b27b6\") " pod="openshift-ingress/e2e-shared-a-w2-d58cd3-openshift-default-d9c46776f-vklx7" Oct 31 00:42:28.151449 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:28.151429 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/608e314b-554b-40a2-90f7-fbdefd2b27b6-istio-data\") pod \"e2e-shared-a-w2-d58cd3-openshift-default-d9c46776f-vklx7\" (UID: \"608e314b-554b-40a2-90f7-fbdefd2b27b6\") " pod="openshift-ingress/e2e-shared-a-w2-d58cd3-openshift-default-d9c46776f-vklx7" Oct 31 00:42:28.151506 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:28.151473 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/608e314b-554b-40a2-90f7-fbdefd2b27b6-istio-envoy\") pod \"e2e-shared-a-w2-d58cd3-openshift-default-d9c46776f-vklx7\" (UID: \"608e314b-554b-40a2-90f7-fbdefd2b27b6\") " pod="openshift-ingress/e2e-shared-a-w2-d58cd3-openshift-default-d9c46776f-vklx7" Oct 31 00:42:28.151562 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:28.151552 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/608e314b-554b-40a2-90f7-fbdefd2b27b6-credential-socket\") pod \"e2e-shared-a-w2-d58cd3-openshift-default-d9c46776f-vklx7\" (UID: \"608e314b-554b-40a2-90f7-fbdefd2b27b6\") " pod="openshift-ingress/e2e-shared-a-w2-d58cd3-openshift-default-d9c46776f-vklx7" Oct 31 00:42:28.172336 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:28.172292 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-d58cd3-openshift-default-dockercfg-djh9q\"" Oct 31 00:42:28.253192 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:28.253155 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/608e314b-554b-40a2-90f7-fbdefd2b27b6-istio-token\") pod \"e2e-shared-a-w2-d58cd3-openshift-default-d9c46776f-vklx7\" (UID: \"608e314b-554b-40a2-90f7-fbdefd2b27b6\") " pod="openshift-ingress/e2e-shared-a-w2-d58cd3-openshift-default-d9c46776f-vklx7" Oct 31 00:42:28.253422 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:28.253210 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/608e314b-554b-40a2-90f7-fbdefd2b27b6-istiod-ca-cert\") pod \"e2e-shared-a-w2-d58cd3-openshift-default-d9c46776f-vklx7\" (UID: \"608e314b-554b-40a2-90f7-fbdefd2b27b6\") " pod="openshift-ingress/e2e-shared-a-w2-d58cd3-openshift-default-d9c46776f-vklx7" Oct 31 00:42:28.253422 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:28.253253 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwhns\" (UniqueName: \"kubernetes.io/projected/608e314b-554b-40a2-90f7-fbdefd2b27b6-kube-api-access-nwhns\") pod \"e2e-shared-a-w2-d58cd3-openshift-default-d9c46776f-vklx7\" (UID: \"608e314b-554b-40a2-90f7-fbdefd2b27b6\") " pod="openshift-ingress/e2e-shared-a-w2-d58cd3-openshift-default-d9c46776f-vklx7" Oct 31 00:42:28.253422 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:28.253288 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/608e314b-554b-40a2-90f7-fbdefd2b27b6-istio-podinfo\") pod \"e2e-shared-a-w2-d58cd3-openshift-default-d9c46776f-vklx7\" (UID: \"608e314b-554b-40a2-90f7-fbdefd2b27b6\") " pod="openshift-ingress/e2e-shared-a-w2-d58cd3-openshift-default-d9c46776f-vklx7" Oct 31 00:42:28.253591 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:28.253433 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/608e314b-554b-40a2-90f7-fbdefd2b27b6-workload-certs\") pod \"e2e-shared-a-w2-d58cd3-openshift-default-d9c46776f-vklx7\" (UID: \"608e314b-554b-40a2-90f7-fbdefd2b27b6\") " pod="openshift-ingress/e2e-shared-a-w2-d58cd3-openshift-default-d9c46776f-vklx7" Oct 31 00:42:28.253591 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:28.253538 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/608e314b-554b-40a2-90f7-fbdefd2b27b6-workload-socket\") pod \"e2e-shared-a-w2-d58cd3-openshift-default-d9c46776f-vklx7\" (UID: \"608e314b-554b-40a2-90f7-fbdefd2b27b6\") " pod="openshift-ingress/e2e-shared-a-w2-d58cd3-openshift-default-d9c46776f-vklx7" Oct 31 00:42:28.253656 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:28.253596 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/608e314b-554b-40a2-90f7-fbdefd2b27b6-istio-data\") pod \"e2e-shared-a-w2-d58cd3-openshift-default-d9c46776f-vklx7\" (UID: \"608e314b-554b-40a2-90f7-fbdefd2b27b6\") " pod="openshift-ingress/e2e-shared-a-w2-d58cd3-openshift-default-d9c46776f-vklx7" Oct 31 00:42:28.253656 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:28.253634 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/608e314b-554b-40a2-90f7-fbdefd2b27b6-istio-envoy\") pod \"e2e-shared-a-w2-d58cd3-openshift-default-d9c46776f-vklx7\" (UID: \"608e314b-554b-40a2-90f7-fbdefd2b27b6\") " pod="openshift-ingress/e2e-shared-a-w2-d58cd3-openshift-default-d9c46776f-vklx7" Oct 31 00:42:28.253740 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:28.253683 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/608e314b-554b-40a2-90f7-fbdefd2b27b6-credential-socket\") pod \"e2e-shared-a-w2-d58cd3-openshift-default-d9c46776f-vklx7\" (UID: \"608e314b-554b-40a2-90f7-fbdefd2b27b6\") " pod="openshift-ingress/e2e-shared-a-w2-d58cd3-openshift-default-d9c46776f-vklx7" Oct 31 00:42:28.253926 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:28.253891 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/608e314b-554b-40a2-90f7-fbdefd2b27b6-workload-certs\") pod \"e2e-shared-a-w2-d58cd3-openshift-default-d9c46776f-vklx7\" (UID: \"608e314b-554b-40a2-90f7-fbdefd2b27b6\") " pod="openshift-ingress/e2e-shared-a-w2-d58cd3-openshift-default-d9c46776f-vklx7" Oct 31 00:42:28.254056 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:28.253952 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/608e314b-554b-40a2-90f7-fbdefd2b27b6-workload-socket\") pod \"e2e-shared-a-w2-d58cd3-openshift-default-d9c46776f-vklx7\" (UID: \"608e314b-554b-40a2-90f7-fbdefd2b27b6\") " pod="openshift-ingress/e2e-shared-a-w2-d58cd3-openshift-default-d9c46776f-vklx7" Oct 31 00:42:28.254056 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:28.253987 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/608e314b-554b-40a2-90f7-fbdefd2b27b6-istio-data\") pod \"e2e-shared-a-w2-d58cd3-openshift-default-d9c46776f-vklx7\" (UID: \"608e314b-554b-40a2-90f7-fbdefd2b27b6\") " pod="openshift-ingress/e2e-shared-a-w2-d58cd3-openshift-default-d9c46776f-vklx7" Oct 31 00:42:28.254186 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:28.254069 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/608e314b-554b-40a2-90f7-fbdefd2b27b6-credential-socket\") pod \"e2e-shared-a-w2-d58cd3-openshift-default-d9c46776f-vklx7\" (UID: \"608e314b-554b-40a2-90f7-fbdefd2b27b6\") " pod="openshift-ingress/e2e-shared-a-w2-d58cd3-openshift-default-d9c46776f-vklx7" Oct 31 00:42:28.254186 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:28.254115 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/608e314b-554b-40a2-90f7-fbdefd2b27b6-istiod-ca-cert\") pod \"e2e-shared-a-w2-d58cd3-openshift-default-d9c46776f-vklx7\" (UID: \"608e314b-554b-40a2-90f7-fbdefd2b27b6\") " pod="openshift-ingress/e2e-shared-a-w2-d58cd3-openshift-default-d9c46776f-vklx7" Oct 31 00:42:28.256171 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:28.256144 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/608e314b-554b-40a2-90f7-fbdefd2b27b6-istio-podinfo\") pod \"e2e-shared-a-w2-d58cd3-openshift-default-d9c46776f-vklx7\" (UID: \"608e314b-554b-40a2-90f7-fbdefd2b27b6\") " pod="openshift-ingress/e2e-shared-a-w2-d58cd3-openshift-default-d9c46776f-vklx7" Oct 31 00:42:28.256317 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:28.256257 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/608e314b-554b-40a2-90f7-fbdefd2b27b6-istio-envoy\") pod \"e2e-shared-a-w2-d58cd3-openshift-default-d9c46776f-vklx7\" (UID: \"608e314b-554b-40a2-90f7-fbdefd2b27b6\") " pod="openshift-ingress/e2e-shared-a-w2-d58cd3-openshift-default-d9c46776f-vklx7" Oct 31 00:42:28.359581 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:28.349228 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwhns\" (UniqueName: \"kubernetes.io/projected/608e314b-554b-40a2-90f7-fbdefd2b27b6-kube-api-access-nwhns\") pod \"e2e-shared-a-w2-d58cd3-openshift-default-d9c46776f-vklx7\" (UID: \"608e314b-554b-40a2-90f7-fbdefd2b27b6\") " pod="openshift-ingress/e2e-shared-a-w2-d58cd3-openshift-default-d9c46776f-vklx7" Oct 31 00:42:28.359581 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:28.356362 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/608e314b-554b-40a2-90f7-fbdefd2b27b6-istio-token\") pod \"e2e-shared-a-w2-d58cd3-openshift-default-d9c46776f-vklx7\" (UID: \"608e314b-554b-40a2-90f7-fbdefd2b27b6\") " pod="openshift-ingress/e2e-shared-a-w2-d58cd3-openshift-default-d9c46776f-vklx7" Oct 31 00:42:28.397692 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:28.397653 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-d58cd3-openshift-default-d9c46776f-vklx7" Oct 31 00:42:29.049882 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:42:29.049852 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod608e314b_554b_40a2_90f7_fbdefd2b27b6.slice/crio-cbc38540d09d92baa1f9a0da46b3893782114a87d23edf07f41a54ead8ce6e3b WatchSource:0}: Error finding container cbc38540d09d92baa1f9a0da46b3893782114a87d23edf07f41a54ead8ce6e3b: Status 404 returned error can't find the container with id cbc38540d09d92baa1f9a0da46b3893782114a87d23edf07f41a54ead8ce6e3b Oct 31 00:42:29.052851 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:29.052813 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:42:29.052954 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:29.052887 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:42:29.052954 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:29.052923 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:42:29.054173 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:29.054155 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-d58cd3-openshift-default-d9c46776f-vklx7"] Oct 31 00:42:29.106600 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:29.106564 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5558bdda-f54k5"] Oct 31 00:42:29.110575 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:29.110545 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5558bdda-f54k5"] Oct 31 00:42:29.643739 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:29.643705 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-d58cd3-openshift-default-d9c46776f-vklx7" event={"ID":"608e314b-554b-40a2-90f7-fbdefd2b27b6","Type":"ContainerStarted","Data":"ff6a9379b42cb98396f10d68f9835af0c5dbe25bc8a4554dd8bcfeb16eb7bbee"} Oct 31 00:42:29.643900 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:29.643765 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-d58cd3-openshift-default-d9c46776f-vklx7" event={"ID":"608e314b-554b-40a2-90f7-fbdefd2b27b6","Type":"ContainerStarted","Data":"cbc38540d09d92baa1f9a0da46b3893782114a87d23edf07f41a54ead8ce6e3b"} Oct 31 00:42:29.656731 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:29.656700 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba482f5f-fe5e-4204-b5ec-99ad3003acb5" path="/var/lib/kubelet/pods/ba482f5f-fe5e-4204-b5ec-99ad3003acb5/volumes" Oct 31 00:42:29.783695 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:29.783632 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-d58cd3-openshift-default-d9c46776f-vklx7" podStartSLOduration=1.783615766 podStartE2EDuration="1.783615766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:42:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:42:29.779666103 +0000 UTC m=+1642.250835850" watchObservedRunningTime="2026-10-31 00:42:29.783615766 +0000 UTC m=+1642.254785512" Oct 31 00:42:30.415036 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:30.399089 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-d58cd3-openshift-default-d9c46776f-vklx7" Oct 31 00:42:30.425611 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:30.425580 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-d58cd3-openshift-default-d9c46776f-vklx7" Oct 31 00:42:30.467394 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:30.467330 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-74c6ba99-2d8jd"] Oct 31 00:42:30.485423 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:30.484773 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-74c6ba99-2d8jd" Oct 31 00:42:30.522861 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:30.518989 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-74c6ba99-2d8jd"] Oct 31 00:42:30.539803 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:30.539778 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:42:30.539998 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:30.539944 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfz9z\"" Oct 31 00:42:30.540208 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:30.540191 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:42:30.540295 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:30.540210 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:42:30.604081 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:30.604047 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwp2s\" (UniqueName: \"kubernetes.io/projected/cd4cd9f5-591c-42ac-8a69-415d4f143f49-kube-api-access-mwp2s\") pod \"maas-api-revoke-keys-e2e-derive-74c6ba99-2d8jd\" (UID: \"cd4cd9f5-591c-42ac-8a69-415d4f143f49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-74c6ba99-2d8jd" Oct 31 00:42:30.604395 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:30.604290 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd4cd9f5-591c-42ac-8a69-415d4f143f49-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-74c6ba99-2d8jd\" (UID: \"cd4cd9f5-591c-42ac-8a69-415d4f143f49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-74c6ba99-2d8jd" Oct 31 00:42:30.604395 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:30.604329 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd4cd9f5-591c-42ac-8a69-415d4f143f49-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-74c6ba99-2d8jd\" (UID: \"cd4cd9f5-591c-42ac-8a69-415d4f143f49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-74c6ba99-2d8jd" Oct 31 00:42:30.651909 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:30.651879 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-d58cd3-openshift-default-d9c46776f-vklx7" Oct 31 00:42:30.653099 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:30.653075 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-d58cd3-openshift-default-d9c46776f-vklx7" Oct 31 00:42:30.706232 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:30.706143 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd4cd9f5-591c-42ac-8a69-415d4f143f49-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-74c6ba99-2d8jd\" (UID: \"cd4cd9f5-591c-42ac-8a69-415d4f143f49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-74c6ba99-2d8jd" Oct 31 00:42:30.706232 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:30.706189 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd4cd9f5-591c-42ac-8a69-415d4f143f49-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-74c6ba99-2d8jd\" (UID: \"cd4cd9f5-591c-42ac-8a69-415d4f143f49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-74c6ba99-2d8jd" Oct 31 00:42:30.706515 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:30.706257 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwp2s\" (UniqueName: \"kubernetes.io/projected/cd4cd9f5-591c-42ac-8a69-415d4f143f49-kube-api-access-mwp2s\") pod \"maas-api-revoke-keys-e2e-derive-74c6ba99-2d8jd\" (UID: \"cd4cd9f5-591c-42ac-8a69-415d4f143f49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-74c6ba99-2d8jd" Oct 31 00:42:30.706903 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:30.706879 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd4cd9f5-591c-42ac-8a69-415d4f143f49-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-74c6ba99-2d8jd\" (UID: \"cd4cd9f5-591c-42ac-8a69-415d4f143f49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-74c6ba99-2d8jd" Oct 31 00:42:30.709002 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:30.708975 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd4cd9f5-591c-42ac-8a69-415d4f143f49-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-74c6ba99-2d8jd\" (UID: \"cd4cd9f5-591c-42ac-8a69-415d4f143f49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-74c6ba99-2d8jd" Oct 31 00:42:30.769203 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:30.769174 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwp2s\" (UniqueName: \"kubernetes.io/projected/cd4cd9f5-591c-42ac-8a69-415d4f143f49-kube-api-access-mwp2s\") pod \"maas-api-revoke-keys-e2e-derive-74c6ba99-2d8jd\" (UID: \"cd4cd9f5-591c-42ac-8a69-415d4f143f49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-74c6ba99-2d8jd" Oct 31 00:42:30.830741 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:30.830708 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-74c6ba99-2d8jd" Oct 31 00:42:31.230850 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:42:31.230817 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd4cd9f5_591c_42ac_8a69_415d4f143f49.slice/crio-fb3fc438717acaa69cbae343e09ab49387908d01c22f8edf38c4fb5708f649c6 WatchSource:0}: Error finding container fb3fc438717acaa69cbae343e09ab49387908d01c22f8edf38c4fb5708f649c6: Status 404 returned error can't find the container with id fb3fc438717acaa69cbae343e09ab49387908d01c22f8edf38c4fb5708f649c6 Oct 31 00:42:31.234642 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:31.234621 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-74c6ba99-2d8jd"] Oct 31 00:42:31.662433 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:31.662398 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-74c6ba99-2d8jd" event={"ID":"cd4cd9f5-591c-42ac-8a69-415d4f143f49","Type":"ContainerStarted","Data":"fb3fc438717acaa69cbae343e09ab49387908d01c22f8edf38c4fb5708f649c6"} Oct 31 00:42:32.235984 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:32.235948 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-dd862feb-9c885f569-n957f"] Oct 31 00:42:32.362245 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:32.361597 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-dd862feb-6c5d597495-9ftp6"] Oct 31 00:42:32.872448 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:32.859122 5222 generic.go:356] "Generic (PLEG): container finished" podID="cd4cd9f5-591c-42ac-8a69-415d4f143f49" containerID="c370cc5ff634f8403704b66ecb4f320bcd409cf7fbc2b96d004f878b2ce452a4" exitCode=0 Oct 31 00:42:32.872448 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:32.860889 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-74c6ba99-2d8jd" event={"ID":"cd4cd9f5-591c-42ac-8a69-415d4f143f49","Type":"ContainerDied","Data":"c370cc5ff634f8403704b66ecb4f320bcd409cf7fbc2b96d004f878b2ce452a4"} Oct 31 00:42:34.068426 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:34.068389 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a81ead5-2ctzc"] Oct 31 00:42:34.119591 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:34.119561 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a81ead5-2ctzc"] Oct 31 00:42:34.120560 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:34.120540 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-74c6ba99-2d8jd" Oct 31 00:42:34.292397 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:34.290923 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd4cd9f5-591c-42ac-8a69-415d4f143f49-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd4cd9f5-591c-42ac-8a69-415d4f143f49-maas-api-openshift-ca\") pod \"cd4cd9f5-591c-42ac-8a69-415d4f143f49\" (UID: \"cd4cd9f5-591c-42ac-8a69-415d4f143f49\") " Oct 31 00:42:34.292397 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:34.291015 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd4cd9f5-591c-42ac-8a69-415d4f143f49-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd4cd9f5-591c-42ac-8a69-415d4f143f49-maas-api-xks-ca\") pod \"cd4cd9f5-591c-42ac-8a69-415d4f143f49\" (UID: \"cd4cd9f5-591c-42ac-8a69-415d4f143f49\") " Oct 31 00:42:34.292397 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:34.291051 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd4cd9f5-591c-42ac-8a69-415d4f143f49-kube-api-access-mwp2s\" (UniqueName: \"kubernetes.io/projected/cd4cd9f5-591c-42ac-8a69-415d4f143f49-kube-api-access-mwp2s\") pod \"cd4cd9f5-591c-42ac-8a69-415d4f143f49\" (UID: \"cd4cd9f5-591c-42ac-8a69-415d4f143f49\") " Oct 31 00:42:34.305138 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:34.304797 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd4cd9f5-591c-42ac-8a69-415d4f143f49-maas-api-openshift-ca" pod "cd4cd9f5-591c-42ac-8a69-415d4f143f49" (UID: "cd4cd9f5-591c-42ac-8a69-415d4f143f49"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:42:34.358611 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:34.354586 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd4cd9f5-591c-42ac-8a69-415d4f143f49-kube-api-access-mwp2s" pod "cd4cd9f5-591c-42ac-8a69-415d4f143f49" (UID: "cd4cd9f5-591c-42ac-8a69-415d4f143f49"). InnerVolumeSpecName "kube-api-access-mwp2s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:42:34.398480 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:34.397333 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd4cd9f5-591c-42ac-8a69-415d4f143f49-maas-api-openshift-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:34.398480 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:34.397384 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwp2s\" (UniqueName: \"kubernetes.io/projected/cd4cd9f5-591c-42ac-8a69-415d4f143f49-kube-api-access-mwp2s\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:34.445279 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:34.441633 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd4cd9f5-591c-42ac-8a69-415d4f143f49-maas-api-xks-ca" pod "cd4cd9f5-591c-42ac-8a69-415d4f143f49" (UID: "cd4cd9f5-591c-42ac-8a69-415d4f143f49"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:42:34.452932 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:34.452892 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-5ttl9"] Oct 31 00:42:34.454000 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:34.453971 5222 cpu_manager.go:414] "containerMap: removing container" podUID="cd4cd9f5-591c-42ac-8a69-415d4f143f49" containerName="revoke-keys" Oct 31 00:42:34.454000 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:34.454001 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd4cd9f5-591c-42ac-8a69-415d4f143f49" containerName="revoke-keys" Oct 31 00:42:34.454243 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:34.454225 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd4cd9f5-591c-42ac-8a69-415d4f143f49" containerName="revoke-keys" Oct 31 00:42:34.534134 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:34.523700 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-5ttl9" Oct 31 00:42:34.539786 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:34.539730 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1062428a-5ea4-4e47-a5a5-3515b7743f33-workload-socket\") pod \"e2e-derive-74c6ba99-openshift-default-65cd5b8f85-5ttl9\" (UID: \"1062428a-5ea4-4e47-a5a5-3515b7743f33\") " pod="openshift-ingress/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-5ttl9" Oct 31 00:42:34.540063 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:34.540030 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1062428a-5ea4-4e47-a5a5-3515b7743f33-credential-socket\") pod \"e2e-derive-74c6ba99-openshift-default-65cd5b8f85-5ttl9\" (UID: \"1062428a-5ea4-4e47-a5a5-3515b7743f33\") " pod="openshift-ingress/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-5ttl9" Oct 31 00:42:34.540197 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:34.540181 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1062428a-5ea4-4e47-a5a5-3515b7743f33-istio-data\") pod \"e2e-derive-74c6ba99-openshift-default-65cd5b8f85-5ttl9\" (UID: \"1062428a-5ea4-4e47-a5a5-3515b7743f33\") " pod="openshift-ingress/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-5ttl9" Oct 31 00:42:34.540502 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:34.540468 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1062428a-5ea4-4e47-a5a5-3515b7743f33-istio-podinfo\") pod \"e2e-derive-74c6ba99-openshift-default-65cd5b8f85-5ttl9\" (UID: \"1062428a-5ea4-4e47-a5a5-3515b7743f33\") " pod="openshift-ingress/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-5ttl9" Oct 31 00:42:34.540788 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:34.540752 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1062428a-5ea4-4e47-a5a5-3515b7743f33-istio-token\") pod \"e2e-derive-74c6ba99-openshift-default-65cd5b8f85-5ttl9\" (UID: \"1062428a-5ea4-4e47-a5a5-3515b7743f33\") " pod="openshift-ingress/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-5ttl9" Oct 31 00:42:34.540986 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:34.540946 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fw4b9\" (UniqueName: \"kubernetes.io/projected/1062428a-5ea4-4e47-a5a5-3515b7743f33-kube-api-access-fw4b9\") pod \"e2e-derive-74c6ba99-openshift-default-65cd5b8f85-5ttl9\" (UID: \"1062428a-5ea4-4e47-a5a5-3515b7743f33\") " pod="openshift-ingress/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-5ttl9" Oct 31 00:42:34.541230 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:34.541205 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1062428a-5ea4-4e47-a5a5-3515b7743f33-istio-envoy\") pod \"e2e-derive-74c6ba99-openshift-default-65cd5b8f85-5ttl9\" (UID: \"1062428a-5ea4-4e47-a5a5-3515b7743f33\") " pod="openshift-ingress/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-5ttl9" Oct 31 00:42:34.541457 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:34.541432 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1062428a-5ea4-4e47-a5a5-3515b7743f33-istiod-ca-cert\") pod \"e2e-derive-74c6ba99-openshift-default-65cd5b8f85-5ttl9\" (UID: \"1062428a-5ea4-4e47-a5a5-3515b7743f33\") " pod="openshift-ingress/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-5ttl9" Oct 31 00:42:34.543979 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:34.543898 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1062428a-5ea4-4e47-a5a5-3515b7743f33-workload-certs\") pod \"e2e-derive-74c6ba99-openshift-default-65cd5b8f85-5ttl9\" (UID: \"1062428a-5ea4-4e47-a5a5-3515b7743f33\") " pod="openshift-ingress/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-5ttl9" Oct 31 00:42:34.544320 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:34.544281 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd4cd9f5-591c-42ac-8a69-415d4f143f49-maas-api-xks-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:34.599998 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:34.577363 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-5ttl9"] Oct 31 00:42:34.620623 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:34.620589 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-74c6ba99-openshift-default-dockercfg-mnr9m\"" Oct 31 00:42:34.646329 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:34.646294 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1062428a-5ea4-4e47-a5a5-3515b7743f33-istio-envoy\") pod \"e2e-derive-74c6ba99-openshift-default-65cd5b8f85-5ttl9\" (UID: \"1062428a-5ea4-4e47-a5a5-3515b7743f33\") " pod="openshift-ingress/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-5ttl9" Oct 31 00:42:34.646527 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:34.646393 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1062428a-5ea4-4e47-a5a5-3515b7743f33-istiod-ca-cert\") pod \"e2e-derive-74c6ba99-openshift-default-65cd5b8f85-5ttl9\" (UID: \"1062428a-5ea4-4e47-a5a5-3515b7743f33\") " pod="openshift-ingress/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-5ttl9" Oct 31 00:42:34.646527 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:34.646437 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1062428a-5ea4-4e47-a5a5-3515b7743f33-workload-certs\") pod \"e2e-derive-74c6ba99-openshift-default-65cd5b8f85-5ttl9\" (UID: \"1062428a-5ea4-4e47-a5a5-3515b7743f33\") " pod="openshift-ingress/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-5ttl9" Oct 31 00:42:34.646527 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:34.646476 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1062428a-5ea4-4e47-a5a5-3515b7743f33-workload-socket\") pod \"e2e-derive-74c6ba99-openshift-default-65cd5b8f85-5ttl9\" (UID: \"1062428a-5ea4-4e47-a5a5-3515b7743f33\") " pod="openshift-ingress/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-5ttl9" Oct 31 00:42:34.646527 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:34.646499 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1062428a-5ea4-4e47-a5a5-3515b7743f33-credential-socket\") pod \"e2e-derive-74c6ba99-openshift-default-65cd5b8f85-5ttl9\" (UID: \"1062428a-5ea4-4e47-a5a5-3515b7743f33\") " pod="openshift-ingress/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-5ttl9" Oct 31 00:42:34.646760 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:34.646531 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1062428a-5ea4-4e47-a5a5-3515b7743f33-istio-data\") pod \"e2e-derive-74c6ba99-openshift-default-65cd5b8f85-5ttl9\" (UID: \"1062428a-5ea4-4e47-a5a5-3515b7743f33\") " pod="openshift-ingress/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-5ttl9" Oct 31 00:42:34.646760 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:34.646631 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1062428a-5ea4-4e47-a5a5-3515b7743f33-istio-podinfo\") pod \"e2e-derive-74c6ba99-openshift-default-65cd5b8f85-5ttl9\" (UID: \"1062428a-5ea4-4e47-a5a5-3515b7743f33\") " pod="openshift-ingress/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-5ttl9" Oct 31 00:42:34.646760 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:34.646728 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1062428a-5ea4-4e47-a5a5-3515b7743f33-istio-token\") pod \"e2e-derive-74c6ba99-openshift-default-65cd5b8f85-5ttl9\" (UID: \"1062428a-5ea4-4e47-a5a5-3515b7743f33\") " pod="openshift-ingress/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-5ttl9" Oct 31 00:42:34.646919 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:34.646780 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fw4b9\" (UniqueName: \"kubernetes.io/projected/1062428a-5ea4-4e47-a5a5-3515b7743f33-kube-api-access-fw4b9\") pod \"e2e-derive-74c6ba99-openshift-default-65cd5b8f85-5ttl9\" (UID: \"1062428a-5ea4-4e47-a5a5-3515b7743f33\") " pod="openshift-ingress/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-5ttl9" Oct 31 00:42:34.647394 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:34.647309 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1062428a-5ea4-4e47-a5a5-3515b7743f33-workload-certs\") pod \"e2e-derive-74c6ba99-openshift-default-65cd5b8f85-5ttl9\" (UID: \"1062428a-5ea4-4e47-a5a5-3515b7743f33\") " pod="openshift-ingress/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-5ttl9" Oct 31 00:42:34.647505 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:34.647480 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1062428a-5ea4-4e47-a5a5-3515b7743f33-workload-socket\") pod \"e2e-derive-74c6ba99-openshift-default-65cd5b8f85-5ttl9\" (UID: \"1062428a-5ea4-4e47-a5a5-3515b7743f33\") " pod="openshift-ingress/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-5ttl9" Oct 31 00:42:34.647581 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:34.647501 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1062428a-5ea4-4e47-a5a5-3515b7743f33-credential-socket\") pod \"e2e-derive-74c6ba99-openshift-default-65cd5b8f85-5ttl9\" (UID: \"1062428a-5ea4-4e47-a5a5-3515b7743f33\") " pod="openshift-ingress/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-5ttl9" Oct 31 00:42:34.647645 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:34.647590 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1062428a-5ea4-4e47-a5a5-3515b7743f33-istiod-ca-cert\") pod \"e2e-derive-74c6ba99-openshift-default-65cd5b8f85-5ttl9\" (UID: \"1062428a-5ea4-4e47-a5a5-3515b7743f33\") " pod="openshift-ingress/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-5ttl9" Oct 31 00:42:34.647709 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:34.647665 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1062428a-5ea4-4e47-a5a5-3515b7743f33-istio-data\") pod \"e2e-derive-74c6ba99-openshift-default-65cd5b8f85-5ttl9\" (UID: \"1062428a-5ea4-4e47-a5a5-3515b7743f33\") " pod="openshift-ingress/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-5ttl9" Oct 31 00:42:34.649241 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:34.649214 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1062428a-5ea4-4e47-a5a5-3515b7743f33-istio-envoy\") pod \"e2e-derive-74c6ba99-openshift-default-65cd5b8f85-5ttl9\" (UID: \"1062428a-5ea4-4e47-a5a5-3515b7743f33\") " pod="openshift-ingress/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-5ttl9" Oct 31 00:42:34.649893 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:34.649873 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1062428a-5ea4-4e47-a5a5-3515b7743f33-istio-podinfo\") pod \"e2e-derive-74c6ba99-openshift-default-65cd5b8f85-5ttl9\" (UID: \"1062428a-5ea4-4e47-a5a5-3515b7743f33\") " pod="openshift-ingress/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-5ttl9" Oct 31 00:42:34.767040 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:34.767002 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1062428a-5ea4-4e47-a5a5-3515b7743f33-istio-token\") pod \"e2e-derive-74c6ba99-openshift-default-65cd5b8f85-5ttl9\" (UID: \"1062428a-5ea4-4e47-a5a5-3515b7743f33\") " pod="openshift-ingress/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-5ttl9" Oct 31 00:42:34.783718 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:34.783070 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fw4b9\" (UniqueName: \"kubernetes.io/projected/1062428a-5ea4-4e47-a5a5-3515b7743f33-kube-api-access-fw4b9\") pod \"e2e-derive-74c6ba99-openshift-default-65cd5b8f85-5ttl9\" (UID: \"1062428a-5ea4-4e47-a5a5-3515b7743f33\") " pod="openshift-ingress/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-5ttl9" Oct 31 00:42:34.903729 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:34.903626 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-5ttl9" Oct 31 00:42:34.921043 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:34.921002 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-74c6ba99-2d8jd" Oct 31 00:42:34.921190 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:34.921010 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-74c6ba99-2d8jd" event={"ID":"cd4cd9f5-591c-42ac-8a69-415d4f143f49","Type":"ContainerDied","Data":"fb3fc438717acaa69cbae343e09ab49387908d01c22f8edf38c4fb5708f649c6"} Oct 31 00:42:34.921190 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:34.921134 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fb3fc438717acaa69cbae343e09ab49387908d01c22f8edf38c4fb5708f649c6" Oct 31 00:42:35.054035 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:35.054003 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-227efc5e-qg6ck"] Oct 31 00:42:35.057904 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:35.057879 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-227efc5e-qg6ck"] Oct 31 00:42:35.340625 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:42:35.340581 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1062428a_5ea4_4e47_a5a5_3515b7743f33.slice/crio-32768d9fc121967e1c99cc4970d86add40a8f39b6b43c1a4e38f2c8c58cbad0e WatchSource:0}: Error finding container 32768d9fc121967e1c99cc4970d86add40a8f39b6b43c1a4e38f2c8c58cbad0e: Status 404 returned error can't find the container with id 32768d9fc121967e1c99cc4970d86add40a8f39b6b43c1a4e38f2c8c58cbad0e Oct 31 00:42:35.343461 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:35.343433 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-5ttl9"] Oct 31 00:42:35.345404 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:35.345318 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:42:35.345561 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:35.345536 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:42:35.345816 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:35.345643 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:42:35.670793 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:35.670715 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bf0aca2-3542-4659-a5fc-eb4078a23488" path="/var/lib/kubelet/pods/2bf0aca2-3542-4659-a5fc-eb4078a23488/volumes" Oct 31 00:42:35.671592 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:35.671575 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cbba00e-45f7-492d-931d-3065700de2db" path="/var/lib/kubelet/pods/2cbba00e-45f7-492d-931d-3065700de2db/volumes" Oct 31 00:42:35.893627 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:42:35.893579 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 00:42:35.933209 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:35.933125 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-5ttl9" event={"ID":"1062428a-5ea4-4e47-a5a5-3515b7743f33","Type":"ContainerStarted","Data":"2b8405e0b003955c1b123f9443ca4a7f73c2537a5559f46950c075b394e7b5be"} Oct 31 00:42:35.933209 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:35.933176 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-5ttl9" event={"ID":"1062428a-5ea4-4e47-a5a5-3515b7743f33","Type":"ContainerStarted","Data":"32768d9fc121967e1c99cc4970d86add40a8f39b6b43c1a4e38f2c8c58cbad0e"} Oct 31 00:42:36.067474 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:36.067349 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22d9d793-snfv9"] Oct 31 00:42:36.071078 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:36.071053 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22d9d793-snfv9"] Oct 31 00:42:36.111735 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:36.111678 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-5ttl9" podStartSLOduration=2.111663234 podStartE2EDuration="2.111663234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:42:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:42:36.06701736 +0000 UTC m=+1648.538187110" watchObservedRunningTime="2026-10-31 00:42:36.111663234 +0000 UTC m=+1648.582832982" Oct 31 00:42:36.903826 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:36.903789 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-5ttl9" Oct 31 00:42:36.905870 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:36.905841 5222 patch_prober.go:29] interesting pod/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-5ttl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Oct 31 00:42:36.906002 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:36.905896 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-5ttl9" podUID="1062428a-5ea4-4e47-a5a5-3515b7743f33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Oct 31 00:42:37.236557 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:37.236461 5222 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-dd862feb-9c885f569-n957f" podUID="39ec242a-8946-48a5-b29a-47a211e70144" containerName="payload-processing" containerID="cri-o://fb72a1f23513ed2b1d852788442913aea9d1f648af6066ea074a9a5b793398ec" gracePeriod=25 Oct 31 00:42:37.363603 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:37.363545 5222 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-dd862feb-6c5d597495-9ftp6" podUID="bdbc3343-c465-4898-a845-f005363dbd36" containerName="payload-pre-processing" containerID="cri-o://a3e50288e2008ff08e5ee50ceed029dcba707e6fd0035d08ae21ed5e90aa2fe4" gracePeriod=25 Oct 31 00:42:37.602822 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:37.602800 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-dd862feb-9c885f569-n957f" Oct 31 00:42:37.634737 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:37.634705 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39ec242a-8946-48a5-b29a-47a211e70144-kube-api-access-nmhtp\" (UniqueName: \"kubernetes.io/projected/39ec242a-8946-48a5-b29a-47a211e70144-kube-api-access-nmhtp\") pod \"39ec242a-8946-48a5-b29a-47a211e70144\" (UID: \"39ec242a-8946-48a5-b29a-47a211e70144\") " Oct 31 00:42:37.635027 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:37.634996 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39ec242a-8946-48a5-b29a-47a211e70144-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39ec242a-8946-48a5-b29a-47a211e70144-plugins-config-volume\") pod \"39ec242a-8946-48a5-b29a-47a211e70144\" (UID: \"39ec242a-8946-48a5-b29a-47a211e70144\") " Oct 31 00:42:37.635330 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:37.635306 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39ec242a-8946-48a5-b29a-47a211e70144-plugins-config-volume" pod "39ec242a-8946-48a5-b29a-47a211e70144" (UID: "39ec242a-8946-48a5-b29a-47a211e70144"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:42:37.636664 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:37.636640 5222 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39ec242a-8946-48a5-b29a-47a211e70144-plugins-config-volume\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:37.668638 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:37.668600 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39ec242a-8946-48a5-b29a-47a211e70144-kube-api-access-nmhtp" pod "39ec242a-8946-48a5-b29a-47a211e70144" (UID: "39ec242a-8946-48a5-b29a-47a211e70144"). InnerVolumeSpecName "kube-api-access-nmhtp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:42:37.679454 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:37.678823 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b5754c6-397f-40a7-87a3-23462c9dcc94" path="/var/lib/kubelet/pods/6b5754c6-397f-40a7-87a3-23462c9dcc94/volumes" Oct 31 00:42:37.721336 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:37.721309 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-dd862feb-6c5d597495-9ftp6" Oct 31 00:42:37.751411 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:37.738599 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmhtp\" (UniqueName: \"kubernetes.io/projected/39ec242a-8946-48a5-b29a-47a211e70144-kube-api-access-nmhtp\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:37.861454 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:37.860934 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bdbc3343-c465-4898-a845-f005363dbd36-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdbc3343-c465-4898-a845-f005363dbd36-plugins-config-volume\") pod \"bdbc3343-c465-4898-a845-f005363dbd36\" (UID: \"bdbc3343-c465-4898-a845-f005363dbd36\") " Oct 31 00:42:37.861454 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:37.861186 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdbc3343-c465-4898-a845-f005363dbd36-kube-api-access-fzjmp\" (UniqueName: \"kubernetes.io/projected/bdbc3343-c465-4898-a845-f005363dbd36-kube-api-access-fzjmp\") pod \"bdbc3343-c465-4898-a845-f005363dbd36\" (UID: \"bdbc3343-c465-4898-a845-f005363dbd36\") " Oct 31 00:42:37.862395 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:37.861804 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdbc3343-c465-4898-a845-f005363dbd36-plugins-config-volume" pod "bdbc3343-c465-4898-a845-f005363dbd36" (UID: "bdbc3343-c465-4898-a845-f005363dbd36"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:42:37.862395 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:37.862028 5222 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdbc3343-c465-4898-a845-f005363dbd36-plugins-config-volume\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:37.865495 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:37.865461 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdbc3343-c465-4898-a845-f005363dbd36-kube-api-access-fzjmp" pod "bdbc3343-c465-4898-a845-f005363dbd36" (UID: "bdbc3343-c465-4898-a845-f005363dbd36"). InnerVolumeSpecName "kube-api-access-fzjmp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:42:37.904954 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:37.904909 5222 patch_prober.go:29] interesting pod/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-5ttl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Oct 31 00:42:37.905442 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:37.904983 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-5ttl9" podUID="1062428a-5ea4-4e47-a5a5-3515b7743f33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Oct 31 00:42:37.965075 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:37.965025 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzjmp\" (UniqueName: \"kubernetes.io/projected/bdbc3343-c465-4898-a845-f005363dbd36-kube-api-access-fzjmp\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:37.974876 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:37.974845 5222 generic.go:356] "Generic (PLEG): container finished" podID="39ec242a-8946-48a5-b29a-47a211e70144" containerID="fb72a1f23513ed2b1d852788442913aea9d1f648af6066ea074a9a5b793398ec" exitCode=0 Oct 31 00:42:37.975053 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:37.974891 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-dd862feb-9c885f569-n957f" Oct 31 00:42:37.975053 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:37.974925 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-dd862feb-9c885f569-n957f" event={"ID":"39ec242a-8946-48a5-b29a-47a211e70144","Type":"ContainerDied","Data":"fb72a1f23513ed2b1d852788442913aea9d1f648af6066ea074a9a5b793398ec"} Oct 31 00:42:37.975053 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:37.974983 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-dd862feb-9c885f569-n957f" event={"ID":"39ec242a-8946-48a5-b29a-47a211e70144","Type":"ContainerDied","Data":"61898b83cfcdd1bd8e9e95a0ca0f5dadce46d9e15abb1d7934af127564b0b998"} Oct 31 00:42:37.975053 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:37.975011 5222 scope.go:122] "RemoveContainer" containerID="fb72a1f23513ed2b1d852788442913aea9d1f648af6066ea074a9a5b793398ec" Oct 31 00:42:37.976994 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:37.976921 5222 generic.go:356] "Generic (PLEG): container finished" podID="bdbc3343-c465-4898-a845-f005363dbd36" containerID="a3e50288e2008ff08e5ee50ceed029dcba707e6fd0035d08ae21ed5e90aa2fe4" exitCode=0 Oct 31 00:42:37.977143 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:37.977027 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-dd862feb-6c5d597495-9ftp6" event={"ID":"bdbc3343-c465-4898-a845-f005363dbd36","Type":"ContainerDied","Data":"a3e50288e2008ff08e5ee50ceed029dcba707e6fd0035d08ae21ed5e90aa2fe4"} Oct 31 00:42:37.977143 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:37.977040 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-dd862feb-6c5d597495-9ftp6" Oct 31 00:42:37.977143 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:37.977078 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-dd862feb-6c5d597495-9ftp6" event={"ID":"bdbc3343-c465-4898-a845-f005363dbd36","Type":"ContainerDied","Data":"4cbe62f383c79753d55a7687f3ea3265a80e81f3b9bd3d8fcdb09b43be8bcd00"} Oct 31 00:42:37.987386 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:37.987354 5222 scope.go:122] "RemoveContainer" containerID="fb72a1f23513ed2b1d852788442913aea9d1f648af6066ea074a9a5b793398ec" Oct 31 00:42:37.987749 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:42:37.987730 5222 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb72a1f23513ed2b1d852788442913aea9d1f648af6066ea074a9a5b793398ec\": container with ID starting with fb72a1f23513ed2b1d852788442913aea9d1f648af6066ea074a9a5b793398ec not found: ID does not exist" containerID="fb72a1f23513ed2b1d852788442913aea9d1f648af6066ea074a9a5b793398ec" Oct 31 00:42:37.987830 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:37.987758 5222 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb72a1f23513ed2b1d852788442913aea9d1f648af6066ea074a9a5b793398ec"} err="failed to get container status \"fb72a1f23513ed2b1d852788442913aea9d1f648af6066ea074a9a5b793398ec\": rpc error: code = NotFound desc = could not find container \"fb72a1f23513ed2b1d852788442913aea9d1f648af6066ea074a9a5b793398ec\": container with ID starting with fb72a1f23513ed2b1d852788442913aea9d1f648af6066ea074a9a5b793398ec not found: ID does not exist" Oct 31 00:42:37.987830 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:37.987778 5222 scope.go:122] "RemoveContainer" containerID="a3e50288e2008ff08e5ee50ceed029dcba707e6fd0035d08ae21ed5e90aa2fe4" Oct 31 00:42:37.998479 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:37.998456 5222 scope.go:122] "RemoveContainer" containerID="a3e50288e2008ff08e5ee50ceed029dcba707e6fd0035d08ae21ed5e90aa2fe4" Oct 31 00:42:37.998808 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:42:37.998789 5222 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3e50288e2008ff08e5ee50ceed029dcba707e6fd0035d08ae21ed5e90aa2fe4\": container with ID starting with a3e50288e2008ff08e5ee50ceed029dcba707e6fd0035d08ae21ed5e90aa2fe4 not found: ID does not exist" containerID="a3e50288e2008ff08e5ee50ceed029dcba707e6fd0035d08ae21ed5e90aa2fe4" Oct 31 00:42:37.998855 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:37.998819 5222 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3e50288e2008ff08e5ee50ceed029dcba707e6fd0035d08ae21ed5e90aa2fe4"} err="failed to get container status \"a3e50288e2008ff08e5ee50ceed029dcba707e6fd0035d08ae21ed5e90aa2fe4\": rpc error: code = NotFound desc = could not find container \"a3e50288e2008ff08e5ee50ceed029dcba707e6fd0035d08ae21ed5e90aa2fe4\": container with ID starting with a3e50288e2008ff08e5ee50ceed029dcba707e6fd0035d08ae21ed5e90aa2fe4 not found: ID does not exist" Oct 31 00:42:38.167865 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:38.167770 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-dd862feb-6c5d597495-9ftp6"] Oct 31 00:42:38.171680 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:38.171647 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-dd862feb-6c5d597495-9ftp6"] Oct 31 00:42:38.274777 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:38.274736 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-dd862feb-9c885f569-n957f"] Oct 31 00:42:38.278951 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:38.278925 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-dd862feb-9c885f569-n957f"] Oct 31 00:42:38.904839 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:38.904796 5222 patch_prober.go:29] interesting pod/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-5ttl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Oct 31 00:42:38.905026 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:38.904859 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-5ttl9" podUID="1062428a-5ea4-4e47-a5a5-3515b7743f33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Oct 31 00:42:39.669841 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:39.669789 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39ec242a-8946-48a5-b29a-47a211e70144" path="/var/lib/kubelet/pods/39ec242a-8946-48a5-b29a-47a211e70144/volumes" Oct 31 00:42:39.670438 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:39.670414 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bdbc3343-c465-4898-a845-f005363dbd36" path="/var/lib/kubelet/pods/bdbc3343-c465-4898-a845-f005363dbd36/volumes" Oct 31 00:42:39.904642 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:39.904609 5222 patch_prober.go:29] interesting pod/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-5ttl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Oct 31 00:42:39.904819 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:39.904665 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-5ttl9" podUID="1062428a-5ea4-4e47-a5a5-3515b7743f33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Oct 31 00:42:40.067120 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:40.067086 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78659d5e-cpxdp"] Oct 31 00:42:40.075676 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:40.075653 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78659d5e-cpxdp"] Oct 31 00:42:40.904866 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:40.904825 5222 patch_prober.go:29] interesting pod/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-5ttl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Oct 31 00:42:40.905054 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:40.904886 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-5ttl9" podUID="1062428a-5ea4-4e47-a5a5-3515b7743f33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Oct 31 00:42:41.658387 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:41.658344 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00ed5f54-bc6b-4bb7-a68c-4e1c78078178" path="/var/lib/kubelet/pods/00ed5f54-bc6b-4bb7-a68c-4e1c78078178/volumes" Oct 31 00:42:41.908914 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:41.908839 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-5ttl9" Oct 31 00:42:41.909295 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:41.909280 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-5ttl9" Oct 31 00:42:41.910109 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:41.910090 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-5ttl9" Oct 31 00:42:42.558398 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:42.558164 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d58cd3-6ff958d595-t9nfb"] Oct 31 00:42:42.559300 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:42.559270 5222 cpu_manager.go:414] "containerMap: removing container" podUID="bdbc3343-c465-4898-a845-f005363dbd36" containerName="payload-pre-processing" Oct 31 00:42:42.559495 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:42.559300 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdbc3343-c465-4898-a845-f005363dbd36" containerName="payload-pre-processing" Oct 31 00:42:42.559495 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:42.559347 5222 cpu_manager.go:414] "containerMap: removing container" podUID="39ec242a-8946-48a5-b29a-47a211e70144" containerName="payload-processing" Oct 31 00:42:42.559495 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:42.559356 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39ec242a-8946-48a5-b29a-47a211e70144" containerName="payload-processing" Oct 31 00:42:42.559675 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:42.559607 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdbc3343-c465-4898-a845-f005363dbd36" containerName="payload-pre-processing" Oct 31 00:42:42.559675 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:42.559629 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="39ec242a-8946-48a5-b29a-47a211e70144" containerName="payload-processing" Oct 31 00:42:42.625430 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:42.608757 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d58cd3-6ff958d595-t9nfb" Oct 31 00:42:42.715395 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:42.695551 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-d58cd3-dockercfg-b246m\"" Oct 31 00:42:42.715395 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:42.695642 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-d58cd3\"" Oct 31 00:42:42.803827 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:42.795672 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dfms\" (UniqueName: \"kubernetes.io/projected/82e8f109-3468-431f-aabf-ef22d4e59878-kube-api-access-2dfms\") pod \"payload-pre-processing-e2e-shared-a-w2-d58cd3-6ff958d595-t9nfb\" (UID: \"82e8f109-3468-431f-aabf-ef22d4e59878\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d58cd3-6ff958d595-t9nfb" Oct 31 00:42:42.803827 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:42.795855 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82e8f109-3468-431f-aabf-ef22d4e59878-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-d58cd3-6ff958d595-t9nfb\" (UID: \"82e8f109-3468-431f-aabf-ef22d4e59878\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d58cd3-6ff958d595-t9nfb" Oct 31 00:42:42.848406 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:42.832519 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d58cd3-6ff958d595-t9nfb"] Oct 31 00:42:42.864014 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:42.863986 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-d58cd3-766b65cfd7-hjv58"] Oct 31 00:42:42.884562 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:42.882773 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-d58cd3-766b65cfd7-hjv58"] Oct 31 00:42:42.884562 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:42.882893 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d58cd3-766b65cfd7-hjv58" Oct 31 00:42:42.897054 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:42.897016 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82e8f109-3468-431f-aabf-ef22d4e59878-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-d58cd3-6ff958d595-t9nfb\" (UID: \"82e8f109-3468-431f-aabf-ef22d4e59878\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d58cd3-6ff958d595-t9nfb" Oct 31 00:42:42.897430 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:42.897405 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dfms\" (UniqueName: \"kubernetes.io/projected/82e8f109-3468-431f-aabf-ef22d4e59878-kube-api-access-2dfms\") pod \"payload-pre-processing-e2e-shared-a-w2-d58cd3-6ff958d595-t9nfb\" (UID: \"82e8f109-3468-431f-aabf-ef22d4e59878\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d58cd3-6ff958d595-t9nfb" Oct 31 00:42:42.898755 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:42.898734 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82e8f109-3468-431f-aabf-ef22d4e59878-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-d58cd3-6ff958d595-t9nfb\" (UID: \"82e8f109-3468-431f-aabf-ef22d4e59878\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d58cd3-6ff958d595-t9nfb" Oct 31 00:42:42.961276 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:42.961238 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dfms\" (UniqueName: \"kubernetes.io/projected/82e8f109-3468-431f-aabf-ef22d4e59878-kube-api-access-2dfms\") pod \"payload-pre-processing-e2e-shared-a-w2-d58cd3-6ff958d595-t9nfb\" (UID: \"82e8f109-3468-431f-aabf-ef22d4e59878\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d58cd3-6ff958d595-t9nfb" Oct 31 00:42:42.999181 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:42.999134 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f5b0e39-04be-4933-9512-8c7ff9bdb9cd-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-d58cd3-766b65cfd7-hjv58\" (UID: \"2f5b0e39-04be-4933-9512-8c7ff9bdb9cd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d58cd3-766b65cfd7-hjv58" Oct 31 00:42:42.999453 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:42.999432 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhrkr\" (UniqueName: \"kubernetes.io/projected/2f5b0e39-04be-4933-9512-8c7ff9bdb9cd-kube-api-access-zhrkr\") pod \"payload-processing-e2e-shared-a-w2-d58cd3-766b65cfd7-hjv58\" (UID: \"2f5b0e39-04be-4933-9512-8c7ff9bdb9cd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d58cd3-766b65cfd7-hjv58" Oct 31 00:42:43.100918 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:43.100826 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f5b0e39-04be-4933-9512-8c7ff9bdb9cd-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-d58cd3-766b65cfd7-hjv58\" (UID: \"2f5b0e39-04be-4933-9512-8c7ff9bdb9cd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d58cd3-766b65cfd7-hjv58" Oct 31 00:42:43.101102 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:43.101045 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhrkr\" (UniqueName: \"kubernetes.io/projected/2f5b0e39-04be-4933-9512-8c7ff9bdb9cd-kube-api-access-zhrkr\") pod \"payload-processing-e2e-shared-a-w2-d58cd3-766b65cfd7-hjv58\" (UID: \"2f5b0e39-04be-4933-9512-8c7ff9bdb9cd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d58cd3-766b65cfd7-hjv58" Oct 31 00:42:43.101678 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:43.101652 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f5b0e39-04be-4933-9512-8c7ff9bdb9cd-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-d58cd3-766b65cfd7-hjv58\" (UID: \"2f5b0e39-04be-4933-9512-8c7ff9bdb9cd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d58cd3-766b65cfd7-hjv58" Oct 31 00:42:43.164518 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:43.164480 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhrkr\" (UniqueName: \"kubernetes.io/projected/2f5b0e39-04be-4933-9512-8c7ff9bdb9cd-kube-api-access-zhrkr\") pod \"payload-processing-e2e-shared-a-w2-d58cd3-766b65cfd7-hjv58\" (UID: \"2f5b0e39-04be-4933-9512-8c7ff9bdb9cd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d58cd3-766b65cfd7-hjv58" Oct 31 00:42:43.191904 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:43.191869 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d58cd3-6ff958d595-t9nfb" Oct 31 00:42:43.207272 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:43.207240 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d58cd3-766b65cfd7-hjv58" Oct 31 00:42:43.585192 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:42:43.585159 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82e8f109_3468_431f_aabf_ef22d4e59878.slice/crio-c834460a5610ca6cccff679d9c3efc1c61c2c7bee122336ed60ec6de44e8c1f3 WatchSource:0}: Error finding container c834460a5610ca6cccff679d9c3efc1c61c2c7bee122336ed60ec6de44e8c1f3: Status 404 returned error can't find the container with id c834460a5610ca6cccff679d9c3efc1c61c2c7bee122336ed60ec6de44e8c1f3 Oct 31 00:42:43.589695 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:43.589666 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d58cd3-6ff958d595-t9nfb"] Oct 31 00:42:43.777507 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:43.777041 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-d58cd3-766b65cfd7-hjv58"] Oct 31 00:42:44.029828 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:44.029790 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d58cd3-6ff958d595-t9nfb" event={"ID":"82e8f109-3468-431f-aabf-ef22d4e59878","Type":"ContainerStarted","Data":"011efb1c2bb3e502346e8bc8f6fcf24e05c48fb3ea1caae656854edeb483cc6f"} Oct 31 00:42:44.030007 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:44.029864 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d58cd3-6ff958d595-t9nfb" Oct 31 00:42:44.030007 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:44.029887 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d58cd3-6ff958d595-t9nfb" event={"ID":"82e8f109-3468-431f-aabf-ef22d4e59878","Type":"ContainerStarted","Data":"c834460a5610ca6cccff679d9c3efc1c61c2c7bee122336ed60ec6de44e8c1f3"} Oct 31 00:42:44.031735 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:44.031704 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d58cd3-766b65cfd7-hjv58" event={"ID":"2f5b0e39-04be-4933-9512-8c7ff9bdb9cd","Type":"ContainerStarted","Data":"c2b0365fa6ee04175375ddc641e6734aa5facd2f44004496fbcb88e17baedd30"} Oct 31 00:42:44.031892 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:44.031754 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d58cd3-766b65cfd7-hjv58" event={"ID":"2f5b0e39-04be-4933-9512-8c7ff9bdb9cd","Type":"ContainerStarted","Data":"c1b4857e3de939e13d15ece1e418604bc1801ad5445599ebff631cfe7e2332d8"} Oct 31 00:42:44.031892 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:44.031775 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d58cd3-766b65cfd7-hjv58" Oct 31 00:42:44.150251 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:44.150135 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d58cd3-6ff958d595-t9nfb" podStartSLOduration=2.1501204879999998 podStartE2EDuration="2.150120488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:42:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:42:44.149222605 +0000 UTC m=+1656.620392354" watchObservedRunningTime="2026-10-31 00:42:44.150120488 +0000 UTC m=+1656.621290236" Oct 31 00:42:45.062647 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:45.062581 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d58cd3-766b65cfd7-hjv58" podStartSLOduration=3.062558486 podStartE2EDuration="3.062558486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:42:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:42:44.284143363 +0000 UTC m=+1656.755313107" watchObservedRunningTime="2026-10-31 00:42:45.062558486 +0000 UTC m=+1657.533728235" Oct 31 00:42:45.162396 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:45.145212 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70ab423e-lz5km"] Oct 31 00:42:45.162396 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:45.152569 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70ab423e-lz5km"] Oct 31 00:42:45.657603 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:45.657570 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67d02085-6466-428c-8315-ab10655f0dd2" path="/var/lib/kubelet/pods/67d02085-6466-428c-8315-ab10655f0dd2/volumes" Oct 31 00:42:47.056435 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:47.056395 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0cdffee-6zrmt"] Oct 31 00:42:47.107448 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:47.107393 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0cdffee-6zrmt"] Oct 31 00:42:47.657499 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:47.657467 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b1e4efb-7f81-45dd-abde-15b6a07ded61" path="/var/lib/kubelet/pods/0b1e4efb-7f81-45dd-abde-15b6a07ded61/volumes" Oct 31 00:42:55.163499 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:55.163466 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d58cd3-6ff958d595-t9nfb" Oct 31 00:42:55.163965 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:55.163533 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d58cd3-766b65cfd7-hjv58" Oct 31 00:42:56.060134 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:56.060090 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3088a498-m4ms8"] Oct 31 00:42:56.074786 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:56.074751 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3088a498-m4ms8"] Oct 31 00:42:57.658383 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:42:57.658336 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55f4e5f6-0ba4-4e7f-acef-07b16a1a523e" path="/var/lib/kubelet/pods/55f4e5f6-0ba4-4e7f-acef-07b16a1a523e/volumes" Oct 31 00:43:05.891563 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:43:05.891524 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 00:43:06.058917 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:06.058886 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-349c9c5e-22ggj"] Oct 31 00:43:06.110116 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:06.110084 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-349c9c5e-22ggj"] Oct 31 00:43:07.657606 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:07.657577 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9da6618-94f5-4e0d-9203-16642c36f901" path="/var/lib/kubelet/pods/c9da6618-94f5-4e0d-9203-16642c36f901/volumes" Oct 31 00:43:08.559051 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:08.559021 5222 scope.go:122] "RemoveContainer" containerID="73a6ea12b22cb3de90da79d70911a1f415fdad53ecdd18d201fcea1017921ea4" Oct 31 00:43:08.569866 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:08.569841 5222 scope.go:122] "RemoveContainer" containerID="2ccfd9e4c81fd99b874ae570a375da55ff324da7feab4cf8c32f9876729e91a8" Oct 31 00:43:08.580934 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:08.580910 5222 scope.go:122] "RemoveContainer" containerID="dbead3a9fee72047efdc001a4cbcc78958583c7fff7b9555efdf572653417966" Oct 31 00:43:08.591749 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:08.591727 5222 scope.go:122] "RemoveContainer" containerID="412fef350af0aeab64a890b63d041929f3a02f0805bdf7d6bf9a19dbc68b6572" Oct 31 00:43:08.603797 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:08.603775 5222 scope.go:122] "RemoveContainer" containerID="3ec14f207b23ab9014d528d3fa8d1ed205aea6c27989fe47789adb7191dcf7b0" Oct 31 00:43:08.614068 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:08.614044 5222 scope.go:122] "RemoveContainer" containerID="4f1952dd4e40840f355fdfc55a22b546e7aa7d917066619051778f385e2a87cb" Oct 31 00:43:08.626800 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:08.626779 5222 scope.go:122] "RemoveContainer" containerID="db4a16dd001f7ce0260b1e9305eead3b0cc915cd26f013b4b420f8821388ea42" Oct 31 00:43:08.638006 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:08.637980 5222 scope.go:122] "RemoveContainer" containerID="d674453d382ca74058341d3556b540aedf9bbc6d8fb279679411bb78b08ad3e0" Oct 31 00:43:08.648077 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:08.648058 5222 scope.go:122] "RemoveContainer" containerID="85779401936cb28c830631bdcccaf0766b51b789218f8bad7a2a2bbb77b1f2aa" Oct 31 00:43:08.658328 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:08.658299 5222 scope.go:122] "RemoveContainer" containerID="7bea0d31f78cfe6d4e92d4d5cd5d90b7cc161c7f477bc2cbc5dc18b172b1cf28" Oct 31 00:43:08.668644 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:08.668624 5222 scope.go:122] "RemoveContainer" containerID="17e63d20b9c2f609e8c3b902a01e95ea6cae7da72d6d45a6b01b086df1637c8e" Oct 31 00:43:13.471950 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:13.471914 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dd862feb-jc9cw"] Oct 31 00:43:13.522204 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:13.522170 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dd862feb-jc9cw"] Oct 31 00:43:13.657282 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:13.657246 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84752b79-f569-4777-af91-fd3604122ba9" path="/var/lib/kubelet/pods/84752b79-f569-4777-af91-fd3604122ba9/volumes" Oct 31 00:43:15.054520 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:15.054484 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-dd862feb-openshift-default-67b7c76895-zrjbk"] Oct 31 00:43:15.054946 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:15.054725 5222 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-dd862feb-openshift-default-67b7c76895-zrjbk" podUID="d7525d83-432f-4c20-aa47-6d51a4c88030" containerName="istio-proxy" containerID="cri-o://854ea1ef31b2f4fe88b1e02dd03bafd291079cb58f4de37c726f029fe3b143fa" gracePeriod=30 Oct 31 00:43:15.546112 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:15.546073 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-65wmw"] Oct 31 00:43:15.546418 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:15.546361 5222 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-65wmw" podUID="0faaa14f-bbf7-4915-ba75-4248ae943904" containerName="istio-proxy" containerID="cri-o://70cecf3323b0806d1e90ffdd4ed2c7290a2acf0caf9d10518818d4e10ab710b6" gracePeriod=30 Oct 31 00:43:15.549997 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:15.549966 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-zvppf"] Oct 31 00:43:15.550197 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:15.550166 5222 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-zvppf" podUID="4d3c7901-be21-4231-b9c9-ae15d4d19f92" containerName="istio-proxy" containerID="cri-o://3d57accae2ce7643502e38db5cc656d3d0fdd74e8057b02c8f9d15b435c6d5e7" gracePeriod=30 Oct 31 00:43:15.558886 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:15.558859 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sxdqn"] Oct 31 00:43:15.559125 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:15.559097 5222 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sxdqn" podUID="05e35f54-2479-4d75-b372-0f10ab6819c3" containerName="istio-proxy" containerID="cri-o://3682d6cda2b31af1a401bc2920770da23a88ca5efa60fab2c4d49bab22167f3c" gracePeriod=30 Oct 31 00:43:20.476008 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:20.475977 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-dd862feb-openshift-default-67b7c76895-zrjbk" Oct 31 00:43:20.654744 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:20.654351 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d7525d83-432f-4c20-aa47-6d51a4c88030-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7525d83-432f-4c20-aa47-6d51a4c88030-istio-podinfo\") pod \"d7525d83-432f-4c20-aa47-6d51a4c88030\" (UID: \"d7525d83-432f-4c20-aa47-6d51a4c88030\") " Oct 31 00:43:20.654930 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:20.654802 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7525d83-432f-4c20-aa47-6d51a4c88030-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7525d83-432f-4c20-aa47-6d51a4c88030-workload-socket\") pod \"d7525d83-432f-4c20-aa47-6d51a4c88030\" (UID: \"d7525d83-432f-4c20-aa47-6d51a4c88030\") " Oct 31 00:43:20.654930 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:20.654838 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7525d83-432f-4c20-aa47-6d51a4c88030-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7525d83-432f-4c20-aa47-6d51a4c88030-workload-certs\") pod \"d7525d83-432f-4c20-aa47-6d51a4c88030\" (UID: \"d7525d83-432f-4c20-aa47-6d51a4c88030\") " Oct 31 00:43:20.654930 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:20.654910 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7525d83-432f-4c20-aa47-6d51a4c88030-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7525d83-432f-4c20-aa47-6d51a4c88030-istio-data\") pod \"d7525d83-432f-4c20-aa47-6d51a4c88030\" (UID: \"d7525d83-432f-4c20-aa47-6d51a4c88030\") " Oct 31 00:43:20.655118 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:20.654988 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7525d83-432f-4c20-aa47-6d51a4c88030-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7525d83-432f-4c20-aa47-6d51a4c88030-istiod-ca-cert\") pod \"d7525d83-432f-4c20-aa47-6d51a4c88030\" (UID: \"d7525d83-432f-4c20-aa47-6d51a4c88030\") " Oct 31 00:43:20.655118 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:20.655057 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7525d83-432f-4c20-aa47-6d51a4c88030-kube-api-access-rzz5v\" (UniqueName: \"kubernetes.io/projected/d7525d83-432f-4c20-aa47-6d51a4c88030-kube-api-access-rzz5v\") pod \"d7525d83-432f-4c20-aa47-6d51a4c88030\" (UID: \"d7525d83-432f-4c20-aa47-6d51a4c88030\") " Oct 31 00:43:20.655118 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:20.655084 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7525d83-432f-4c20-aa47-6d51a4c88030-workload-socket" pod "d7525d83-432f-4c20-aa47-6d51a4c88030" (UID: "d7525d83-432f-4c20-aa47-6d51a4c88030"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:43:20.655495 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:20.655275 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7525d83-432f-4c20-aa47-6d51a4c88030-istio-data" pod "d7525d83-432f-4c20-aa47-6d51a4c88030" (UID: "d7525d83-432f-4c20-aa47-6d51a4c88030"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:43:20.655495 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:20.655474 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7525d83-432f-4c20-aa47-6d51a4c88030-workload-certs" pod "d7525d83-432f-4c20-aa47-6d51a4c88030" (UID: "d7525d83-432f-4c20-aa47-6d51a4c88030"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:43:20.655694 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:20.655673 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7525d83-432f-4c20-aa47-6d51a4c88030-credential-socket" pod "d7525d83-432f-4c20-aa47-6d51a4c88030" (UID: "d7525d83-432f-4c20-aa47-6d51a4c88030"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:43:20.656188 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:20.656162 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7525d83-432f-4c20-aa47-6d51a4c88030-istiod-ca-cert" pod "d7525d83-432f-4c20-aa47-6d51a4c88030" (UID: "d7525d83-432f-4c20-aa47-6d51a4c88030"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:43:20.656295 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:20.656198 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7525d83-432f-4c20-aa47-6d51a4c88030-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7525d83-432f-4c20-aa47-6d51a4c88030-credential-socket\") pod \"d7525d83-432f-4c20-aa47-6d51a4c88030\" (UID: \"d7525d83-432f-4c20-aa47-6d51a4c88030\") " Oct 31 00:43:20.656444 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:20.656341 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7525d83-432f-4c20-aa47-6d51a4c88030-istio-token\" (UniqueName: \"kubernetes.io/projected/d7525d83-432f-4c20-aa47-6d51a4c88030-istio-token\") pod \"d7525d83-432f-4c20-aa47-6d51a4c88030\" (UID: \"d7525d83-432f-4c20-aa47-6d51a4c88030\") " Oct 31 00:43:20.656532 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:20.656462 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7525d83-432f-4c20-aa47-6d51a4c88030-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7525d83-432f-4c20-aa47-6d51a4c88030-istio-envoy\") pod \"d7525d83-432f-4c20-aa47-6d51a4c88030\" (UID: \"d7525d83-432f-4c20-aa47-6d51a4c88030\") " Oct 31 00:43:20.657254 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:20.657171 5222 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7525d83-432f-4c20-aa47-6d51a4c88030-workload-socket\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:20.657254 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:20.657197 5222 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7525d83-432f-4c20-aa47-6d51a4c88030-workload-certs\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:20.657254 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:20.657214 5222 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7525d83-432f-4c20-aa47-6d51a4c88030-istio-data\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:20.657254 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:20.657228 5222 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7525d83-432f-4c20-aa47-6d51a4c88030-istiod-ca-cert\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:20.657254 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:20.657242 5222 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7525d83-432f-4c20-aa47-6d51a4c88030-credential-socket\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:20.657819 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:20.657792 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d7525d83-432f-4c20-aa47-6d51a4c88030-istio-podinfo" pod "d7525d83-432f-4c20-aa47-6d51a4c88030" (UID: "d7525d83-432f-4c20-aa47-6d51a4c88030"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:43:20.659691 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:20.659659 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7525d83-432f-4c20-aa47-6d51a4c88030-kube-api-access-rzz5v" pod "d7525d83-432f-4c20-aa47-6d51a4c88030" (UID: "d7525d83-432f-4c20-aa47-6d51a4c88030"). InnerVolumeSpecName "kube-api-access-rzz5v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:43:20.659875 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:20.659848 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7525d83-432f-4c20-aa47-6d51a4c88030-istio-envoy" pod "d7525d83-432f-4c20-aa47-6d51a4c88030" (UID: "d7525d83-432f-4c20-aa47-6d51a4c88030"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:43:20.660695 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:20.660388 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7525d83-432f-4c20-aa47-6d51a4c88030-istio-token" pod "d7525d83-432f-4c20-aa47-6d51a4c88030" (UID: "d7525d83-432f-4c20-aa47-6d51a4c88030"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:43:20.732799 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:20.732705 5222 generic.go:356] "Generic (PLEG): container finished" podID="4d3c7901-be21-4231-b9c9-ae15d4d19f92" containerID="3d57accae2ce7643502e38db5cc656d3d0fdd74e8057b02c8f9d15b435c6d5e7" exitCode=0 Oct 31 00:43:20.733118 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:20.733093 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-zvppf" event={"ID":"4d3c7901-be21-4231-b9c9-ae15d4d19f92","Type":"ContainerDied","Data":"3d57accae2ce7643502e38db5cc656d3d0fdd74e8057b02c8f9d15b435c6d5e7"} Oct 31 00:43:20.743589 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:20.740868 5222 generic.go:356] "Generic (PLEG): container finished" podID="0faaa14f-bbf7-4915-ba75-4248ae943904" containerID="70cecf3323b0806d1e90ffdd4ed2c7290a2acf0caf9d10518818d4e10ab710b6" exitCode=0 Oct 31 00:43:20.743589 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:20.740981 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-65wmw" event={"ID":"0faaa14f-bbf7-4915-ba75-4248ae943904","Type":"ContainerDied","Data":"70cecf3323b0806d1e90ffdd4ed2c7290a2acf0caf9d10518818d4e10ab710b6"} Oct 31 00:43:20.755005 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:20.754973 5222 generic.go:356] "Generic (PLEG): container finished" podID="d7525d83-432f-4c20-aa47-6d51a4c88030" containerID="854ea1ef31b2f4fe88b1e02dd03bafd291079cb58f4de37c726f029fe3b143fa" exitCode=0 Oct 31 00:43:20.755298 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:20.755057 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-dd862feb-openshift-default-67b7c76895-zrjbk" event={"ID":"d7525d83-432f-4c20-aa47-6d51a4c88030","Type":"ContainerDied","Data":"854ea1ef31b2f4fe88b1e02dd03bafd291079cb58f4de37c726f029fe3b143fa"} Oct 31 00:43:20.755423 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:20.755340 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-dd862feb-openshift-default-67b7c76895-zrjbk" event={"ID":"d7525d83-432f-4c20-aa47-6d51a4c88030","Type":"ContainerDied","Data":"dfa827e91a34fac84923099bb584ca887c2c13d56d117a252351bc6f16cd3826"} Oct 31 00:43:20.755423 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:20.755113 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-dd862feb-openshift-default-67b7c76895-zrjbk" Oct 31 00:43:20.755423 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:20.755407 5222 scope.go:122] "RemoveContainer" containerID="854ea1ef31b2f4fe88b1e02dd03bafd291079cb58f4de37c726f029fe3b143fa" Oct 31 00:43:20.758951 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:20.758462 5222 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7525d83-432f-4c20-aa47-6d51a4c88030-istio-podinfo\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:20.758951 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:20.758488 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzz5v\" (UniqueName: \"kubernetes.io/projected/d7525d83-432f-4c20-aa47-6d51a4c88030-kube-api-access-rzz5v\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:20.758951 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:20.758503 5222 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7525d83-432f-4c20-aa47-6d51a4c88030-istio-token\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:20.758951 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:20.758516 5222 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7525d83-432f-4c20-aa47-6d51a4c88030-istio-envoy\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:20.760640 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:20.760571 5222 generic.go:356] "Generic (PLEG): container finished" podID="05e35f54-2479-4d75-b372-0f10ab6819c3" containerID="3682d6cda2b31af1a401bc2920770da23a88ca5efa60fab2c4d49bab22167f3c" exitCode=0 Oct 31 00:43:20.760742 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:20.760638 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sxdqn" event={"ID":"05e35f54-2479-4d75-b372-0f10ab6819c3","Type":"ContainerDied","Data":"3682d6cda2b31af1a401bc2920770da23a88ca5efa60fab2c4d49bab22167f3c"} Oct 31 00:43:20.781532 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:20.773572 5222 scope.go:122] "RemoveContainer" containerID="8832255eadbeeddacb5f8ede93066f4fe2e30ad169a899e6f6541120691329b8" Oct 31 00:43:20.797597 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:20.797565 5222 scope.go:122] "RemoveContainer" containerID="854ea1ef31b2f4fe88b1e02dd03bafd291079cb58f4de37c726f029fe3b143fa" Oct 31 00:43:20.798796 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:43:20.798751 5222 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"854ea1ef31b2f4fe88b1e02dd03bafd291079cb58f4de37c726f029fe3b143fa\": container with ID starting with 854ea1ef31b2f4fe88b1e02dd03bafd291079cb58f4de37c726f029fe3b143fa not found: ID does not exist" containerID="854ea1ef31b2f4fe88b1e02dd03bafd291079cb58f4de37c726f029fe3b143fa" Oct 31 00:43:20.798983 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:20.798795 5222 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"854ea1ef31b2f4fe88b1e02dd03bafd291079cb58f4de37c726f029fe3b143fa"} err="failed to get container status \"854ea1ef31b2f4fe88b1e02dd03bafd291079cb58f4de37c726f029fe3b143fa\": rpc error: code = NotFound desc = could not find container \"854ea1ef31b2f4fe88b1e02dd03bafd291079cb58f4de37c726f029fe3b143fa\": container with ID starting with 854ea1ef31b2f4fe88b1e02dd03bafd291079cb58f4de37c726f029fe3b143fa not found: ID does not exist" Oct 31 00:43:20.798983 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:20.798823 5222 scope.go:122] "RemoveContainer" containerID="8832255eadbeeddacb5f8ede93066f4fe2e30ad169a899e6f6541120691329b8" Oct 31 00:43:20.799226 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:43:20.799101 5222 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8832255eadbeeddacb5f8ede93066f4fe2e30ad169a899e6f6541120691329b8\": container with ID starting with 8832255eadbeeddacb5f8ede93066f4fe2e30ad169a899e6f6541120691329b8 not found: ID does not exist" containerID="8832255eadbeeddacb5f8ede93066f4fe2e30ad169a899e6f6541120691329b8" Oct 31 00:43:20.799226 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:20.799127 5222 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8832255eadbeeddacb5f8ede93066f4fe2e30ad169a899e6f6541120691329b8"} err="failed to get container status \"8832255eadbeeddacb5f8ede93066f4fe2e30ad169a899e6f6541120691329b8\": rpc error: code = NotFound desc = could not find container \"8832255eadbeeddacb5f8ede93066f4fe2e30ad169a899e6f6541120691329b8\": container with ID starting with 8832255eadbeeddacb5f8ede93066f4fe2e30ad169a899e6f6541120691329b8 not found: ID does not exist" Oct 31 00:43:20.879358 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:20.879328 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-65wmw" Oct 31 00:43:20.923405 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:20.923356 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sxdqn" Oct 31 00:43:20.932820 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:20.932793 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-zvppf" Oct 31 00:43:20.933057 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:20.933046 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-dd862feb-openshift-default-67b7c76895-zrjbk"] Oct 31 00:43:20.936435 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:20.936407 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-dd862feb-openshift-default-67b7c76895-zrjbk"] Oct 31 00:43:21.062624 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:21.062591 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0faaa14f-bbf7-4915-ba75-4248ae943904-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0faaa14f-bbf7-4915-ba75-4248ae943904-credential-socket\") pod \"0faaa14f-bbf7-4915-ba75-4248ae943904\" (UID: \"0faaa14f-bbf7-4915-ba75-4248ae943904\") " Oct 31 00:43:21.062836 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:21.062641 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/05e35f54-2479-4d75-b372-0f10ab6819c3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05e35f54-2479-4d75-b372-0f10ab6819c3-istio-podinfo\") pod \"05e35f54-2479-4d75-b372-0f10ab6819c3\" (UID: \"05e35f54-2479-4d75-b372-0f10ab6819c3\") " Oct 31 00:43:21.062836 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:21.062666 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05e35f54-2479-4d75-b372-0f10ab6819c3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05e35f54-2479-4d75-b372-0f10ab6819c3-workload-socket\") pod \"05e35f54-2479-4d75-b372-0f10ab6819c3\" (UID: \"05e35f54-2479-4d75-b372-0f10ab6819c3\") " Oct 31 00:43:21.062836 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:21.062701 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05e35f54-2479-4d75-b372-0f10ab6819c3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05e35f54-2479-4d75-b372-0f10ab6819c3-istio-data\") pod \"05e35f54-2479-4d75-b372-0f10ab6819c3\" (UID: \"05e35f54-2479-4d75-b372-0f10ab6819c3\") " Oct 31 00:43:21.062836 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:21.062818 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0faaa14f-bbf7-4915-ba75-4248ae943904-kube-api-access-sf9rl\" (UniqueName: \"kubernetes.io/projected/0faaa14f-bbf7-4915-ba75-4248ae943904-kube-api-access-sf9rl\") pod \"0faaa14f-bbf7-4915-ba75-4248ae943904\" (UID: \"0faaa14f-bbf7-4915-ba75-4248ae943904\") " Oct 31 00:43:21.063122 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:21.062919 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d3c7901-be21-4231-b9c9-ae15d4d19f92-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d3c7901-be21-4231-b9c9-ae15d4d19f92-istio-envoy\") pod \"4d3c7901-be21-4231-b9c9-ae15d4d19f92\" (UID: \"4d3c7901-be21-4231-b9c9-ae15d4d19f92\") " Oct 31 00:43:21.063122 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:21.062932 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0faaa14f-bbf7-4915-ba75-4248ae943904-credential-socket" pod "0faaa14f-bbf7-4915-ba75-4248ae943904" (UID: "0faaa14f-bbf7-4915-ba75-4248ae943904"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:43:21.063122 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:21.062983 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4d3c7901-be21-4231-b9c9-ae15d4d19f92-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d3c7901-be21-4231-b9c9-ae15d4d19f92-istio-podinfo\") pod \"4d3c7901-be21-4231-b9c9-ae15d4d19f92\" (UID: \"4d3c7901-be21-4231-b9c9-ae15d4d19f92\") " Oct 31 00:43:21.063122 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:21.063038 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d3c7901-be21-4231-b9c9-ae15d4d19f92-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d3c7901-be21-4231-b9c9-ae15d4d19f92-workload-certs\") pod \"4d3c7901-be21-4231-b9c9-ae15d4d19f92\" (UID: \"4d3c7901-be21-4231-b9c9-ae15d4d19f92\") " Oct 31 00:43:21.063122 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:21.063064 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0faaa14f-bbf7-4915-ba75-4248ae943904-istio-token\" (UniqueName: \"kubernetes.io/projected/0faaa14f-bbf7-4915-ba75-4248ae943904-istio-token\") pod \"0faaa14f-bbf7-4915-ba75-4248ae943904\" (UID: \"0faaa14f-bbf7-4915-ba75-4248ae943904\") " Oct 31 00:43:21.063122 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:21.063109 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05e35f54-2479-4d75-b372-0f10ab6819c3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05e35f54-2479-4d75-b372-0f10ab6819c3-credential-socket\") pod \"05e35f54-2479-4d75-b372-0f10ab6819c3\" (UID: \"05e35f54-2479-4d75-b372-0f10ab6819c3\") " Oct 31 00:43:21.063444 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:21.063143 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0faaa14f-bbf7-4915-ba75-4248ae943904-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0faaa14f-bbf7-4915-ba75-4248ae943904-istio-data\") pod \"0faaa14f-bbf7-4915-ba75-4248ae943904\" (UID: \"0faaa14f-bbf7-4915-ba75-4248ae943904\") " Oct 31 00:43:21.063444 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:21.063188 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0faaa14f-bbf7-4915-ba75-4248ae943904-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0faaa14f-bbf7-4915-ba75-4248ae943904-istiod-ca-cert\") pod \"0faaa14f-bbf7-4915-ba75-4248ae943904\" (UID: \"0faaa14f-bbf7-4915-ba75-4248ae943904\") " Oct 31 00:43:21.063444 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:21.063272 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0faaa14f-bbf7-4915-ba75-4248ae943904-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0faaa14f-bbf7-4915-ba75-4248ae943904-istio-envoy\") pod \"0faaa14f-bbf7-4915-ba75-4248ae943904\" (UID: \"0faaa14f-bbf7-4915-ba75-4248ae943904\") " Oct 31 00:43:21.063444 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:21.063306 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d3c7901-be21-4231-b9c9-ae15d4d19f92-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d3c7901-be21-4231-b9c9-ae15d4d19f92-credential-socket\") pod \"4d3c7901-be21-4231-b9c9-ae15d4d19f92\" (UID: \"4d3c7901-be21-4231-b9c9-ae15d4d19f92\") " Oct 31 00:43:21.063444 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:21.063316 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d3c7901-be21-4231-b9c9-ae15d4d19f92-workload-certs" pod "4d3c7901-be21-4231-b9c9-ae15d4d19f92" (UID: "4d3c7901-be21-4231-b9c9-ae15d4d19f92"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:43:21.063444 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:21.063410 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05e35f54-2479-4d75-b372-0f10ab6819c3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05e35f54-2479-4d75-b372-0f10ab6819c3-istiod-ca-cert\") pod \"05e35f54-2479-4d75-b372-0f10ab6819c3\" (UID: \"05e35f54-2479-4d75-b372-0f10ab6819c3\") " Oct 31 00:43:21.063742 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:21.063446 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d3c7901-be21-4231-b9c9-ae15d4d19f92-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d3c7901-be21-4231-b9c9-ae15d4d19f92-istio-data\") pod \"4d3c7901-be21-4231-b9c9-ae15d4d19f92\" (UID: \"4d3c7901-be21-4231-b9c9-ae15d4d19f92\") " Oct 31 00:43:21.063742 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:21.063533 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0faaa14f-bbf7-4915-ba75-4248ae943904-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0faaa14f-bbf7-4915-ba75-4248ae943904-istio-podinfo\") pod \"0faaa14f-bbf7-4915-ba75-4248ae943904\" (UID: \"0faaa14f-bbf7-4915-ba75-4248ae943904\") " Oct 31 00:43:21.063742 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:21.063542 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0faaa14f-bbf7-4915-ba75-4248ae943904-istio-data" pod "0faaa14f-bbf7-4915-ba75-4248ae943904" (UID: "0faaa14f-bbf7-4915-ba75-4248ae943904"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:43:21.063742 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:21.063572 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05e35f54-2479-4d75-b372-0f10ab6819c3-istio-token\" (UniqueName: \"kubernetes.io/projected/05e35f54-2479-4d75-b372-0f10ab6819c3-istio-token\") pod \"05e35f54-2479-4d75-b372-0f10ab6819c3\" (UID: \"05e35f54-2479-4d75-b372-0f10ab6819c3\") " Oct 31 00:43:21.063742 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:21.063608 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d3c7901-be21-4231-b9c9-ae15d4d19f92-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d3c7901-be21-4231-b9c9-ae15d4d19f92-workload-socket\") pod \"4d3c7901-be21-4231-b9c9-ae15d4d19f92\" (UID: \"4d3c7901-be21-4231-b9c9-ae15d4d19f92\") " Oct 31 00:43:21.063742 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:21.063675 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d3c7901-be21-4231-b9c9-ae15d4d19f92-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d3c7901-be21-4231-b9c9-ae15d4d19f92-istiod-ca-cert\") pod \"4d3c7901-be21-4231-b9c9-ae15d4d19f92\" (UID: \"4d3c7901-be21-4231-b9c9-ae15d4d19f92\") " Oct 31 00:43:21.063742 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:21.063710 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05e35f54-2479-4d75-b372-0f10ab6819c3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05e35f54-2479-4d75-b372-0f10ab6819c3-workload-certs\") pod \"05e35f54-2479-4d75-b372-0f10ab6819c3\" (UID: \"05e35f54-2479-4d75-b372-0f10ab6819c3\") " Oct 31 00:43:21.063742 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:21.063740 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05e35f54-2479-4d75-b372-0f10ab6819c3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05e35f54-2479-4d75-b372-0f10ab6819c3-istio-envoy\") pod \"05e35f54-2479-4d75-b372-0f10ab6819c3\" (UID: \"05e35f54-2479-4d75-b372-0f10ab6819c3\") " Oct 31 00:43:21.064145 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:21.063766 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d3c7901-be21-4231-b9c9-ae15d4d19f92-istio-token\" (UniqueName: \"kubernetes.io/projected/4d3c7901-be21-4231-b9c9-ae15d4d19f92-istio-token\") pod \"4d3c7901-be21-4231-b9c9-ae15d4d19f92\" (UID: \"4d3c7901-be21-4231-b9c9-ae15d4d19f92\") " Oct 31 00:43:21.064145 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:21.063802 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0faaa14f-bbf7-4915-ba75-4248ae943904-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0faaa14f-bbf7-4915-ba75-4248ae943904-workload-certs\") pod \"0faaa14f-bbf7-4915-ba75-4248ae943904\" (UID: \"0faaa14f-bbf7-4915-ba75-4248ae943904\") " Oct 31 00:43:21.064145 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:21.063887 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0faaa14f-bbf7-4915-ba75-4248ae943904-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0faaa14f-bbf7-4915-ba75-4248ae943904-workload-socket\") pod \"0faaa14f-bbf7-4915-ba75-4248ae943904\" (UID: \"0faaa14f-bbf7-4915-ba75-4248ae943904\") " Oct 31 00:43:21.064145 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:21.063971 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d3c7901-be21-4231-b9c9-ae15d4d19f92-kube-api-access-8qgw9\" (UniqueName: \"kubernetes.io/projected/4d3c7901-be21-4231-b9c9-ae15d4d19f92-kube-api-access-8qgw9\") pod \"4d3c7901-be21-4231-b9c9-ae15d4d19f92\" (UID: \"4d3c7901-be21-4231-b9c9-ae15d4d19f92\") " Oct 31 00:43:21.064145 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:21.064041 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05e35f54-2479-4d75-b372-0f10ab6819c3-kube-api-access-gcwh7\" (UniqueName: \"kubernetes.io/projected/05e35f54-2479-4d75-b372-0f10ab6819c3-kube-api-access-gcwh7\") pod \"05e35f54-2479-4d75-b372-0f10ab6819c3\" (UID: \"05e35f54-2479-4d75-b372-0f10ab6819c3\") " Oct 31 00:43:21.064815 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:21.064795 5222 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d3c7901-be21-4231-b9c9-ae15d4d19f92-workload-certs\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:21.064959 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:21.064944 5222 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0faaa14f-bbf7-4915-ba75-4248ae943904-credential-socket\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:21.065974 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:21.065307 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05e35f54-2479-4d75-b372-0f10ab6819c3-workload-socket" pod "05e35f54-2479-4d75-b372-0f10ab6819c3" (UID: "05e35f54-2479-4d75-b372-0f10ab6819c3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:43:21.065974 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:21.065553 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05e35f54-2479-4d75-b372-0f10ab6819c3-credential-socket" pod "05e35f54-2479-4d75-b372-0f10ab6819c3" (UID: "05e35f54-2479-4d75-b372-0f10ab6819c3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:43:21.065974 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:21.065791 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0faaa14f-bbf7-4915-ba75-4248ae943904-istiod-ca-cert" pod "0faaa14f-bbf7-4915-ba75-4248ae943904" (UID: "0faaa14f-bbf7-4915-ba75-4248ae943904"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:43:21.067173 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:21.067137 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05e35f54-2479-4d75-b372-0f10ab6819c3-istio-data" pod "05e35f54-2479-4d75-b372-0f10ab6819c3" (UID: "05e35f54-2479-4d75-b372-0f10ab6819c3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:43:21.068327 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:21.068301 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d3c7901-be21-4231-b9c9-ae15d4d19f92-istio-data" pod "4d3c7901-be21-4231-b9c9-ae15d4d19f92" (UID: "4d3c7901-be21-4231-b9c9-ae15d4d19f92"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:43:21.068500 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:21.068481 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d3c7901-be21-4231-b9c9-ae15d4d19f92-credential-socket" pod "4d3c7901-be21-4231-b9c9-ae15d4d19f92" (UID: "4d3c7901-be21-4231-b9c9-ae15d4d19f92"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:43:21.068722 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:21.068706 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05e35f54-2479-4d75-b372-0f10ab6819c3-istiod-ca-cert" pod "05e35f54-2479-4d75-b372-0f10ab6819c3" (UID: "05e35f54-2479-4d75-b372-0f10ab6819c3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:43:21.068898 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:21.068870 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4d3c7901-be21-4231-b9c9-ae15d4d19f92-istio-podinfo" pod "4d3c7901-be21-4231-b9c9-ae15d4d19f92" (UID: "4d3c7901-be21-4231-b9c9-ae15d4d19f92"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:43:21.070045 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:21.069257 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/05e35f54-2479-4d75-b372-0f10ab6819c3-istio-podinfo" pod "05e35f54-2479-4d75-b372-0f10ab6819c3" (UID: "05e35f54-2479-4d75-b372-0f10ab6819c3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:43:21.070045 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:21.069260 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0faaa14f-bbf7-4915-ba75-4248ae943904-istio-token" pod "0faaa14f-bbf7-4915-ba75-4248ae943904" (UID: "0faaa14f-bbf7-4915-ba75-4248ae943904"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:43:21.070045 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:21.069671 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d3c7901-be21-4231-b9c9-ae15d4d19f92-istiod-ca-cert" pod "4d3c7901-be21-4231-b9c9-ae15d4d19f92" (UID: "4d3c7901-be21-4231-b9c9-ae15d4d19f92"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:43:21.070045 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:21.069733 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0faaa14f-bbf7-4915-ba75-4248ae943904-istio-envoy" pod "0faaa14f-bbf7-4915-ba75-4248ae943904" (UID: "0faaa14f-bbf7-4915-ba75-4248ae943904"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:43:21.070045 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:21.069887 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d3c7901-be21-4231-b9c9-ae15d4d19f92-workload-socket" pod "4d3c7901-be21-4231-b9c9-ae15d4d19f92" (UID: "4d3c7901-be21-4231-b9c9-ae15d4d19f92"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:43:21.070045 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:21.069963 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0faaa14f-bbf7-4915-ba75-4248ae943904-workload-socket" pod "0faaa14f-bbf7-4915-ba75-4248ae943904" (UID: "0faaa14f-bbf7-4915-ba75-4248ae943904"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:43:21.070045 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:21.069988 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05e35f54-2479-4d75-b372-0f10ab6819c3-workload-certs" pod "05e35f54-2479-4d75-b372-0f10ab6819c3" (UID: "05e35f54-2479-4d75-b372-0f10ab6819c3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:43:21.070045 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:21.070021 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0faaa14f-bbf7-4915-ba75-4248ae943904-workload-certs" pod "0faaa14f-bbf7-4915-ba75-4248ae943904" (UID: "0faaa14f-bbf7-4915-ba75-4248ae943904"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:43:21.070721 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:21.070577 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05e35f54-2479-4d75-b372-0f10ab6819c3-kube-api-access-gcwh7" pod "05e35f54-2479-4d75-b372-0f10ab6819c3" (UID: "05e35f54-2479-4d75-b372-0f10ab6819c3"). InnerVolumeSpecName "kube-api-access-gcwh7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:43:21.071480 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:21.071455 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0faaa14f-bbf7-4915-ba75-4248ae943904-istio-podinfo" pod "0faaa14f-bbf7-4915-ba75-4248ae943904" (UID: "0faaa14f-bbf7-4915-ba75-4248ae943904"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:43:21.072891 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:21.072866 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05e35f54-2479-4d75-b372-0f10ab6819c3-istio-envoy" pod "05e35f54-2479-4d75-b372-0f10ab6819c3" (UID: "05e35f54-2479-4d75-b372-0f10ab6819c3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:43:21.078967 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:21.078915 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d3c7901-be21-4231-b9c9-ae15d4d19f92-istio-envoy" pod "4d3c7901-be21-4231-b9c9-ae15d4d19f92" (UID: "4d3c7901-be21-4231-b9c9-ae15d4d19f92"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:43:21.078967 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:21.078955 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05e35f54-2479-4d75-b372-0f10ab6819c3-istio-token" pod "05e35f54-2479-4d75-b372-0f10ab6819c3" (UID: "05e35f54-2479-4d75-b372-0f10ab6819c3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:43:21.079166 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:21.079092 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d3c7901-be21-4231-b9c9-ae15d4d19f92-kube-api-access-8qgw9" pod "4d3c7901-be21-4231-b9c9-ae15d4d19f92" (UID: "4d3c7901-be21-4231-b9c9-ae15d4d19f92"). InnerVolumeSpecName "kube-api-access-8qgw9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:43:21.079232 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:21.079187 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0faaa14f-bbf7-4915-ba75-4248ae943904-kube-api-access-sf9rl" pod "0faaa14f-bbf7-4915-ba75-4248ae943904" (UID: "0faaa14f-bbf7-4915-ba75-4248ae943904"). InnerVolumeSpecName "kube-api-access-sf9rl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:43:21.080498 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:21.080472 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d3c7901-be21-4231-b9c9-ae15d4d19f92-istio-token" pod "4d3c7901-be21-4231-b9c9-ae15d4d19f92" (UID: "4d3c7901-be21-4231-b9c9-ae15d4d19f92"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:43:21.166147 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:21.166110 5222 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0faaa14f-bbf7-4915-ba75-4248ae943904-istio-podinfo\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:21.166147 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:21.166141 5222 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05e35f54-2479-4d75-b372-0f10ab6819c3-istio-token\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:21.166147 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:21.166151 5222 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d3c7901-be21-4231-b9c9-ae15d4d19f92-workload-socket\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:21.166420 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:21.166161 5222 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d3c7901-be21-4231-b9c9-ae15d4d19f92-istiod-ca-cert\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:21.166420 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:21.166172 5222 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05e35f54-2479-4d75-b372-0f10ab6819c3-workload-certs\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:21.166420 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:21.166180 5222 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05e35f54-2479-4d75-b372-0f10ab6819c3-istio-envoy\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:21.166420 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:21.166190 5222 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d3c7901-be21-4231-b9c9-ae15d4d19f92-istio-token\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:21.166420 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:21.166198 5222 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0faaa14f-bbf7-4915-ba75-4248ae943904-workload-certs\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:21.166420 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:21.166206 5222 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0faaa14f-bbf7-4915-ba75-4248ae943904-workload-socket\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:21.166420 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:21.166215 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qgw9\" (UniqueName: \"kubernetes.io/projected/4d3c7901-be21-4231-b9c9-ae15d4d19f92-kube-api-access-8qgw9\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:21.166420 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:21.166224 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcwh7\" (UniqueName: \"kubernetes.io/projected/05e35f54-2479-4d75-b372-0f10ab6819c3-kube-api-access-gcwh7\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:21.166420 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:21.166233 5222 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05e35f54-2479-4d75-b372-0f10ab6819c3-istio-podinfo\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:21.166420 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:21.166242 5222 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05e35f54-2479-4d75-b372-0f10ab6819c3-workload-socket\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:21.166420 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:21.166250 5222 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05e35f54-2479-4d75-b372-0f10ab6819c3-istio-data\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:21.166420 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:21.166258 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sf9rl\" (UniqueName: \"kubernetes.io/projected/0faaa14f-bbf7-4915-ba75-4248ae943904-kube-api-access-sf9rl\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:21.166420 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:21.166266 5222 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d3c7901-be21-4231-b9c9-ae15d4d19f92-istio-envoy\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:21.166420 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:21.166274 5222 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d3c7901-be21-4231-b9c9-ae15d4d19f92-istio-podinfo\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:21.166420 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:21.166282 5222 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0faaa14f-bbf7-4915-ba75-4248ae943904-istio-token\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:21.166420 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:21.166289 5222 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05e35f54-2479-4d75-b372-0f10ab6819c3-credential-socket\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:21.166420 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:21.166297 5222 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0faaa14f-bbf7-4915-ba75-4248ae943904-istio-data\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:21.166420 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:21.166305 5222 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0faaa14f-bbf7-4915-ba75-4248ae943904-istiod-ca-cert\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:21.166420 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:21.166312 5222 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0faaa14f-bbf7-4915-ba75-4248ae943904-istio-envoy\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:21.166420 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:21.166320 5222 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d3c7901-be21-4231-b9c9-ae15d4d19f92-credential-socket\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:21.166420 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:21.166328 5222 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05e35f54-2479-4d75-b372-0f10ab6819c3-istiod-ca-cert\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:21.166420 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:21.166338 5222 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d3c7901-be21-4231-b9c9-ae15d4d19f92-istio-data\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:21.657042 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:21.656991 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7525d83-432f-4c20-aa47-6d51a4c88030" path="/var/lib/kubelet/pods/d7525d83-432f-4c20-aa47-6d51a4c88030/volumes" Oct 31 00:43:21.770096 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:21.770061 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sxdqn" Oct 31 00:43:21.770292 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:21.770089 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sxdqn" event={"ID":"05e35f54-2479-4d75-b372-0f10ab6819c3","Type":"ContainerDied","Data":"04e54fd43944373cd5d77d1b48bbaa742d0ec5fd46a0fb12f5b15671f8ddaeeb"} Oct 31 00:43:21.770292 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:21.770164 5222 scope.go:122] "RemoveContainer" containerID="3682d6cda2b31af1a401bc2920770da23a88ca5efa60fab2c4d49bab22167f3c" Oct 31 00:43:21.772136 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:21.772105 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-zvppf" event={"ID":"4d3c7901-be21-4231-b9c9-ae15d4d19f92","Type":"ContainerDied","Data":"0bfd0f49b8fcd76f03f9697d774e6081f3496fd5df80abb805139cd2f2cff18f"} Oct 31 00:43:21.772136 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:21.772129 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-zvppf" Oct 31 00:43:21.776191 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:21.776165 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-65wmw" event={"ID":"0faaa14f-bbf7-4915-ba75-4248ae943904","Type":"ContainerDied","Data":"aba18a424d06984a70a5c5d85585af7582d3e7d9b4c247626a3f0bd9215009f9"} Oct 31 00:43:21.776191 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:21.776183 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-65wmw" Oct 31 00:43:21.783134 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:21.783110 5222 scope.go:122] "RemoveContainer" containerID="3d57accae2ce7643502e38db5cc656d3d0fdd74e8057b02c8f9d15b435c6d5e7" Oct 31 00:43:21.794618 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:21.794595 5222 scope.go:122] "RemoveContainer" containerID="70cecf3323b0806d1e90ffdd4ed2c7290a2acf0caf9d10518818d4e10ab710b6" Oct 31 00:43:21.944074 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:21.943988 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-zvppf"] Oct 31 00:43:21.949708 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:21.949675 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-zvppf"] Oct 31 00:43:22.067044 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:22.066979 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-65wmw"] Oct 31 00:43:22.071538 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:22.071511 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-65wmw"] Oct 31 00:43:22.222170 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:22.222074 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sxdqn"] Oct 31 00:43:22.226306 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:22.226268 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sxdqn"] Oct 31 00:43:23.656956 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:23.656924 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05e35f54-2479-4d75-b372-0f10ab6819c3" path="/var/lib/kubelet/pods/05e35f54-2479-4d75-b372-0f10ab6819c3/volumes" Oct 31 00:43:23.657458 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:23.657437 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0faaa14f-bbf7-4915-ba75-4248ae943904" path="/var/lib/kubelet/pods/0faaa14f-bbf7-4915-ba75-4248ae943904/volumes" Oct 31 00:43:23.657869 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:23.657856 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d3c7901-be21-4231-b9c9-ae15d4d19f92" path="/var/lib/kubelet/pods/4d3c7901-be21-4231-b9c9-ae15d4d19f92/volumes" Oct 31 00:43:24.996233 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:24.996192 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-74c6ba99-2d8jd"] Oct 31 00:43:25.045038 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:25.045005 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-74c6ba99-2d8jd"] Oct 31 00:43:25.051218 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:25.051187 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-5ttl9"] Oct 31 00:43:25.051520 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:25.051468 5222 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-5ttl9" podUID="1062428a-5ea4-4e47-a5a5-3515b7743f33" containerName="istio-proxy" containerID="cri-o://2b8405e0b003955c1b123f9443ca4a7f73c2537a5559f46950c075b394e7b5be" gracePeriod=30 Oct 31 00:43:25.627127 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:25.627089 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-0da668-openshift-default-5467f58b7-tzrb7"] Oct 31 00:43:25.627876 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:25.627827 5222 cpu_manager.go:414] "containerMap: removing container" podUID="d7525d83-432f-4c20-aa47-6d51a4c88030" containerName="istio-proxy" Oct 31 00:43:25.627876 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:25.627845 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7525d83-432f-4c20-aa47-6d51a4c88030" containerName="istio-proxy" Oct 31 00:43:25.627876 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:25.627877 5222 cpu_manager.go:414] "containerMap: removing container" podUID="05e35f54-2479-4d75-b372-0f10ab6819c3" containerName="istio-proxy" Oct 31 00:43:25.627876 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:25.627882 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05e35f54-2479-4d75-b372-0f10ab6819c3" containerName="istio-proxy" Oct 31 00:43:25.628241 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:25.627895 5222 cpu_manager.go:414] "containerMap: removing container" podUID="0faaa14f-bbf7-4915-ba75-4248ae943904" containerName="istio-proxy" Oct 31 00:43:25.628241 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:25.627900 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0faaa14f-bbf7-4915-ba75-4248ae943904" containerName="istio-proxy" Oct 31 00:43:25.628241 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:25.627918 5222 cpu_manager.go:414] "containerMap: removing container" podUID="4d3c7901-be21-4231-b9c9-ae15d4d19f92" containerName="istio-proxy" Oct 31 00:43:25.628241 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:25.627924 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d3c7901-be21-4231-b9c9-ae15d4d19f92" containerName="istio-proxy" Oct 31 00:43:25.628241 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:25.628033 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="0faaa14f-bbf7-4915-ba75-4248ae943904" containerName="istio-proxy" Oct 31 00:43:25.628241 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:25.628048 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d3c7901-be21-4231-b9c9-ae15d4d19f92" containerName="istio-proxy" Oct 31 00:43:25.628241 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:25.628058 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7525d83-432f-4c20-aa47-6d51a4c88030" containerName="istio-proxy" Oct 31 00:43:25.628241 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:25.628073 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7525d83-432f-4c20-aa47-6d51a4c88030" containerName="istio-proxy" Oct 31 00:43:25.628241 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:25.628086 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="05e35f54-2479-4d75-b372-0f10ab6819c3" containerName="istio-proxy" Oct 31 00:43:25.631896 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:25.631875 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-0da668-openshift-default-5467f58b7-tzrb7" Oct 31 00:43:25.769489 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:25.769345 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5eb6fe4-1be0-468b-91e0-d398870307b6-workload-socket\") pod \"e2e-shared-b-w4-0da668-openshift-default-5467f58b7-tzrb7\" (UID: \"d5eb6fe4-1be0-468b-91e0-d398870307b6\") " pod="openshift-ingress/e2e-shared-b-w4-0da668-openshift-default-5467f58b7-tzrb7" Oct 31 00:43:25.769489 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:25.769486 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5eb6fe4-1be0-468b-91e0-d398870307b6-credential-socket\") pod \"e2e-shared-b-w4-0da668-openshift-default-5467f58b7-tzrb7\" (UID: \"d5eb6fe4-1be0-468b-91e0-d398870307b6\") " pod="openshift-ingress/e2e-shared-b-w4-0da668-openshift-default-5467f58b7-tzrb7" Oct 31 00:43:25.769742 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:25.769580 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5eb6fe4-1be0-468b-91e0-d398870307b6-workload-certs\") pod \"e2e-shared-b-w4-0da668-openshift-default-5467f58b7-tzrb7\" (UID: \"d5eb6fe4-1be0-468b-91e0-d398870307b6\") " pod="openshift-ingress/e2e-shared-b-w4-0da668-openshift-default-5467f58b7-tzrb7" Oct 31 00:43:25.769742 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:25.769627 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5eb6fe4-1be0-468b-91e0-d398870307b6-istio-data\") pod \"e2e-shared-b-w4-0da668-openshift-default-5467f58b7-tzrb7\" (UID: \"d5eb6fe4-1be0-468b-91e0-d398870307b6\") " pod="openshift-ingress/e2e-shared-b-w4-0da668-openshift-default-5467f58b7-tzrb7" Oct 31 00:43:25.769742 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:25.769720 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5eb6fe4-1be0-468b-91e0-d398870307b6-istio-envoy\") pod \"e2e-shared-b-w4-0da668-openshift-default-5467f58b7-tzrb7\" (UID: \"d5eb6fe4-1be0-468b-91e0-d398870307b6\") " pod="openshift-ingress/e2e-shared-b-w4-0da668-openshift-default-5467f58b7-tzrb7" Oct 31 00:43:25.769902 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:25.769770 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5eb6fe4-1be0-468b-91e0-d398870307b6-istio-podinfo\") pod \"e2e-shared-b-w4-0da668-openshift-default-5467f58b7-tzrb7\" (UID: \"d5eb6fe4-1be0-468b-91e0-d398870307b6\") " pod="openshift-ingress/e2e-shared-b-w4-0da668-openshift-default-5467f58b7-tzrb7" Oct 31 00:43:25.769902 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:25.769802 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22ms5\" (UniqueName: \"kubernetes.io/projected/d5eb6fe4-1be0-468b-91e0-d398870307b6-kube-api-access-22ms5\") pod \"e2e-shared-b-w4-0da668-openshift-default-5467f58b7-tzrb7\" (UID: \"d5eb6fe4-1be0-468b-91e0-d398870307b6\") " pod="openshift-ingress/e2e-shared-b-w4-0da668-openshift-default-5467f58b7-tzrb7" Oct 31 00:43:25.769902 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:25.769844 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5eb6fe4-1be0-468b-91e0-d398870307b6-istio-token\") pod \"e2e-shared-b-w4-0da668-openshift-default-5467f58b7-tzrb7\" (UID: \"d5eb6fe4-1be0-468b-91e0-d398870307b6\") " pod="openshift-ingress/e2e-shared-b-w4-0da668-openshift-default-5467f58b7-tzrb7" Oct 31 00:43:25.770054 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:25.769901 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5eb6fe4-1be0-468b-91e0-d398870307b6-istiod-ca-cert\") pod \"e2e-shared-b-w4-0da668-openshift-default-5467f58b7-tzrb7\" (UID: \"d5eb6fe4-1be0-468b-91e0-d398870307b6\") " pod="openshift-ingress/e2e-shared-b-w4-0da668-openshift-default-5467f58b7-tzrb7" Oct 31 00:43:25.863389 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:25.836914 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-0da668-openshift-default-dockercfg-fpg6c\"" Oct 31 00:43:25.900495 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:25.878084 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5eb6fe4-1be0-468b-91e0-d398870307b6-istio-token\") pod \"e2e-shared-b-w4-0da668-openshift-default-5467f58b7-tzrb7\" (UID: \"d5eb6fe4-1be0-468b-91e0-d398870307b6\") " pod="openshift-ingress/e2e-shared-b-w4-0da668-openshift-default-5467f58b7-tzrb7" Oct 31 00:43:25.900495 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:25.878148 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5eb6fe4-1be0-468b-91e0-d398870307b6-istiod-ca-cert\") pod \"e2e-shared-b-w4-0da668-openshift-default-5467f58b7-tzrb7\" (UID: \"d5eb6fe4-1be0-468b-91e0-d398870307b6\") " pod="openshift-ingress/e2e-shared-b-w4-0da668-openshift-default-5467f58b7-tzrb7" Oct 31 00:43:25.900495 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:25.878222 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5eb6fe4-1be0-468b-91e0-d398870307b6-workload-socket\") pod \"e2e-shared-b-w4-0da668-openshift-default-5467f58b7-tzrb7\" (UID: \"d5eb6fe4-1be0-468b-91e0-d398870307b6\") " pod="openshift-ingress/e2e-shared-b-w4-0da668-openshift-default-5467f58b7-tzrb7" Oct 31 00:43:25.900495 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:25.878310 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5eb6fe4-1be0-468b-91e0-d398870307b6-credential-socket\") pod \"e2e-shared-b-w4-0da668-openshift-default-5467f58b7-tzrb7\" (UID: \"d5eb6fe4-1be0-468b-91e0-d398870307b6\") " pod="openshift-ingress/e2e-shared-b-w4-0da668-openshift-default-5467f58b7-tzrb7" Oct 31 00:43:25.900495 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:25.878403 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5eb6fe4-1be0-468b-91e0-d398870307b6-workload-certs\") pod \"e2e-shared-b-w4-0da668-openshift-default-5467f58b7-tzrb7\" (UID: \"d5eb6fe4-1be0-468b-91e0-d398870307b6\") " pod="openshift-ingress/e2e-shared-b-w4-0da668-openshift-default-5467f58b7-tzrb7" Oct 31 00:43:25.900495 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:25.878442 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5eb6fe4-1be0-468b-91e0-d398870307b6-istio-data\") pod \"e2e-shared-b-w4-0da668-openshift-default-5467f58b7-tzrb7\" (UID: \"d5eb6fe4-1be0-468b-91e0-d398870307b6\") " pod="openshift-ingress/e2e-shared-b-w4-0da668-openshift-default-5467f58b7-tzrb7" Oct 31 00:43:25.900495 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:25.878517 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5eb6fe4-1be0-468b-91e0-d398870307b6-istio-envoy\") pod \"e2e-shared-b-w4-0da668-openshift-default-5467f58b7-tzrb7\" (UID: \"d5eb6fe4-1be0-468b-91e0-d398870307b6\") " pod="openshift-ingress/e2e-shared-b-w4-0da668-openshift-default-5467f58b7-tzrb7" Oct 31 00:43:25.900495 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:25.878559 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5eb6fe4-1be0-468b-91e0-d398870307b6-istio-podinfo\") pod \"e2e-shared-b-w4-0da668-openshift-default-5467f58b7-tzrb7\" (UID: \"d5eb6fe4-1be0-468b-91e0-d398870307b6\") " pod="openshift-ingress/e2e-shared-b-w4-0da668-openshift-default-5467f58b7-tzrb7" Oct 31 00:43:25.900495 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:25.878582 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22ms5\" (UniqueName: \"kubernetes.io/projected/d5eb6fe4-1be0-468b-91e0-d398870307b6-kube-api-access-22ms5\") pod \"e2e-shared-b-w4-0da668-openshift-default-5467f58b7-tzrb7\" (UID: \"d5eb6fe4-1be0-468b-91e0-d398870307b6\") " pod="openshift-ingress/e2e-shared-b-w4-0da668-openshift-default-5467f58b7-tzrb7" Oct 31 00:43:25.900495 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:25.879627 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5eb6fe4-1be0-468b-91e0-d398870307b6-istiod-ca-cert\") pod \"e2e-shared-b-w4-0da668-openshift-default-5467f58b7-tzrb7\" (UID: \"d5eb6fe4-1be0-468b-91e0-d398870307b6\") " pod="openshift-ingress/e2e-shared-b-w4-0da668-openshift-default-5467f58b7-tzrb7" Oct 31 00:43:25.900495 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:25.879846 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5eb6fe4-1be0-468b-91e0-d398870307b6-workload-socket\") pod \"e2e-shared-b-w4-0da668-openshift-default-5467f58b7-tzrb7\" (UID: \"d5eb6fe4-1be0-468b-91e0-d398870307b6\") " pod="openshift-ingress/e2e-shared-b-w4-0da668-openshift-default-5467f58b7-tzrb7" Oct 31 00:43:25.900495 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:25.880009 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5eb6fe4-1be0-468b-91e0-d398870307b6-credential-socket\") pod \"e2e-shared-b-w4-0da668-openshift-default-5467f58b7-tzrb7\" (UID: \"d5eb6fe4-1be0-468b-91e0-d398870307b6\") " pod="openshift-ingress/e2e-shared-b-w4-0da668-openshift-default-5467f58b7-tzrb7" Oct 31 00:43:25.900495 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:25.880174 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5eb6fe4-1be0-468b-91e0-d398870307b6-workload-certs\") pod \"e2e-shared-b-w4-0da668-openshift-default-5467f58b7-tzrb7\" (UID: \"d5eb6fe4-1be0-468b-91e0-d398870307b6\") " pod="openshift-ingress/e2e-shared-b-w4-0da668-openshift-default-5467f58b7-tzrb7" Oct 31 00:43:25.900495 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:25.880343 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5eb6fe4-1be0-468b-91e0-d398870307b6-istio-data\") pod \"e2e-shared-b-w4-0da668-openshift-default-5467f58b7-tzrb7\" (UID: \"d5eb6fe4-1be0-468b-91e0-d398870307b6\") " pod="openshift-ingress/e2e-shared-b-w4-0da668-openshift-default-5467f58b7-tzrb7" Oct 31 00:43:26.004790 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:25.994100 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5eb6fe4-1be0-468b-91e0-d398870307b6-istio-envoy\") pod \"e2e-shared-b-w4-0da668-openshift-default-5467f58b7-tzrb7\" (UID: \"d5eb6fe4-1be0-468b-91e0-d398870307b6\") " pod="openshift-ingress/e2e-shared-b-w4-0da668-openshift-default-5467f58b7-tzrb7" Oct 31 00:43:26.008782 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:26.006096 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd4cd9f5-591c-42ac-8a69-415d4f143f49" path="/var/lib/kubelet/pods/cd4cd9f5-591c-42ac-8a69-415d4f143f49/volumes" Oct 31 00:43:26.009075 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:26.009047 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-0da668-openshift-default-5467f58b7-tzrb7"] Oct 31 00:43:26.018479 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:26.016168 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5eb6fe4-1be0-468b-91e0-d398870307b6-istio-podinfo\") pod \"e2e-shared-b-w4-0da668-openshift-default-5467f58b7-tzrb7\" (UID: \"d5eb6fe4-1be0-468b-91e0-d398870307b6\") " pod="openshift-ingress/e2e-shared-b-w4-0da668-openshift-default-5467f58b7-tzrb7" Oct 31 00:43:26.020584 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:26.020553 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22ms5\" (UniqueName: \"kubernetes.io/projected/d5eb6fe4-1be0-468b-91e0-d398870307b6-kube-api-access-22ms5\") pod \"e2e-shared-b-w4-0da668-openshift-default-5467f58b7-tzrb7\" (UID: \"d5eb6fe4-1be0-468b-91e0-d398870307b6\") " pod="openshift-ingress/e2e-shared-b-w4-0da668-openshift-default-5467f58b7-tzrb7" Oct 31 00:43:26.021903 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:26.021876 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5eb6fe4-1be0-468b-91e0-d398870307b6-istio-token\") pod \"e2e-shared-b-w4-0da668-openshift-default-5467f58b7-tzrb7\" (UID: \"d5eb6fe4-1be0-468b-91e0-d398870307b6\") " pod="openshift-ingress/e2e-shared-b-w4-0da668-openshift-default-5467f58b7-tzrb7" Oct 31 00:43:26.255229 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:26.255153 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-0da668-openshift-default-5467f58b7-tzrb7" Oct 31 00:43:26.890198 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:43:26.890149 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5eb6fe4_1be0_468b_91e0_d398870307b6.slice/crio-11248aeafc83293bda7e0f97d82528c46414f324e5eac55ddd3faaacc252bf25 WatchSource:0}: Error finding container 11248aeafc83293bda7e0f97d82528c46414f324e5eac55ddd3faaacc252bf25: Status 404 returned error can't find the container with id 11248aeafc83293bda7e0f97d82528c46414f324e5eac55ddd3faaacc252bf25 Oct 31 00:43:26.893211 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:26.893173 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:43:26.893323 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:26.893247 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:43:26.893323 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:26.893275 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:43:26.894285 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:26.894261 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-0da668-openshift-default-5467f58b7-tzrb7"] Oct 31 00:43:26.912198 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:26.912168 5222 patch_prober.go:29] interesting pod/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-5ttl9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:43:26.912286 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:26.912228 5222 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-5ttl9" podUID="1062428a-5ea4-4e47-a5a5-3515b7743f33" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:43:27.005350 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:27.004627 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-0da668-openshift-default-5467f58b7-tzrb7" event={"ID":"d5eb6fe4-1be0-468b-91e0-d398870307b6","Type":"ContainerStarted","Data":"7c7c18c30eb9362a22bd2e11df1913e7e2d5bacdb670e046c93ef47dc33eac3f"} Oct 31 00:43:27.005350 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:27.004691 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-0da668-openshift-default-5467f58b7-tzrb7" event={"ID":"d5eb6fe4-1be0-468b-91e0-d398870307b6","Type":"ContainerStarted","Data":"11248aeafc83293bda7e0f97d82528c46414f324e5eac55ddd3faaacc252bf25"} Oct 31 00:43:27.115121 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:27.115086 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-e030e0-openshift-default-5c976d9f4f-lt8jf"] Oct 31 00:43:27.116048 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:27.116020 5222 cpu_manager.go:414] "containerMap: removing container" podUID="d7525d83-432f-4c20-aa47-6d51a4c88030" containerName="istio-proxy" Oct 31 00:43:27.116048 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:27.116048 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7525d83-432f-4c20-aa47-6d51a4c88030" containerName="istio-proxy" Oct 31 00:43:27.120866 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:27.120844 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-e030e0-openshift-default-5c976d9f4f-lt8jf" Oct 31 00:43:27.148001 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:27.147939 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-0da668-openshift-default-5467f58b7-tzrb7" podStartSLOduration=2.147918341 podStartE2EDuration="2.147918341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:43:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:43:27.142757156 +0000 UTC m=+1699.613926915" watchObservedRunningTime="2026-10-31 00:43:27.147918341 +0000 UTC m=+1699.619088091" Oct 31 00:43:27.150483 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:27.150455 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-e030e0-openshift-default-5c976d9f4f-lt8jf"] Oct 31 00:43:27.167890 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:27.167856 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcada5bd-0764-4b0c-883a-d5d1867ca38e-istiod-ca-cert\") pod \"e2e-shared-a-w6-e030e0-openshift-default-5c976d9f4f-lt8jf\" (UID: \"fcada5bd-0764-4b0c-883a-d5d1867ca38e\") " pod="openshift-ingress/e2e-shared-a-w6-e030e0-openshift-default-5c976d9f4f-lt8jf" Oct 31 00:43:27.168100 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:27.167913 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcada5bd-0764-4b0c-883a-d5d1867ca38e-istio-token\") pod \"e2e-shared-a-w6-e030e0-openshift-default-5c976d9f4f-lt8jf\" (UID: \"fcada5bd-0764-4b0c-883a-d5d1867ca38e\") " pod="openshift-ingress/e2e-shared-a-w6-e030e0-openshift-default-5c976d9f4f-lt8jf" Oct 31 00:43:27.168100 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:27.168072 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcada5bd-0764-4b0c-883a-d5d1867ca38e-workload-socket\") pod \"e2e-shared-a-w6-e030e0-openshift-default-5c976d9f4f-lt8jf\" (UID: \"fcada5bd-0764-4b0c-883a-d5d1867ca38e\") " pod="openshift-ingress/e2e-shared-a-w6-e030e0-openshift-default-5c976d9f4f-lt8jf" Oct 31 00:43:27.168233 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:27.168215 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcada5bd-0764-4b0c-883a-d5d1867ca38e-credential-socket\") pod \"e2e-shared-a-w6-e030e0-openshift-default-5c976d9f4f-lt8jf\" (UID: \"fcada5bd-0764-4b0c-883a-d5d1867ca38e\") " pod="openshift-ingress/e2e-shared-a-w6-e030e0-openshift-default-5c976d9f4f-lt8jf" Oct 31 00:43:27.168293 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:27.168254 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcada5bd-0764-4b0c-883a-d5d1867ca38e-istio-data\") pod \"e2e-shared-a-w6-e030e0-openshift-default-5c976d9f4f-lt8jf\" (UID: \"fcada5bd-0764-4b0c-883a-d5d1867ca38e\") " pod="openshift-ingress/e2e-shared-a-w6-e030e0-openshift-default-5c976d9f4f-lt8jf" Oct 31 00:43:27.168464 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:27.168439 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcada5bd-0764-4b0c-883a-d5d1867ca38e-istio-podinfo\") pod \"e2e-shared-a-w6-e030e0-openshift-default-5c976d9f4f-lt8jf\" (UID: \"fcada5bd-0764-4b0c-883a-d5d1867ca38e\") " pod="openshift-ingress/e2e-shared-a-w6-e030e0-openshift-default-5c976d9f4f-lt8jf" Oct 31 00:43:27.168532 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:27.168504 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcada5bd-0764-4b0c-883a-d5d1867ca38e-workload-certs\") pod \"e2e-shared-a-w6-e030e0-openshift-default-5c976d9f4f-lt8jf\" (UID: \"fcada5bd-0764-4b0c-883a-d5d1867ca38e\") " pod="openshift-ingress/e2e-shared-a-w6-e030e0-openshift-default-5c976d9f4f-lt8jf" Oct 31 00:43:27.168571 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:27.168548 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcada5bd-0764-4b0c-883a-d5d1867ca38e-istio-envoy\") pod \"e2e-shared-a-w6-e030e0-openshift-default-5c976d9f4f-lt8jf\" (UID: \"fcada5bd-0764-4b0c-883a-d5d1867ca38e\") " pod="openshift-ingress/e2e-shared-a-w6-e030e0-openshift-default-5c976d9f4f-lt8jf" Oct 31 00:43:27.168621 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:27.168609 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqfcg\" (UniqueName: \"kubernetes.io/projected/fcada5bd-0764-4b0c-883a-d5d1867ca38e-kube-api-access-hqfcg\") pod \"e2e-shared-a-w6-e030e0-openshift-default-5c976d9f4f-lt8jf\" (UID: \"fcada5bd-0764-4b0c-883a-d5d1867ca38e\") " pod="openshift-ingress/e2e-shared-a-w6-e030e0-openshift-default-5c976d9f4f-lt8jf" Oct 31 00:43:27.176110 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:27.176081 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w6-e030e0-openshift-default-dockercfg-cn6kv\"" Oct 31 00:43:27.270472 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:27.257610 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-0da668-openshift-default-5467f58b7-tzrb7" Oct 31 00:43:27.278745 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:27.272759 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcada5bd-0764-4b0c-883a-d5d1867ca38e-istio-podinfo\") pod \"e2e-shared-a-w6-e030e0-openshift-default-5c976d9f4f-lt8jf\" (UID: \"fcada5bd-0764-4b0c-883a-d5d1867ca38e\") " pod="openshift-ingress/e2e-shared-a-w6-e030e0-openshift-default-5c976d9f4f-lt8jf" Oct 31 00:43:27.278745 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:27.272832 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcada5bd-0764-4b0c-883a-d5d1867ca38e-workload-certs\") pod \"e2e-shared-a-w6-e030e0-openshift-default-5c976d9f4f-lt8jf\" (UID: \"fcada5bd-0764-4b0c-883a-d5d1867ca38e\") " pod="openshift-ingress/e2e-shared-a-w6-e030e0-openshift-default-5c976d9f4f-lt8jf" Oct 31 00:43:27.278745 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:27.272882 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcada5bd-0764-4b0c-883a-d5d1867ca38e-istio-envoy\") pod \"e2e-shared-a-w6-e030e0-openshift-default-5c976d9f4f-lt8jf\" (UID: \"fcada5bd-0764-4b0c-883a-d5d1867ca38e\") " pod="openshift-ingress/e2e-shared-a-w6-e030e0-openshift-default-5c976d9f4f-lt8jf" Oct 31 00:43:27.278745 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:27.272971 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqfcg\" (UniqueName: \"kubernetes.io/projected/fcada5bd-0764-4b0c-883a-d5d1867ca38e-kube-api-access-hqfcg\") pod \"e2e-shared-a-w6-e030e0-openshift-default-5c976d9f4f-lt8jf\" (UID: \"fcada5bd-0764-4b0c-883a-d5d1867ca38e\") " pod="openshift-ingress/e2e-shared-a-w6-e030e0-openshift-default-5c976d9f4f-lt8jf" Oct 31 00:43:27.278745 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:27.273032 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcada5bd-0764-4b0c-883a-d5d1867ca38e-istiod-ca-cert\") pod \"e2e-shared-a-w6-e030e0-openshift-default-5c976d9f4f-lt8jf\" (UID: \"fcada5bd-0764-4b0c-883a-d5d1867ca38e\") " pod="openshift-ingress/e2e-shared-a-w6-e030e0-openshift-default-5c976d9f4f-lt8jf" Oct 31 00:43:27.278745 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:27.273067 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcada5bd-0764-4b0c-883a-d5d1867ca38e-istio-token\") pod \"e2e-shared-a-w6-e030e0-openshift-default-5c976d9f4f-lt8jf\" (UID: \"fcada5bd-0764-4b0c-883a-d5d1867ca38e\") " pod="openshift-ingress/e2e-shared-a-w6-e030e0-openshift-default-5c976d9f4f-lt8jf" Oct 31 00:43:27.278745 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:27.273160 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcada5bd-0764-4b0c-883a-d5d1867ca38e-workload-socket\") pod \"e2e-shared-a-w6-e030e0-openshift-default-5c976d9f4f-lt8jf\" (UID: \"fcada5bd-0764-4b0c-883a-d5d1867ca38e\") " pod="openshift-ingress/e2e-shared-a-w6-e030e0-openshift-default-5c976d9f4f-lt8jf" Oct 31 00:43:27.278745 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:27.273286 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcada5bd-0764-4b0c-883a-d5d1867ca38e-credential-socket\") pod \"e2e-shared-a-w6-e030e0-openshift-default-5c976d9f4f-lt8jf\" (UID: \"fcada5bd-0764-4b0c-883a-d5d1867ca38e\") " pod="openshift-ingress/e2e-shared-a-w6-e030e0-openshift-default-5c976d9f4f-lt8jf" Oct 31 00:43:27.278745 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:27.273320 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcada5bd-0764-4b0c-883a-d5d1867ca38e-istio-data\") pod \"e2e-shared-a-w6-e030e0-openshift-default-5c976d9f4f-lt8jf\" (UID: \"fcada5bd-0764-4b0c-883a-d5d1867ca38e\") " pod="openshift-ingress/e2e-shared-a-w6-e030e0-openshift-default-5c976d9f4f-lt8jf" Oct 31 00:43:27.278745 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:27.274029 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcada5bd-0764-4b0c-883a-d5d1867ca38e-istio-data\") pod \"e2e-shared-a-w6-e030e0-openshift-default-5c976d9f4f-lt8jf\" (UID: \"fcada5bd-0764-4b0c-883a-d5d1867ca38e\") " pod="openshift-ingress/e2e-shared-a-w6-e030e0-openshift-default-5c976d9f4f-lt8jf" Oct 31 00:43:27.282927 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:27.282887 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcada5bd-0764-4b0c-883a-d5d1867ca38e-workload-certs\") pod \"e2e-shared-a-w6-e030e0-openshift-default-5c976d9f4f-lt8jf\" (UID: \"fcada5bd-0764-4b0c-883a-d5d1867ca38e\") " pod="openshift-ingress/e2e-shared-a-w6-e030e0-openshift-default-5c976d9f4f-lt8jf" Oct 31 00:43:27.292406 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:27.284904 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcada5bd-0764-4b0c-883a-d5d1867ca38e-istiod-ca-cert\") pod \"e2e-shared-a-w6-e030e0-openshift-default-5c976d9f4f-lt8jf\" (UID: \"fcada5bd-0764-4b0c-883a-d5d1867ca38e\") " pod="openshift-ingress/e2e-shared-a-w6-e030e0-openshift-default-5c976d9f4f-lt8jf" Oct 31 00:43:27.292406 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:27.285217 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcada5bd-0764-4b0c-883a-d5d1867ca38e-workload-socket\") pod \"e2e-shared-a-w6-e030e0-openshift-default-5c976d9f4f-lt8jf\" (UID: \"fcada5bd-0764-4b0c-883a-d5d1867ca38e\") " pod="openshift-ingress/e2e-shared-a-w6-e030e0-openshift-default-5c976d9f4f-lt8jf" Oct 31 00:43:27.292406 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:27.285490 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcada5bd-0764-4b0c-883a-d5d1867ca38e-credential-socket\") pod \"e2e-shared-a-w6-e030e0-openshift-default-5c976d9f4f-lt8jf\" (UID: \"fcada5bd-0764-4b0c-883a-d5d1867ca38e\") " pod="openshift-ingress/e2e-shared-a-w6-e030e0-openshift-default-5c976d9f4f-lt8jf" Oct 31 00:43:27.340396 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:27.336996 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcada5bd-0764-4b0c-883a-d5d1867ca38e-istio-envoy\") pod \"e2e-shared-a-w6-e030e0-openshift-default-5c976d9f4f-lt8jf\" (UID: \"fcada5bd-0764-4b0c-883a-d5d1867ca38e\") " pod="openshift-ingress/e2e-shared-a-w6-e030e0-openshift-default-5c976d9f4f-lt8jf" Oct 31 00:43:27.340396 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:27.337600 5222 patch_prober.go:29] interesting pod/e2e-shared-b-w4-0da668-openshift-default-5467f58b7-tzrb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Oct 31 00:43:27.340396 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:27.337653 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-0da668-openshift-default-5467f58b7-tzrb7" podUID="d5eb6fe4-1be0-468b-91e0-d398870307b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Oct 31 00:43:27.356445 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:27.355050 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcada5bd-0764-4b0c-883a-d5d1867ca38e-istio-podinfo\") pod \"e2e-shared-a-w6-e030e0-openshift-default-5c976d9f4f-lt8jf\" (UID: \"fcada5bd-0764-4b0c-883a-d5d1867ca38e\") " pod="openshift-ingress/e2e-shared-a-w6-e030e0-openshift-default-5c976d9f4f-lt8jf" Oct 31 00:43:27.383464 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:27.383415 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcada5bd-0764-4b0c-883a-d5d1867ca38e-istio-token\") pod \"e2e-shared-a-w6-e030e0-openshift-default-5c976d9f4f-lt8jf\" (UID: \"fcada5bd-0764-4b0c-883a-d5d1867ca38e\") " pod="openshift-ingress/e2e-shared-a-w6-e030e0-openshift-default-5c976d9f4f-lt8jf" Oct 31 00:43:27.405820 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:27.402273 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqfcg\" (UniqueName: \"kubernetes.io/projected/fcada5bd-0764-4b0c-883a-d5d1867ca38e-kube-api-access-hqfcg\") pod \"e2e-shared-a-w6-e030e0-openshift-default-5c976d9f4f-lt8jf\" (UID: \"fcada5bd-0764-4b0c-883a-d5d1867ca38e\") " pod="openshift-ingress/e2e-shared-a-w6-e030e0-openshift-default-5c976d9f4f-lt8jf" Oct 31 00:43:27.452551 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:27.452299 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-e030e0-openshift-default-5c976d9f4f-lt8jf" Oct 31 00:43:27.924030 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:43:27.923985 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcada5bd_0764_4b0c_883a_d5d1867ca38e.slice/crio-b1cedde882e131ac0930f2539ebf3d8236a7632ed11d5dbe3f4748fea2760274 WatchSource:0}: Error finding container b1cedde882e131ac0930f2539ebf3d8236a7632ed11d5dbe3f4748fea2760274: Status 404 returned error can't find the container with id b1cedde882e131ac0930f2539ebf3d8236a7632ed11d5dbe3f4748fea2760274 Oct 31 00:43:27.926809 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:27.926783 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-e030e0-openshift-default-5c976d9f4f-lt8jf"] Oct 31 00:43:27.927029 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:27.926996 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:43:27.927076 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:27.927067 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:43:27.927115 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:27.927094 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:43:28.014695 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:28.014662 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-e030e0-openshift-default-5c976d9f4f-lt8jf" event={"ID":"fcada5bd-0764-4b0c-883a-d5d1867ca38e","Type":"ContainerStarted","Data":"b1cedde882e131ac0930f2539ebf3d8236a7632ed11d5dbe3f4748fea2760274"} Oct 31 00:43:28.261084 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:28.260988 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-0da668-openshift-default-5467f58b7-tzrb7" Oct 31 00:43:29.048622 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:29.048587 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-e030e0-openshift-default-5c976d9f4f-lt8jf" event={"ID":"fcada5bd-0764-4b0c-883a-d5d1867ca38e","Type":"ContainerStarted","Data":"aba32c5fa7816f3b70cd111f5250a6773ea14a12a382519cf4288f57f4ca8115"} Oct 31 00:43:29.050424 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:29.050399 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-0da668-openshift-default-5467f58b7-tzrb7" Oct 31 00:43:29.056457 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:29.053011 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-0da668-openshift-default-5467f58b7-tzrb7" Oct 31 00:43:29.403420 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:29.402743 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-e030e0-openshift-default-5c976d9f4f-lt8jf" podStartSLOduration=2.402720373 podStartE2EDuration="2.402720373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:43:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:43:29.385138291 +0000 UTC m=+1701.856308055" watchObservedRunningTime="2026-10-31 00:43:29.402720373 +0000 UTC m=+1701.873890123" Oct 31 00:43:29.485392 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:29.464667 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-e030e0-openshift-default-5c976d9f4f-lt8jf" Oct 31 00:43:29.536395 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:29.531635 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-e030e0-openshift-default-5c976d9f4f-lt8jf" Oct 31 00:43:30.057864 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:30.057833 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-e030e0-openshift-default-5c976d9f4f-lt8jf" Oct 31 00:43:30.058935 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:30.058919 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-e030e0-openshift-default-5c976d9f4f-lt8jf" Oct 31 00:43:30.470879 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:30.470832 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-5ttl9" Oct 31 00:43:30.638447 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:30.637591 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1062428a-5ea4-4e47-a5a5-3515b7743f33-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1062428a-5ea4-4e47-a5a5-3515b7743f33-credential-socket\") pod \"1062428a-5ea4-4e47-a5a5-3515b7743f33\" (UID: \"1062428a-5ea4-4e47-a5a5-3515b7743f33\") " Oct 31 00:43:30.638447 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:30.637651 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1062428a-5ea4-4e47-a5a5-3515b7743f33-istio-token\" (UniqueName: \"kubernetes.io/projected/1062428a-5ea4-4e47-a5a5-3515b7743f33-istio-token\") pod \"1062428a-5ea4-4e47-a5a5-3515b7743f33\" (UID: \"1062428a-5ea4-4e47-a5a5-3515b7743f33\") " Oct 31 00:43:30.638447 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:30.637750 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1062428a-5ea4-4e47-a5a5-3515b7743f33-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1062428a-5ea4-4e47-a5a5-3515b7743f33-workload-socket\") pod \"1062428a-5ea4-4e47-a5a5-3515b7743f33\" (UID: \"1062428a-5ea4-4e47-a5a5-3515b7743f33\") " Oct 31 00:43:30.638447 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:30.637785 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1062428a-5ea4-4e47-a5a5-3515b7743f33-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1062428a-5ea4-4e47-a5a5-3515b7743f33-workload-certs\") pod \"1062428a-5ea4-4e47-a5a5-3515b7743f33\" (UID: \"1062428a-5ea4-4e47-a5a5-3515b7743f33\") " Oct 31 00:43:30.638447 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:30.637926 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1062428a-5ea4-4e47-a5a5-3515b7743f33-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1062428a-5ea4-4e47-a5a5-3515b7743f33-istio-envoy\") pod \"1062428a-5ea4-4e47-a5a5-3515b7743f33\" (UID: \"1062428a-5ea4-4e47-a5a5-3515b7743f33\") " Oct 31 00:43:30.638447 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:30.637976 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1062428a-5ea4-4e47-a5a5-3515b7743f33-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1062428a-5ea4-4e47-a5a5-3515b7743f33-istiod-ca-cert\") pod \"1062428a-5ea4-4e47-a5a5-3515b7743f33\" (UID: \"1062428a-5ea4-4e47-a5a5-3515b7743f33\") " Oct 31 00:43:30.638447 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:30.638050 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1062428a-5ea4-4e47-a5a5-3515b7743f33-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1062428a-5ea4-4e47-a5a5-3515b7743f33-istio-data\") pod \"1062428a-5ea4-4e47-a5a5-3515b7743f33\" (UID: \"1062428a-5ea4-4e47-a5a5-3515b7743f33\") " Oct 31 00:43:30.638447 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:30.638109 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1062428a-5ea4-4e47-a5a5-3515b7743f33-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1062428a-5ea4-4e47-a5a5-3515b7743f33-istio-podinfo\") pod \"1062428a-5ea4-4e47-a5a5-3515b7743f33\" (UID: \"1062428a-5ea4-4e47-a5a5-3515b7743f33\") " Oct 31 00:43:30.638447 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:30.638139 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1062428a-5ea4-4e47-a5a5-3515b7743f33-kube-api-access-fw4b9\" (UniqueName: \"kubernetes.io/projected/1062428a-5ea4-4e47-a5a5-3515b7743f33-kube-api-access-fw4b9\") pod \"1062428a-5ea4-4e47-a5a5-3515b7743f33\" (UID: \"1062428a-5ea4-4e47-a5a5-3515b7743f33\") " Oct 31 00:43:30.655425 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:30.640109 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1062428a-5ea4-4e47-a5a5-3515b7743f33-workload-certs" pod "1062428a-5ea4-4e47-a5a5-3515b7743f33" (UID: "1062428a-5ea4-4e47-a5a5-3515b7743f33"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:43:30.655425 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:30.640320 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1062428a-5ea4-4e47-a5a5-3515b7743f33-workload-socket" pod "1062428a-5ea4-4e47-a5a5-3515b7743f33" (UID: "1062428a-5ea4-4e47-a5a5-3515b7743f33"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:43:30.655425 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:30.640537 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1062428a-5ea4-4e47-a5a5-3515b7743f33-credential-socket" pod "1062428a-5ea4-4e47-a5a5-3515b7743f33" (UID: "1062428a-5ea4-4e47-a5a5-3515b7743f33"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:43:30.655425 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:30.640882 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1062428a-5ea4-4e47-a5a5-3515b7743f33-istiod-ca-cert" pod "1062428a-5ea4-4e47-a5a5-3515b7743f33" (UID: "1062428a-5ea4-4e47-a5a5-3515b7743f33"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:43:30.655425 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:30.641428 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1062428a-5ea4-4e47-a5a5-3515b7743f33-istio-data" pod "1062428a-5ea4-4e47-a5a5-3515b7743f33" (UID: "1062428a-5ea4-4e47-a5a5-3515b7743f33"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:43:30.655425 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:30.648188 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1062428a-5ea4-4e47-a5a5-3515b7743f33-kube-api-access-fw4b9" pod "1062428a-5ea4-4e47-a5a5-3515b7743f33" (UID: "1062428a-5ea4-4e47-a5a5-3515b7743f33"). InnerVolumeSpecName "kube-api-access-fw4b9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:43:30.655425 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:30.651870 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1062428a-5ea4-4e47-a5a5-3515b7743f33-istio-envoy" pod "1062428a-5ea4-4e47-a5a5-3515b7743f33" (UID: "1062428a-5ea4-4e47-a5a5-3515b7743f33"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:43:30.655425 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:30.651980 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1062428a-5ea4-4e47-a5a5-3515b7743f33-istio-token" pod "1062428a-5ea4-4e47-a5a5-3515b7743f33" (UID: "1062428a-5ea4-4e47-a5a5-3515b7743f33"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:43:30.655425 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:30.652037 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1062428a-5ea4-4e47-a5a5-3515b7743f33-istio-podinfo" pod "1062428a-5ea4-4e47-a5a5-3515b7743f33" (UID: "1062428a-5ea4-4e47-a5a5-3515b7743f33"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:43:30.740237 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:30.740200 5222 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1062428a-5ea4-4e47-a5a5-3515b7743f33-istio-data\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:30.740237 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:30.740239 5222 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1062428a-5ea4-4e47-a5a5-3515b7743f33-istio-podinfo\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:30.740487 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:30.740259 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fw4b9\" (UniqueName: \"kubernetes.io/projected/1062428a-5ea4-4e47-a5a5-3515b7743f33-kube-api-access-fw4b9\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:30.740487 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:30.740275 5222 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1062428a-5ea4-4e47-a5a5-3515b7743f33-credential-socket\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:30.740487 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:30.740293 5222 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1062428a-5ea4-4e47-a5a5-3515b7743f33-istio-token\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:30.740487 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:30.740313 5222 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1062428a-5ea4-4e47-a5a5-3515b7743f33-workload-socket\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:30.740487 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:30.740326 5222 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1062428a-5ea4-4e47-a5a5-3515b7743f33-workload-certs\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:30.740487 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:30.740335 5222 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1062428a-5ea4-4e47-a5a5-3515b7743f33-istio-envoy\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:30.740487 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:30.740343 5222 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1062428a-5ea4-4e47-a5a5-3515b7743f33-istiod-ca-cert\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:31.067177 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:31.067134 5222 generic.go:356] "Generic (PLEG): container finished" podID="1062428a-5ea4-4e47-a5a5-3515b7743f33" containerID="2b8405e0b003955c1b123f9443ca4a7f73c2537a5559f46950c075b394e7b5be" exitCode=0 Oct 31 00:43:31.067695 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:31.067203 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-5ttl9" Oct 31 00:43:31.067695 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:31.067232 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-5ttl9" event={"ID":"1062428a-5ea4-4e47-a5a5-3515b7743f33","Type":"ContainerDied","Data":"2b8405e0b003955c1b123f9443ca4a7f73c2537a5559f46950c075b394e7b5be"} Oct 31 00:43:31.067695 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:31.067290 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-5ttl9" event={"ID":"1062428a-5ea4-4e47-a5a5-3515b7743f33","Type":"ContainerDied","Data":"32768d9fc121967e1c99cc4970d86add40a8f39b6b43c1a4e38f2c8c58cbad0e"} Oct 31 00:43:31.067695 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:31.067319 5222 scope.go:122] "RemoveContainer" containerID="2b8405e0b003955c1b123f9443ca4a7f73c2537a5559f46950c075b394e7b5be" Oct 31 00:43:31.087420 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:31.087400 5222 scope.go:122] "RemoveContainer" containerID="2b8405e0b003955c1b123f9443ca4a7f73c2537a5559f46950c075b394e7b5be" Oct 31 00:43:31.087766 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:43:31.087728 5222 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b8405e0b003955c1b123f9443ca4a7f73c2537a5559f46950c075b394e7b5be\": container with ID starting with 2b8405e0b003955c1b123f9443ca4a7f73c2537a5559f46950c075b394e7b5be not found: ID does not exist" containerID="2b8405e0b003955c1b123f9443ca4a7f73c2537a5559f46950c075b394e7b5be" Oct 31 00:43:31.087868 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:31.087760 5222 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b8405e0b003955c1b123f9443ca4a7f73c2537a5559f46950c075b394e7b5be"} err="failed to get container status \"2b8405e0b003955c1b123f9443ca4a7f73c2537a5559f46950c075b394e7b5be\": rpc error: code = NotFound desc = could not find container \"2b8405e0b003955c1b123f9443ca4a7f73c2537a5559f46950c075b394e7b5be\": container with ID starting with 2b8405e0b003955c1b123f9443ca4a7f73c2537a5559f46950c075b394e7b5be not found: ID does not exist" Oct 31 00:43:31.240859 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:31.240828 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-5ttl9"] Oct 31 00:43:31.291670 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:31.291634 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-5ttl9"] Oct 31 00:43:31.657558 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:31.657528 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1062428a-5ea4-4e47-a5a5-3515b7743f33" path="/var/lib/kubelet/pods/1062428a-5ea4-4e47-a5a5-3515b7743f33/volumes" Oct 31 00:43:35.891200 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:43:35.891162 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 00:43:40.811204 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:40.792300 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-0da668-dff997c59-8nrck"] Oct 31 00:43:40.812779 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:40.812755 5222 cpu_manager.go:414] "containerMap: removing container" podUID="1062428a-5ea4-4e47-a5a5-3515b7743f33" containerName="istio-proxy" Oct 31 00:43:40.812951 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:40.812938 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1062428a-5ea4-4e47-a5a5-3515b7743f33" containerName="istio-proxy" Oct 31 00:43:40.813257 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:40.813245 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="1062428a-5ea4-4e47-a5a5-3515b7743f33" containerName="istio-proxy" Oct 31 00:43:40.819666 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:40.819647 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-0da668-dff997c59-8nrck" Oct 31 00:43:40.833087 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:40.833064 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-0da668-dff997c59-8nrck"] Oct 31 00:43:40.878562 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:40.878529 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-0da668\"" Oct 31 00:43:40.878781 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:40.878652 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-0da668-dockercfg-js6c6\"" Oct 31 00:43:40.897917 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:40.897779 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8532f94b-ef23-4b4d-967f-dc2aa9e3c3dd-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-0da668-dff997c59-8nrck\" (UID: \"8532f94b-ef23-4b4d-967f-dc2aa9e3c3dd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-0da668-dff997c59-8nrck" Oct 31 00:43:40.898534 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:40.898508 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdjwd\" (UniqueName: \"kubernetes.io/projected/8532f94b-ef23-4b4d-967f-dc2aa9e3c3dd-kube-api-access-sdjwd\") pod \"payload-pre-processing-e2e-shared-b-w4-0da668-dff997c59-8nrck\" (UID: \"8532f94b-ef23-4b4d-967f-dc2aa9e3c3dd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-0da668-dff997c59-8nrck" Oct 31 00:43:40.993180 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:40.993147 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-0da668-78b845796f-5srmm"] Oct 31 00:43:41.027740 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:41.027696 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8532f94b-ef23-4b4d-967f-dc2aa9e3c3dd-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-0da668-dff997c59-8nrck\" (UID: \"8532f94b-ef23-4b4d-967f-dc2aa9e3c3dd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-0da668-dff997c59-8nrck" Oct 31 00:43:41.029135 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:41.029109 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdjwd\" (UniqueName: \"kubernetes.io/projected/8532f94b-ef23-4b4d-967f-dc2aa9e3c3dd-kube-api-access-sdjwd\") pod \"payload-pre-processing-e2e-shared-b-w4-0da668-dff997c59-8nrck\" (UID: \"8532f94b-ef23-4b4d-967f-dc2aa9e3c3dd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-0da668-dff997c59-8nrck" Oct 31 00:43:41.029904 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:41.028860 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8532f94b-ef23-4b4d-967f-dc2aa9e3c3dd-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-0da668-dff997c59-8nrck\" (UID: \"8532f94b-ef23-4b4d-967f-dc2aa9e3c3dd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-0da668-dff997c59-8nrck" Oct 31 00:43:41.056760 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:41.056727 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-0da668-78b845796f-5srmm" Oct 31 00:43:41.065488 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:41.062541 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f0651cac-k697q"] Oct 31 00:43:41.088246 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:41.087459 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-0da668-78b845796f-5srmm"] Oct 31 00:43:41.088246 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:41.087591 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f0651cac-k697q" Oct 31 00:43:41.148913 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:41.148888 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:43:41.149135 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:41.149116 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:43:41.151138 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:41.151110 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f0651cac-k697q"] Oct 31 00:43:41.156278 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:41.156230 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdjwd\" (UniqueName: \"kubernetes.io/projected/8532f94b-ef23-4b4d-967f-dc2aa9e3c3dd-kube-api-access-sdjwd\") pod \"payload-pre-processing-e2e-shared-b-w4-0da668-dff997c59-8nrck\" (UID: \"8532f94b-ef23-4b4d-967f-dc2aa9e3c3dd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-0da668-dff997c59-8nrck" Oct 31 00:43:41.204835 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:41.204808 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfz9z\"" Oct 31 00:43:41.234663 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:41.234627 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb5fa2a8-608d-4449-af5f-ab87a659ac07-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-f0651cac-k697q\" (UID: \"bb5fa2a8-608d-4449-af5f-ab87a659ac07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f0651cac-k697q" Oct 31 00:43:41.234835 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:41.234759 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/958dc2bf-130f-4861-981b-c5fb31d39f67-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-0da668-78b845796f-5srmm\" (UID: \"958dc2bf-130f-4861-981b-c5fb31d39f67\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-0da668-78b845796f-5srmm" Oct 31 00:43:41.234883 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:41.234833 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9r6n\" (UniqueName: \"kubernetes.io/projected/bb5fa2a8-608d-4449-af5f-ab87a659ac07-kube-api-access-s9r6n\") pod \"maas-api-revoke-keys-e2e-mt-f0651cac-k697q\" (UID: \"bb5fa2a8-608d-4449-af5f-ab87a659ac07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f0651cac-k697q" Oct 31 00:43:41.234984 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:41.234963 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljk86\" (UniqueName: \"kubernetes.io/projected/958dc2bf-130f-4861-981b-c5fb31d39f67-kube-api-access-ljk86\") pod \"payload-processing-e2e-shared-b-w4-0da668-78b845796f-5srmm\" (UID: \"958dc2bf-130f-4861-981b-c5fb31d39f67\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-0da668-78b845796f-5srmm" Oct 31 00:43:41.235159 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:41.235136 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb5fa2a8-608d-4449-af5f-ab87a659ac07-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-f0651cac-k697q\" (UID: \"bb5fa2a8-608d-4449-af5f-ab87a659ac07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f0651cac-k697q" Oct 31 00:43:41.248755 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:41.248718 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:43:41.336885 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:41.336795 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb5fa2a8-608d-4449-af5f-ab87a659ac07-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-f0651cac-k697q\" (UID: \"bb5fa2a8-608d-4449-af5f-ab87a659ac07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f0651cac-k697q" Oct 31 00:43:41.336885 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:41.336853 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb5fa2a8-608d-4449-af5f-ab87a659ac07-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-f0651cac-k697q\" (UID: \"bb5fa2a8-608d-4449-af5f-ab87a659ac07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f0651cac-k697q" Oct 31 00:43:41.337138 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:41.336954 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/958dc2bf-130f-4861-981b-c5fb31d39f67-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-0da668-78b845796f-5srmm\" (UID: \"958dc2bf-130f-4861-981b-c5fb31d39f67\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-0da668-78b845796f-5srmm" Oct 31 00:43:41.337138 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:41.337029 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9r6n\" (UniqueName: \"kubernetes.io/projected/bb5fa2a8-608d-4449-af5f-ab87a659ac07-kube-api-access-s9r6n\") pod \"maas-api-revoke-keys-e2e-mt-f0651cac-k697q\" (UID: \"bb5fa2a8-608d-4449-af5f-ab87a659ac07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f0651cac-k697q" Oct 31 00:43:41.337241 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:41.337150 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljk86\" (UniqueName: \"kubernetes.io/projected/958dc2bf-130f-4861-981b-c5fb31d39f67-kube-api-access-ljk86\") pod \"payload-processing-e2e-shared-b-w4-0da668-78b845796f-5srmm\" (UID: \"958dc2bf-130f-4861-981b-c5fb31d39f67\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-0da668-78b845796f-5srmm" Oct 31 00:43:41.339018 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:41.338986 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/958dc2bf-130f-4861-981b-c5fb31d39f67-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-0da668-78b845796f-5srmm\" (UID: \"958dc2bf-130f-4861-981b-c5fb31d39f67\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-0da668-78b845796f-5srmm" Oct 31 00:43:41.339745 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:41.339716 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb5fa2a8-608d-4449-af5f-ab87a659ac07-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-f0651cac-k697q\" (UID: \"bb5fa2a8-608d-4449-af5f-ab87a659ac07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f0651cac-k697q" Oct 31 00:43:41.346405 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:41.341026 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb5fa2a8-608d-4449-af5f-ab87a659ac07-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-f0651cac-k697q\" (UID: \"bb5fa2a8-608d-4449-af5f-ab87a659ac07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f0651cac-k697q" Oct 31 00:43:41.401833 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:41.401800 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljk86\" (UniqueName: \"kubernetes.io/projected/958dc2bf-130f-4861-981b-c5fb31d39f67-kube-api-access-ljk86\") pod \"payload-processing-e2e-shared-b-w4-0da668-78b845796f-5srmm\" (UID: \"958dc2bf-130f-4861-981b-c5fb31d39f67\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-0da668-78b845796f-5srmm" Oct 31 00:43:41.403325 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:41.403295 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9r6n\" (UniqueName: \"kubernetes.io/projected/bb5fa2a8-608d-4449-af5f-ab87a659ac07-kube-api-access-s9r6n\") pod \"maas-api-revoke-keys-e2e-mt-f0651cac-k697q\" (UID: \"bb5fa2a8-608d-4449-af5f-ab87a659ac07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f0651cac-k697q" Oct 31 00:43:41.419453 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:41.419414 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-0da668-78b845796f-5srmm" Oct 31 00:43:41.438579 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:41.438548 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-0da668-dff997c59-8nrck" Oct 31 00:43:41.466324 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:41.466294 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f0651cac-k697q" Oct 31 00:43:42.038596 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:42.038558 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-0da668-78b845796f-5srmm"] Oct 31 00:43:42.045672 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:43:42.045637 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod958dc2bf_130f_4861_981b_c5fb31d39f67.slice/crio-a581d3a324ac3b302c300889fd23f2c182236965363ba7f751eb3d54f9698dab WatchSource:0}: Error finding container a581d3a324ac3b302c300889fd23f2c182236965363ba7f751eb3d54f9698dab: Status 404 returned error can't find the container with id a581d3a324ac3b302c300889fd23f2c182236965363ba7f751eb3d54f9698dab Oct 31 00:43:42.064926 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:42.064884 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-0da668-dff997c59-8nrck"] Oct 31 00:43:42.154508 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:42.150220 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f0651cac-k697q"] Oct 31 00:43:42.239134 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:42.239100 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-79tk8"] Oct 31 00:43:42.286013 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:42.285978 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-0da668-78b845796f-5srmm" event={"ID":"958dc2bf-130f-4861-981b-c5fb31d39f67","Type":"ContainerStarted","Data":"a05915bea71c90a673f33bdcac902414d31334ce5369c6a55ad4ab7dbf783340"} Oct 31 00:43:42.286215 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:42.286044 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-0da668-78b845796f-5srmm" event={"ID":"958dc2bf-130f-4861-981b-c5fb31d39f67","Type":"ContainerStarted","Data":"a581d3a324ac3b302c300889fd23f2c182236965363ba7f751eb3d54f9698dab"} Oct 31 00:43:42.286215 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:42.286068 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-0da668-78b845796f-5srmm" Oct 31 00:43:42.287578 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:42.287552 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f0651cac-k697q" event={"ID":"bb5fa2a8-608d-4449-af5f-ab87a659ac07","Type":"ContainerStarted","Data":"7e1514703b3cb00ac1426d51338097a568d240e0dd1ccfa64b2aba57322e6988"} Oct 31 00:43:42.289338 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:42.289268 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-0da668-dff997c59-8nrck" event={"ID":"8532f94b-ef23-4b4d-967f-dc2aa9e3c3dd","Type":"ContainerStarted","Data":"4b15be256121d29b82d6b5480b34faa7485c400e99e849b86eeb73ad610a1ffa"} Oct 31 00:43:42.289338 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:42.289319 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-0da668-dff997c59-8nrck" event={"ID":"8532f94b-ef23-4b4d-967f-dc2aa9e3c3dd","Type":"ContainerStarted","Data":"fbb4834e664e93ee81f780fa8b1539d8dd0adfaa8eddd5609f535649ac9eac51"} Oct 31 00:43:42.289540 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:42.289362 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-0da668-dff997c59-8nrck" Oct 31 00:43:42.426018 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:42.425955 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-0da668-78b845796f-5srmm" podStartSLOduration=2.42593545 podStartE2EDuration="2.42593545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:43:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:43:42.422119339 +0000 UTC m=+1714.893289087" watchObservedRunningTime="2026-10-31 00:43:42.42593545 +0000 UTC m=+1714.897105200" Oct 31 00:43:42.599036 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:42.598925 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-0da668-dff997c59-8nrck" podStartSLOduration=2.598900957 podStartE2EDuration="2.598900957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:43:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:43:42.560192534 +0000 UTC m=+1715.031362285" watchObservedRunningTime="2026-10-31 00:43:42.598900957 +0000 UTC m=+1715.070070708" Oct 31 00:43:43.299106 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:43.299028 5222 generic.go:356] "Generic (PLEG): container finished" podID="bb5fa2a8-608d-4449-af5f-ab87a659ac07" containerID="862386579a94025313386c22d5180c95b20944fd4836a10df96002e8f86ffa8d" exitCode=0 Oct 31 00:43:43.299473 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:43.299107 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f0651cac-k697q" event={"ID":"bb5fa2a8-608d-4449-af5f-ab87a659ac07","Type":"ContainerDied","Data":"862386579a94025313386c22d5180c95b20944fd4836a10df96002e8f86ffa8d"} Oct 31 00:43:44.654230 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:44.654203 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f0651cac-k697q" Oct 31 00:43:44.749357 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:44.749321 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb5fa2a8-608d-4449-af5f-ab87a659ac07-kube-api-access-s9r6n\" (UniqueName: \"kubernetes.io/projected/bb5fa2a8-608d-4449-af5f-ab87a659ac07-kube-api-access-s9r6n\") pod \"bb5fa2a8-608d-4449-af5f-ab87a659ac07\" (UID: \"bb5fa2a8-608d-4449-af5f-ab87a659ac07\") " Oct 31 00:43:44.749547 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:44.749437 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb5fa2a8-608d-4449-af5f-ab87a659ac07-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb5fa2a8-608d-4449-af5f-ab87a659ac07-maas-api-openshift-ca\") pod \"bb5fa2a8-608d-4449-af5f-ab87a659ac07\" (UID: \"bb5fa2a8-608d-4449-af5f-ab87a659ac07\") " Oct 31 00:43:44.749605 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:44.749592 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb5fa2a8-608d-4449-af5f-ab87a659ac07-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb5fa2a8-608d-4449-af5f-ab87a659ac07-maas-api-xks-ca\") pod \"bb5fa2a8-608d-4449-af5f-ab87a659ac07\" (UID: \"bb5fa2a8-608d-4449-af5f-ab87a659ac07\") " Oct 31 00:43:44.749800 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:44.749776 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb5fa2a8-608d-4449-af5f-ab87a659ac07-maas-api-openshift-ca" pod "bb5fa2a8-608d-4449-af5f-ab87a659ac07" (UID: "bb5fa2a8-608d-4449-af5f-ab87a659ac07"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:43:44.750314 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:44.750291 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb5fa2a8-608d-4449-af5f-ab87a659ac07-maas-api-openshift-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:44.751753 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:44.751726 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb5fa2a8-608d-4449-af5f-ab87a659ac07-kube-api-access-s9r6n" pod "bb5fa2a8-608d-4449-af5f-ab87a659ac07" (UID: "bb5fa2a8-608d-4449-af5f-ab87a659ac07"). InnerVolumeSpecName "kube-api-access-s9r6n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:43:44.751858 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:44.751783 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb5fa2a8-608d-4449-af5f-ab87a659ac07-maas-api-xks-ca" pod "bb5fa2a8-608d-4449-af5f-ab87a659ac07" (UID: "bb5fa2a8-608d-4449-af5f-ab87a659ac07"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:43:44.852140 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:44.852103 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9r6n\" (UniqueName: \"kubernetes.io/projected/bb5fa2a8-608d-4449-af5f-ab87a659ac07-kube-api-access-s9r6n\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:44.852140 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:44.852135 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb5fa2a8-608d-4449-af5f-ab87a659ac07-maas-api-xks-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:45.362354 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:45.362322 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f0651cac-k697q" Oct 31 00:43:45.362549 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:45.362356 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f0651cac-k697q" event={"ID":"bb5fa2a8-608d-4449-af5f-ab87a659ac07","Type":"ContainerDied","Data":"7e1514703b3cb00ac1426d51338097a568d240e0dd1ccfa64b2aba57322e6988"} Oct 31 00:43:45.362549 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:45.362423 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7e1514703b3cb00ac1426d51338097a568d240e0dd1ccfa64b2aba57322e6988" Oct 31 00:43:47.969398 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:47.968618 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-e030e0-5686f5c779-sx2sx"] Oct 31 00:43:47.969890 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:47.969677 5222 cpu_manager.go:414] "containerMap: removing container" podUID="bb5fa2a8-608d-4449-af5f-ab87a659ac07" containerName="revoke-keys" Oct 31 00:43:47.969890 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:47.969698 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb5fa2a8-608d-4449-af5f-ab87a659ac07" containerName="revoke-keys" Oct 31 00:43:47.969890 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:47.969855 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb5fa2a8-608d-4449-af5f-ab87a659ac07" containerName="revoke-keys" Oct 31 00:43:47.987613 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:47.987580 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-e030e0-5686f5c779-sx2sx" Oct 31 00:43:48.012686 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:48.012652 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-e030e0-5686f5c779-sx2sx"] Oct 31 00:43:48.040954 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:48.040298 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60ee69ce-043d-4529-b9be-7a36cd7468ca-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-e030e0-5686f5c779-sx2sx\" (UID: \"60ee69ce-043d-4529-b9be-7a36cd7468ca\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-e030e0-5686f5c779-sx2sx" Oct 31 00:43:48.040954 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:48.040531 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmfks\" (UniqueName: \"kubernetes.io/projected/60ee69ce-043d-4529-b9be-7a36cd7468ca-kube-api-access-gmfks\") pod \"payload-pre-processing-e2e-shared-a-w6-e030e0-5686f5c779-sx2sx\" (UID: \"60ee69ce-043d-4529-b9be-7a36cd7468ca\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-e030e0-5686f5c779-sx2sx" Oct 31 00:43:48.055391 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:48.054675 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w6-e030e0-dockercfg-6gwx5\"" Oct 31 00:43:48.055391 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:48.054751 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w6-e030e0\"" Oct 31 00:43:48.211411 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:48.205458 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-e030e0-57f56bb655-wfdmp"] Oct 31 00:43:48.213964 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:48.213933 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60ee69ce-043d-4529-b9be-7a36cd7468ca-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-e030e0-5686f5c779-sx2sx\" (UID: \"60ee69ce-043d-4529-b9be-7a36cd7468ca\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-e030e0-5686f5c779-sx2sx" Oct 31 00:43:48.214138 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:48.214049 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmfks\" (UniqueName: \"kubernetes.io/projected/60ee69ce-043d-4529-b9be-7a36cd7468ca-kube-api-access-gmfks\") pod \"payload-pre-processing-e2e-shared-a-w6-e030e0-5686f5c779-sx2sx\" (UID: \"60ee69ce-043d-4529-b9be-7a36cd7468ca\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-e030e0-5686f5c779-sx2sx" Oct 31 00:43:48.215277 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:48.215252 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60ee69ce-043d-4529-b9be-7a36cd7468ca-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-e030e0-5686f5c779-sx2sx\" (UID: \"60ee69ce-043d-4529-b9be-7a36cd7468ca\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-e030e0-5686f5c779-sx2sx" Oct 31 00:43:48.224098 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:48.224016 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-e030e0-57f56bb655-wfdmp" Oct 31 00:43:48.235655 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:48.235354 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-e030e0-57f56bb655-wfdmp"] Oct 31 00:43:48.283250 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:48.283217 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmfks\" (UniqueName: \"kubernetes.io/projected/60ee69ce-043d-4529-b9be-7a36cd7468ca-kube-api-access-gmfks\") pod \"payload-pre-processing-e2e-shared-a-w6-e030e0-5686f5c779-sx2sx\" (UID: \"60ee69ce-043d-4529-b9be-7a36cd7468ca\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-e030e0-5686f5c779-sx2sx" Oct 31 00:43:48.315527 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:48.315491 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prtg2\" (UniqueName: \"kubernetes.io/projected/c6fa3226-46ba-4b95-a4c4-bdba6073277c-kube-api-access-prtg2\") pod \"payload-processing-e2e-shared-a-w6-e030e0-57f56bb655-wfdmp\" (UID: \"c6fa3226-46ba-4b95-a4c4-bdba6073277c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-e030e0-57f56bb655-wfdmp" Oct 31 00:43:48.315790 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:48.315769 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6fa3226-46ba-4b95-a4c4-bdba6073277c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-e030e0-57f56bb655-wfdmp\" (UID: \"c6fa3226-46ba-4b95-a4c4-bdba6073277c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-e030e0-57f56bb655-wfdmp" Oct 31 00:43:48.342266 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:48.342227 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-e030e0-5686f5c779-sx2sx" Oct 31 00:43:48.424007 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:48.423967 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prtg2\" (UniqueName: \"kubernetes.io/projected/c6fa3226-46ba-4b95-a4c4-bdba6073277c-kube-api-access-prtg2\") pod \"payload-processing-e2e-shared-a-w6-e030e0-57f56bb655-wfdmp\" (UID: \"c6fa3226-46ba-4b95-a4c4-bdba6073277c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-e030e0-57f56bb655-wfdmp" Oct 31 00:43:48.424235 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:48.424215 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6fa3226-46ba-4b95-a4c4-bdba6073277c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-e030e0-57f56bb655-wfdmp\" (UID: \"c6fa3226-46ba-4b95-a4c4-bdba6073277c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-e030e0-57f56bb655-wfdmp" Oct 31 00:43:48.425339 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:48.425310 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6fa3226-46ba-4b95-a4c4-bdba6073277c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-e030e0-57f56bb655-wfdmp\" (UID: \"c6fa3226-46ba-4b95-a4c4-bdba6073277c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-e030e0-57f56bb655-wfdmp" Oct 31 00:43:48.487855 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:48.487780 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prtg2\" (UniqueName: \"kubernetes.io/projected/c6fa3226-46ba-4b95-a4c4-bdba6073277c-kube-api-access-prtg2\") pod \"payload-processing-e2e-shared-a-w6-e030e0-57f56bb655-wfdmp\" (UID: \"c6fa3226-46ba-4b95-a4c4-bdba6073277c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-e030e0-57f56bb655-wfdmp" Oct 31 00:43:48.547211 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:48.547181 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-e030e0-57f56bb655-wfdmp" Oct 31 00:43:48.731324 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:43:48.731295 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60ee69ce_043d_4529_b9be_7a36cd7468ca.slice/crio-db72535b020a8b2e6fe8639a7bd0cd9a5829d593fa9c5350b86f8514423e3049 WatchSource:0}: Error finding container db72535b020a8b2e6fe8639a7bd0cd9a5829d593fa9c5350b86f8514423e3049: Status 404 returned error can't find the container with id db72535b020a8b2e6fe8639a7bd0cd9a5829d593fa9c5350b86f8514423e3049 Oct 31 00:43:48.733505 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:48.733478 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-e030e0-5686f5c779-sx2sx"] Oct 31 00:43:48.926985 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:43:48.926954 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6fa3226_46ba_4b95_a4c4_bdba6073277c.slice/crio-0f0078002388c1704b31f05dca9e07049eec892873c7a05288b4f1b33a6d5807 WatchSource:0}: Error finding container 0f0078002388c1704b31f05dca9e07049eec892873c7a05288b4f1b33a6d5807: Status 404 returned error can't find the container with id 0f0078002388c1704b31f05dca9e07049eec892873c7a05288b4f1b33a6d5807 Oct 31 00:43:48.931050 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:48.931026 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-e030e0-57f56bb655-wfdmp"] Oct 31 00:43:49.399103 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:49.399067 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-e030e0-5686f5c779-sx2sx" event={"ID":"60ee69ce-043d-4529-b9be-7a36cd7468ca","Type":"ContainerStarted","Data":"0289592cd71f72ab252d00b52aca1bcb06f9d0c043f2672bb14f21f81581b17a"} Oct 31 00:43:49.399635 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:49.399127 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-e030e0-5686f5c779-sx2sx" event={"ID":"60ee69ce-043d-4529-b9be-7a36cd7468ca","Type":"ContainerStarted","Data":"db72535b020a8b2e6fe8639a7bd0cd9a5829d593fa9c5350b86f8514423e3049"} Oct 31 00:43:49.399635 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:49.399200 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-e030e0-5686f5c779-sx2sx" Oct 31 00:43:49.400984 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:49.400956 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-e030e0-57f56bb655-wfdmp" event={"ID":"c6fa3226-46ba-4b95-a4c4-bdba6073277c","Type":"ContainerStarted","Data":"724027af61a844b561486b6eedbc57fac3ec9035ac2f0a27d03227eddbb1fb51"} Oct 31 00:43:49.401123 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:49.401010 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-e030e0-57f56bb655-wfdmp" Oct 31 00:43:49.401123 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:49.401027 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-e030e0-57f56bb655-wfdmp" event={"ID":"c6fa3226-46ba-4b95-a4c4-bdba6073277c","Type":"ContainerStarted","Data":"0f0078002388c1704b31f05dca9e07049eec892873c7a05288b4f1b33a6d5807"} Oct 31 00:43:49.522398 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:49.522332 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-e030e0-5686f5c779-sx2sx" podStartSLOduration=2.522318835 podStartE2EDuration="2.522318835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:43:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:43:49.519503587 +0000 UTC m=+1721.990673338" watchObservedRunningTime="2026-10-31 00:43:49.522318835 +0000 UTC m=+1721.993488605" Oct 31 00:43:49.706547 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:49.703099 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-e030e0-57f56bb655-wfdmp" podStartSLOduration=1.703083729 podStartE2EDuration="1.703083729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:43:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:43:49.70277911 +0000 UTC m=+1722.173948858" watchObservedRunningTime="2026-10-31 00:43:49.703083729 +0000 UTC m=+1722.174253475" Oct 31 00:43:53.302571 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:53.302542 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-0da668-dff997c59-8nrck" Oct 31 00:43:53.303060 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:43:53.302616 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-0da668-78b845796f-5srmm" Oct 31 00:44:00.412603 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:00.412575 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-e030e0-5686f5c779-sx2sx" Oct 31 00:44:00.413004 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:00.412651 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-e030e0-57f56bb655-wfdmp" Oct 31 00:44:01.405853 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:01.405823 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-hx9qr"] Oct 31 00:44:01.406090 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:01.406038 5222 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-hx9qr" podUID="267e2bb7-9215-4fc3-9874-ae4c2ad13749" containerName="istio-proxy" containerID="cri-o://e284b72bce31ba8bf8d2c22f1ea971f1ae0e704f32a8a59ccc0da90903bd74aa" gracePeriod=30 Oct 31 00:44:05.892207 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:44:05.892166 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 00:44:06.544125 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:06.544090 5222 generic.go:356] "Generic (PLEG): container finished" podID="267e2bb7-9215-4fc3-9874-ae4c2ad13749" containerID="e284b72bce31ba8bf8d2c22f1ea971f1ae0e704f32a8a59ccc0da90903bd74aa" exitCode=0 Oct 31 00:44:06.544320 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:06.544138 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-hx9qr" event={"ID":"267e2bb7-9215-4fc3-9874-ae4c2ad13749","Type":"ContainerDied","Data":"e284b72bce31ba8bf8d2c22f1ea971f1ae0e704f32a8a59ccc0da90903bd74aa"} Oct 31 00:44:06.692382 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:06.692298 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-hx9qr" Oct 31 00:44:06.891825 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:06.891786 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/267e2bb7-9215-4fc3-9874-ae4c2ad13749-kube-api-access-74wrk\" (UniqueName: \"kubernetes.io/projected/267e2bb7-9215-4fc3-9874-ae4c2ad13749-kube-api-access-74wrk\") pod \"267e2bb7-9215-4fc3-9874-ae4c2ad13749\" (UID: \"267e2bb7-9215-4fc3-9874-ae4c2ad13749\") " Oct 31 00:44:06.892023 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:06.891884 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/267e2bb7-9215-4fc3-9874-ae4c2ad13749-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/267e2bb7-9215-4fc3-9874-ae4c2ad13749-istio-envoy\") pod \"267e2bb7-9215-4fc3-9874-ae4c2ad13749\" (UID: \"267e2bb7-9215-4fc3-9874-ae4c2ad13749\") " Oct 31 00:44:06.892023 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:06.891963 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/267e2bb7-9215-4fc3-9874-ae4c2ad13749-istio-token\" (UniqueName: \"kubernetes.io/projected/267e2bb7-9215-4fc3-9874-ae4c2ad13749-istio-token\") pod \"267e2bb7-9215-4fc3-9874-ae4c2ad13749\" (UID: \"267e2bb7-9215-4fc3-9874-ae4c2ad13749\") " Oct 31 00:44:06.892147 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:06.892028 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/267e2bb7-9215-4fc3-9874-ae4c2ad13749-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/267e2bb7-9215-4fc3-9874-ae4c2ad13749-istio-podinfo\") pod \"267e2bb7-9215-4fc3-9874-ae4c2ad13749\" (UID: \"267e2bb7-9215-4fc3-9874-ae4c2ad13749\") " Oct 31 00:44:06.892218 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:06.892148 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/267e2bb7-9215-4fc3-9874-ae4c2ad13749-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/267e2bb7-9215-4fc3-9874-ae4c2ad13749-workload-certs\") pod \"267e2bb7-9215-4fc3-9874-ae4c2ad13749\" (UID: \"267e2bb7-9215-4fc3-9874-ae4c2ad13749\") " Oct 31 00:44:06.892218 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:06.892194 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/267e2bb7-9215-4fc3-9874-ae4c2ad13749-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/267e2bb7-9215-4fc3-9874-ae4c2ad13749-credential-socket\") pod \"267e2bb7-9215-4fc3-9874-ae4c2ad13749\" (UID: \"267e2bb7-9215-4fc3-9874-ae4c2ad13749\") " Oct 31 00:44:06.892863 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:06.892230 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/267e2bb7-9215-4fc3-9874-ae4c2ad13749-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/267e2bb7-9215-4fc3-9874-ae4c2ad13749-istio-data\") pod \"267e2bb7-9215-4fc3-9874-ae4c2ad13749\" (UID: \"267e2bb7-9215-4fc3-9874-ae4c2ad13749\") " Oct 31 00:44:06.892863 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:06.892315 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/267e2bb7-9215-4fc3-9874-ae4c2ad13749-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/267e2bb7-9215-4fc3-9874-ae4c2ad13749-workload-socket\") pod \"267e2bb7-9215-4fc3-9874-ae4c2ad13749\" (UID: \"267e2bb7-9215-4fc3-9874-ae4c2ad13749\") " Oct 31 00:44:06.892863 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:06.892348 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/267e2bb7-9215-4fc3-9874-ae4c2ad13749-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/267e2bb7-9215-4fc3-9874-ae4c2ad13749-istiod-ca-cert\") pod \"267e2bb7-9215-4fc3-9874-ae4c2ad13749\" (UID: \"267e2bb7-9215-4fc3-9874-ae4c2ad13749\") " Oct 31 00:44:06.893767 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:06.893711 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/267e2bb7-9215-4fc3-9874-ae4c2ad13749-credential-socket" pod "267e2bb7-9215-4fc3-9874-ae4c2ad13749" (UID: "267e2bb7-9215-4fc3-9874-ae4c2ad13749"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:44:06.894106 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:06.894072 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/267e2bb7-9215-4fc3-9874-ae4c2ad13749-istio-data" pod "267e2bb7-9215-4fc3-9874-ae4c2ad13749" (UID: "267e2bb7-9215-4fc3-9874-ae4c2ad13749"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:44:06.895104 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:06.894098 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/267e2bb7-9215-4fc3-9874-ae4c2ad13749-workload-socket" pod "267e2bb7-9215-4fc3-9874-ae4c2ad13749" (UID: "267e2bb7-9215-4fc3-9874-ae4c2ad13749"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:44:06.895104 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:06.894170 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/267e2bb7-9215-4fc3-9874-ae4c2ad13749-workload-certs" pod "267e2bb7-9215-4fc3-9874-ae4c2ad13749" (UID: "267e2bb7-9215-4fc3-9874-ae4c2ad13749"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:44:06.895104 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:06.894265 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/267e2bb7-9215-4fc3-9874-ae4c2ad13749-istiod-ca-cert" pod "267e2bb7-9215-4fc3-9874-ae4c2ad13749" (UID: "267e2bb7-9215-4fc3-9874-ae4c2ad13749"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:44:06.895866 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:06.895843 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/267e2bb7-9215-4fc3-9874-ae4c2ad13749-istio-token" pod "267e2bb7-9215-4fc3-9874-ae4c2ad13749" (UID: "267e2bb7-9215-4fc3-9874-ae4c2ad13749"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:44:06.895968 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:06.895883 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/267e2bb7-9215-4fc3-9874-ae4c2ad13749-kube-api-access-74wrk" pod "267e2bb7-9215-4fc3-9874-ae4c2ad13749" (UID: "267e2bb7-9215-4fc3-9874-ae4c2ad13749"). InnerVolumeSpecName "kube-api-access-74wrk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:44:06.899809 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:06.899783 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/267e2bb7-9215-4fc3-9874-ae4c2ad13749-istio-podinfo" pod "267e2bb7-9215-4fc3-9874-ae4c2ad13749" (UID: "267e2bb7-9215-4fc3-9874-ae4c2ad13749"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:44:06.899899 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:06.899780 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/267e2bb7-9215-4fc3-9874-ae4c2ad13749-istio-envoy" pod "267e2bb7-9215-4fc3-9874-ae4c2ad13749" (UID: "267e2bb7-9215-4fc3-9874-ae4c2ad13749"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:44:06.994445 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:06.994306 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74wrk\" (UniqueName: \"kubernetes.io/projected/267e2bb7-9215-4fc3-9874-ae4c2ad13749-kube-api-access-74wrk\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:06.994445 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:06.994343 5222 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/267e2bb7-9215-4fc3-9874-ae4c2ad13749-istio-envoy\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:06.994445 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:06.994353 5222 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/267e2bb7-9215-4fc3-9874-ae4c2ad13749-istio-token\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:06.994445 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:06.994364 5222 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/267e2bb7-9215-4fc3-9874-ae4c2ad13749-istio-podinfo\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:06.994445 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:06.994393 5222 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/267e2bb7-9215-4fc3-9874-ae4c2ad13749-workload-certs\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:06.994445 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:06.994405 5222 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/267e2bb7-9215-4fc3-9874-ae4c2ad13749-credential-socket\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:06.994445 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:06.994417 5222 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/267e2bb7-9215-4fc3-9874-ae4c2ad13749-istio-data\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:06.994445 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:06.994429 5222 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/267e2bb7-9215-4fc3-9874-ae4c2ad13749-workload-socket\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:06.994445 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:06.994445 5222 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/267e2bb7-9215-4fc3-9874-ae4c2ad13749-istiod-ca-cert\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:07.554289 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:07.554257 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-hx9qr" Oct 31 00:44:07.554289 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:07.554281 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-hx9qr" event={"ID":"267e2bb7-9215-4fc3-9874-ae4c2ad13749","Type":"ContainerDied","Data":"c790a5edfe1ec3b5f18c2fb3313d6fad6c15f92f05788c6edfd60f1d3f8d8ae7"} Oct 31 00:44:07.554548 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:07.554348 5222 scope.go:122] "RemoveContainer" containerID="e284b72bce31ba8bf8d2c22f1ea971f1ae0e704f32a8a59ccc0da90903bd74aa" Oct 31 00:44:07.781177 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:07.781137 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-hx9qr"] Oct 31 00:44:07.832859 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:07.832761 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-hx9qr"] Oct 31 00:44:08.775964 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:08.775928 5222 scope.go:122] "RemoveContainer" containerID="36bd3135f4f9451b799b97129e8b50b338c2777e89c27a7e136a1ded6d4ba621" Oct 31 00:44:08.786677 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:08.786655 5222 scope.go:122] "RemoveContainer" containerID="d6b07d7835a4fcba7670fc1965a39e3b60bdc3f5f3e3a1aaee9e20f044c002ae" Oct 31 00:44:08.797023 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:08.796999 5222 scope.go:122] "RemoveContainer" containerID="a06f6c61b4701cf00861c598f717ea535f5d6a779adb3c8dffee138fc1ab61de" Oct 31 00:44:08.807411 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:08.807358 5222 scope.go:122] "RemoveContainer" containerID="cfc69315fe615a4dcb3175d08c307bd3b167d4ea910e586086d97c115127a80f" Oct 31 00:44:08.818743 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:08.818722 5222 scope.go:122] "RemoveContainer" containerID="a67dba162cddb54c5ccac804e82cc87fb16b52f5b913436f43ea4b98d3b6ea54" Oct 31 00:44:08.828783 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:08.828761 5222 scope.go:122] "RemoveContainer" containerID="4a5c4c517c5fa77706975c311eeabacceca5ad190656a68e0e72594dad7cb990" Oct 31 00:44:08.839677 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:08.839657 5222 scope.go:122] "RemoveContainer" containerID="fb65a47e597744ff8b421cacceca6f8019eb67ae8a363e752994e6b2cd3c7785" Oct 31 00:44:08.849905 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:08.849882 5222 scope.go:122] "RemoveContainer" containerID="c49d2948aff3ebd3aa87e4514b97e61a6f8913329fdec514da1bd1fb7571a78e" Oct 31 00:44:08.860227 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:08.860204 5222 scope.go:122] "RemoveContainer" containerID="8e14a7ea384665198e5f727a580c5487d5c196db81faa6666cb0b4d1456f379d" Oct 31 00:44:08.870070 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:08.870050 5222 scope.go:122] "RemoveContainer" containerID="5004c0e7ca67d9557448cf97f326c352505544c489345e1f4f7e91c1923a41d6" Oct 31 00:44:08.880185 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:08.880160 5222 scope.go:122] "RemoveContainer" containerID="a8fe5cded79bc9d4cea137ede29bda6e43993c4e2b106cdecb3ef722cdb3cc36" Oct 31 00:44:08.892927 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:08.892904 5222 scope.go:122] "RemoveContainer" containerID="f2fcc2b99cd9c5ecf6dc4a9ebd38f41e1e1bf0b824e2e04c301a12a68f7ae33d" Oct 31 00:44:08.903443 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:08.903420 5222 scope.go:122] "RemoveContainer" containerID="cec70e4cc7b562990b22c53ee9670cd2f56c5b2796c0d696cfa0edd0b2bf5143" Oct 31 00:44:08.913895 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:08.913873 5222 scope.go:122] "RemoveContainer" containerID="57c2d1b4686ff56ee7d96686cc37a6f1dab92d848313f71470e5edf215d4e5d0" Oct 31 00:44:09.658986 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:09.658950 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="267e2bb7-9215-4fc3-9874-ae4c2ad13749" path="/var/lib/kubelet/pods/267e2bb7-9215-4fc3-9874-ae4c2ad13749/volumes" Oct 31 00:44:16.527574 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:16.521737 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-bcc9n"] Oct 31 00:44:16.527574 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:16.522008 5222 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-bcc9n" podUID="bc854b8d-c4e0-4355-a059-09b118cf87b7" containerName="istio-proxy" containerID="cri-o://cf4387b7f512f38cc2ea4e1e3e8a585a653bee5dc49022d7a1a9b40bb40e3895" gracePeriod=30 Oct 31 00:44:16.571405 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:16.569817 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-4hsbr"] Oct 31 00:44:16.571405 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:16.570210 5222 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-4hsbr" podUID="6825ee8a-8ef9-4d40-8046-3e5bd337b6e0" containerName="istio-proxy" containerID="cri-o://55271c1d80b8c03372e3b9481f2b215a19a47fbec4c26b0c808c432459732daf" gracePeriod=30 Oct 31 00:44:16.583871 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:16.583839 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vbgx5"] Oct 31 00:44:16.584116 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:16.584084 5222 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vbgx5" podUID="1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf" containerName="istio-proxy" containerID="cri-o://727faaa62ad980a28fcacfec163bf748ad91159ed70e9b234c57594c0ac17834" gracePeriod=30 Oct 31 00:44:16.601329 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:16.601298 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-2jrfg"] Oct 31 00:44:16.601527 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:16.601507 5222 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-2jrfg" podUID="2fe80038-c64b-4114-ac0f-337a29f20204" containerName="istio-proxy" containerID="cri-o://fb22614abc89ee4bce35a3f610f5394eac15c2e48293e5d248f3b217798685ca" gracePeriod=30 Oct 31 00:44:16.709846 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:16.709806 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-79tk8"] Oct 31 00:44:21.693404 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:21.691570 5222 generic.go:356] "Generic (PLEG): container finished" podID="bc854b8d-c4e0-4355-a059-09b118cf87b7" containerID="cf4387b7f512f38cc2ea4e1e3e8a585a653bee5dc49022d7a1a9b40bb40e3895" exitCode=0 Oct 31 00:44:21.693404 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:21.691657 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-bcc9n" event={"ID":"bc854b8d-c4e0-4355-a059-09b118cf87b7","Type":"ContainerDied","Data":"cf4387b7f512f38cc2ea4e1e3e8a585a653bee5dc49022d7a1a9b40bb40e3895"} Oct 31 00:44:21.972722 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:21.972689 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-4hsbr" Oct 31 00:44:22.001112 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.001069 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-bcc9n" Oct 31 00:44:22.032911 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.032875 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6825ee8a-8ef9-4d40-8046-3e5bd337b6e0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6825ee8a-8ef9-4d40-8046-3e5bd337b6e0-istio-data\") pod \"6825ee8a-8ef9-4d40-8046-3e5bd337b6e0\" (UID: \"6825ee8a-8ef9-4d40-8046-3e5bd337b6e0\") " Oct 31 00:44:22.033122 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.033021 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc854b8d-c4e0-4355-a059-09b118cf87b7-istio-token\" (UniqueName: \"kubernetes.io/projected/bc854b8d-c4e0-4355-a059-09b118cf87b7-istio-token\") pod \"bc854b8d-c4e0-4355-a059-09b118cf87b7\" (UID: \"bc854b8d-c4e0-4355-a059-09b118cf87b7\") " Oct 31 00:44:22.033122 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.033069 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc854b8d-c4e0-4355-a059-09b118cf87b7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc854b8d-c4e0-4355-a059-09b118cf87b7-istio-data\") pod \"bc854b8d-c4e0-4355-a059-09b118cf87b7\" (UID: \"bc854b8d-c4e0-4355-a059-09b118cf87b7\") " Oct 31 00:44:22.033328 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.033149 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6825ee8a-8ef9-4d40-8046-3e5bd337b6e0-kube-api-access-89td9\" (UniqueName: \"kubernetes.io/projected/6825ee8a-8ef9-4d40-8046-3e5bd337b6e0-kube-api-access-89td9\") pod \"6825ee8a-8ef9-4d40-8046-3e5bd337b6e0\" (UID: \"6825ee8a-8ef9-4d40-8046-3e5bd337b6e0\") " Oct 31 00:44:22.033328 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.033199 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc854b8d-c4e0-4355-a059-09b118cf87b7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc854b8d-c4e0-4355-a059-09b118cf87b7-workload-certs\") pod \"bc854b8d-c4e0-4355-a059-09b118cf87b7\" (UID: \"bc854b8d-c4e0-4355-a059-09b118cf87b7\") " Oct 31 00:44:22.033328 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.033261 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6825ee8a-8ef9-4d40-8046-3e5bd337b6e0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6825ee8a-8ef9-4d40-8046-3e5bd337b6e0-istio-envoy\") pod \"6825ee8a-8ef9-4d40-8046-3e5bd337b6e0\" (UID: \"6825ee8a-8ef9-4d40-8046-3e5bd337b6e0\") " Oct 31 00:44:22.033328 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.033290 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6825ee8a-8ef9-4d40-8046-3e5bd337b6e0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6825ee8a-8ef9-4d40-8046-3e5bd337b6e0-istiod-ca-cert\") pod \"6825ee8a-8ef9-4d40-8046-3e5bd337b6e0\" (UID: \"6825ee8a-8ef9-4d40-8046-3e5bd337b6e0\") " Oct 31 00:44:22.033328 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.033326 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc854b8d-c4e0-4355-a059-09b118cf87b7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc854b8d-c4e0-4355-a059-09b118cf87b7-credential-socket\") pod \"bc854b8d-c4e0-4355-a059-09b118cf87b7\" (UID: \"bc854b8d-c4e0-4355-a059-09b118cf87b7\") " Oct 31 00:44:22.033631 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.033419 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc854b8d-c4e0-4355-a059-09b118cf87b7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc854b8d-c4e0-4355-a059-09b118cf87b7-istio-envoy\") pod \"bc854b8d-c4e0-4355-a059-09b118cf87b7\" (UID: \"bc854b8d-c4e0-4355-a059-09b118cf87b7\") " Oct 31 00:44:22.033631 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.033463 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc854b8d-c4e0-4355-a059-09b118cf87b7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc854b8d-c4e0-4355-a059-09b118cf87b7-istiod-ca-cert\") pod \"bc854b8d-c4e0-4355-a059-09b118cf87b7\" (UID: \"bc854b8d-c4e0-4355-a059-09b118cf87b7\") " Oct 31 00:44:22.033631 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.033572 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6825ee8a-8ef9-4d40-8046-3e5bd337b6e0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6825ee8a-8ef9-4d40-8046-3e5bd337b6e0-workload-socket\") pod \"6825ee8a-8ef9-4d40-8046-3e5bd337b6e0\" (UID: \"6825ee8a-8ef9-4d40-8046-3e5bd337b6e0\") " Oct 31 00:44:22.033631 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.033607 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6825ee8a-8ef9-4d40-8046-3e5bd337b6e0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6825ee8a-8ef9-4d40-8046-3e5bd337b6e0-workload-certs\") pod \"6825ee8a-8ef9-4d40-8046-3e5bd337b6e0\" (UID: \"6825ee8a-8ef9-4d40-8046-3e5bd337b6e0\") " Oct 31 00:44:22.033842 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.033689 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6825ee8a-8ef9-4d40-8046-3e5bd337b6e0-istio-token\" (UniqueName: \"kubernetes.io/projected/6825ee8a-8ef9-4d40-8046-3e5bd337b6e0-istio-token\") pod \"6825ee8a-8ef9-4d40-8046-3e5bd337b6e0\" (UID: \"6825ee8a-8ef9-4d40-8046-3e5bd337b6e0\") " Oct 31 00:44:22.033842 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.033716 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc854b8d-c4e0-4355-a059-09b118cf87b7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc854b8d-c4e0-4355-a059-09b118cf87b7-workload-socket\") pod \"bc854b8d-c4e0-4355-a059-09b118cf87b7\" (UID: \"bc854b8d-c4e0-4355-a059-09b118cf87b7\") " Oct 31 00:44:22.033842 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.033753 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6825ee8a-8ef9-4d40-8046-3e5bd337b6e0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6825ee8a-8ef9-4d40-8046-3e5bd337b6e0-istio-podinfo\") pod \"6825ee8a-8ef9-4d40-8046-3e5bd337b6e0\" (UID: \"6825ee8a-8ef9-4d40-8046-3e5bd337b6e0\") " Oct 31 00:44:22.033842 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.033810 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6825ee8a-8ef9-4d40-8046-3e5bd337b6e0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6825ee8a-8ef9-4d40-8046-3e5bd337b6e0-credential-socket\") pod \"6825ee8a-8ef9-4d40-8046-3e5bd337b6e0\" (UID: \"6825ee8a-8ef9-4d40-8046-3e5bd337b6e0\") " Oct 31 00:44:22.034053 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.033840 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bc854b8d-c4e0-4355-a059-09b118cf87b7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc854b8d-c4e0-4355-a059-09b118cf87b7-istio-podinfo\") pod \"bc854b8d-c4e0-4355-a059-09b118cf87b7\" (UID: \"bc854b8d-c4e0-4355-a059-09b118cf87b7\") " Oct 31 00:44:22.034053 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.034007 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc854b8d-c4e0-4355-a059-09b118cf87b7-kube-api-access-6dz85\" (UniqueName: \"kubernetes.io/projected/bc854b8d-c4e0-4355-a059-09b118cf87b7-kube-api-access-6dz85\") pod \"bc854b8d-c4e0-4355-a059-09b118cf87b7\" (UID: \"bc854b8d-c4e0-4355-a059-09b118cf87b7\") " Oct 31 00:44:22.035940 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.035898 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6825ee8a-8ef9-4d40-8046-3e5bd337b6e0-workload-socket" pod "6825ee8a-8ef9-4d40-8046-3e5bd337b6e0" (UID: "6825ee8a-8ef9-4d40-8046-3e5bd337b6e0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:44:22.036726 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.036100 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc854b8d-c4e0-4355-a059-09b118cf87b7-workload-socket" pod "bc854b8d-c4e0-4355-a059-09b118cf87b7" (UID: "bc854b8d-c4e0-4355-a059-09b118cf87b7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:44:22.036726 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.036217 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6825ee8a-8ef9-4d40-8046-3e5bd337b6e0-istio-data" pod "6825ee8a-8ef9-4d40-8046-3e5bd337b6e0" (UID: "6825ee8a-8ef9-4d40-8046-3e5bd337b6e0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:44:22.036726 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.036319 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc854b8d-c4e0-4355-a059-09b118cf87b7-istio-data" pod "bc854b8d-c4e0-4355-a059-09b118cf87b7" (UID: "bc854b8d-c4e0-4355-a059-09b118cf87b7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:44:22.036726 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.036474 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc854b8d-c4e0-4355-a059-09b118cf87b7-istiod-ca-cert" pod "bc854b8d-c4e0-4355-a059-09b118cf87b7" (UID: "bc854b8d-c4e0-4355-a059-09b118cf87b7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:44:22.036726 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.036611 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc854b8d-c4e0-4355-a059-09b118cf87b7-credential-socket" pod "bc854b8d-c4e0-4355-a059-09b118cf87b7" (UID: "bc854b8d-c4e0-4355-a059-09b118cf87b7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:44:22.036726 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.036617 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6825ee8a-8ef9-4d40-8046-3e5bd337b6e0-istiod-ca-cert" pod "6825ee8a-8ef9-4d40-8046-3e5bd337b6e0" (UID: "6825ee8a-8ef9-4d40-8046-3e5bd337b6e0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:44:22.037270 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.036752 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc854b8d-c4e0-4355-a059-09b118cf87b7-workload-certs" pod "bc854b8d-c4e0-4355-a059-09b118cf87b7" (UID: "bc854b8d-c4e0-4355-a059-09b118cf87b7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:44:22.037270 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.036811 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6825ee8a-8ef9-4d40-8046-3e5bd337b6e0-credential-socket" pod "6825ee8a-8ef9-4d40-8046-3e5bd337b6e0" (UID: "6825ee8a-8ef9-4d40-8046-3e5bd337b6e0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:44:22.037270 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.036915 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6825ee8a-8ef9-4d40-8046-3e5bd337b6e0-workload-certs" pod "6825ee8a-8ef9-4d40-8046-3e5bd337b6e0" (UID: "6825ee8a-8ef9-4d40-8046-3e5bd337b6e0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:44:22.065447 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.065388 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc854b8d-c4e0-4355-a059-09b118cf87b7-istio-token" pod "bc854b8d-c4e0-4355-a059-09b118cf87b7" (UID: "bc854b8d-c4e0-4355-a059-09b118cf87b7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:44:22.066058 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.066013 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6825ee8a-8ef9-4d40-8046-3e5bd337b6e0-istio-token" pod "6825ee8a-8ef9-4d40-8046-3e5bd337b6e0" (UID: "6825ee8a-8ef9-4d40-8046-3e5bd337b6e0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:44:22.067434 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.067333 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bc854b8d-c4e0-4355-a059-09b118cf87b7-istio-podinfo" pod "bc854b8d-c4e0-4355-a059-09b118cf87b7" (UID: "bc854b8d-c4e0-4355-a059-09b118cf87b7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:44:22.068831 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.068808 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6825ee8a-8ef9-4d40-8046-3e5bd337b6e0-istio-podinfo" pod "6825ee8a-8ef9-4d40-8046-3e5bd337b6e0" (UID: "6825ee8a-8ef9-4d40-8046-3e5bd337b6e0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:44:22.069035 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.069018 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc854b8d-c4e0-4355-a059-09b118cf87b7-istio-envoy" pod "bc854b8d-c4e0-4355-a059-09b118cf87b7" (UID: "bc854b8d-c4e0-4355-a059-09b118cf87b7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:44:22.069141 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.069093 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6825ee8a-8ef9-4d40-8046-3e5bd337b6e0-istio-envoy" pod "6825ee8a-8ef9-4d40-8046-3e5bd337b6e0" (UID: "6825ee8a-8ef9-4d40-8046-3e5bd337b6e0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:44:22.069260 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.069232 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc854b8d-c4e0-4355-a059-09b118cf87b7-kube-api-access-6dz85" pod "bc854b8d-c4e0-4355-a059-09b118cf87b7" (UID: "bc854b8d-c4e0-4355-a059-09b118cf87b7"). InnerVolumeSpecName "kube-api-access-6dz85". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:44:22.071037 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.071004 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6825ee8a-8ef9-4d40-8046-3e5bd337b6e0-kube-api-access-89td9" pod "6825ee8a-8ef9-4d40-8046-3e5bd337b6e0" (UID: "6825ee8a-8ef9-4d40-8046-3e5bd337b6e0"). InnerVolumeSpecName "kube-api-access-89td9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:44:22.079346 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.079284 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vbgx5" Oct 31 00:44:22.081527 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.081504 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-2jrfg" Oct 31 00:44:22.136666 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.136629 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fe80038-c64b-4114-ac0f-337a29f20204-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fe80038-c64b-4114-ac0f-337a29f20204-workload-socket\") pod \"2fe80038-c64b-4114-ac0f-337a29f20204\" (UID: \"2fe80038-c64b-4114-ac0f-337a29f20204\") " Oct 31 00:44:22.136666 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.136669 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fe80038-c64b-4114-ac0f-337a29f20204-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fe80038-c64b-4114-ac0f-337a29f20204-istio-data\") pod \"2fe80038-c64b-4114-ac0f-337a29f20204\" (UID: \"2fe80038-c64b-4114-ac0f-337a29f20204\") " Oct 31 00:44:22.136899 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.136708 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf-istio-data\") pod \"1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf\" (UID: \"1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf\") " Oct 31 00:44:22.136899 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.136755 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf-workload-socket\") pod \"1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf\" (UID: \"1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf\") " Oct 31 00:44:22.136899 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.136817 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fe80038-c64b-4114-ac0f-337a29f20204-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fe80038-c64b-4114-ac0f-337a29f20204-workload-certs\") pod \"2fe80038-c64b-4114-ac0f-337a29f20204\" (UID: \"2fe80038-c64b-4114-ac0f-337a29f20204\") " Oct 31 00:44:22.136899 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.136854 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf-istiod-ca-cert\") pod \"1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf\" (UID: \"1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf\") " Oct 31 00:44:22.136899 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.136892 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf-istio-token\" (UniqueName: \"kubernetes.io/projected/1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf-istio-token\") pod \"1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf\" (UID: \"1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf\") " Oct 31 00:44:22.137122 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.136923 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fe80038-c64b-4114-ac0f-337a29f20204-istio-data" pod "2fe80038-c64b-4114-ac0f-337a29f20204" (UID: "2fe80038-c64b-4114-ac0f-337a29f20204"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:44:22.137122 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.136941 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fe80038-c64b-4114-ac0f-337a29f20204-workload-socket" pod "2fe80038-c64b-4114-ac0f-337a29f20204" (UID: "2fe80038-c64b-4114-ac0f-337a29f20204"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:44:22.137122 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.136980 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fe80038-c64b-4114-ac0f-337a29f20204-istio-token\" (UniqueName: \"kubernetes.io/projected/2fe80038-c64b-4114-ac0f-337a29f20204-istio-token\") pod \"2fe80038-c64b-4114-ac0f-337a29f20204\" (UID: \"2fe80038-c64b-4114-ac0f-337a29f20204\") " Oct 31 00:44:22.137122 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.137020 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf-istio-podinfo\") pod \"1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf\" (UID: \"1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf\") " Oct 31 00:44:22.137122 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.137064 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fe80038-c64b-4114-ac0f-337a29f20204-workload-certs" pod "2fe80038-c64b-4114-ac0f-337a29f20204" (UID: "2fe80038-c64b-4114-ac0f-337a29f20204"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:44:22.137122 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.137079 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fe80038-c64b-4114-ac0f-337a29f20204-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fe80038-c64b-4114-ac0f-337a29f20204-istio-envoy\") pod \"2fe80038-c64b-4114-ac0f-337a29f20204\" (UID: \"2fe80038-c64b-4114-ac0f-337a29f20204\") " Oct 31 00:44:22.137122 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.137109 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf-workload-certs\") pod \"1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf\" (UID: \"1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf\") " Oct 31 00:44:22.137517 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.137136 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fe80038-c64b-4114-ac0f-337a29f20204-kube-api-access-9mrhl\" (UniqueName: \"kubernetes.io/projected/2fe80038-c64b-4114-ac0f-337a29f20204-kube-api-access-9mrhl\") pod \"2fe80038-c64b-4114-ac0f-337a29f20204\" (UID: \"2fe80038-c64b-4114-ac0f-337a29f20204\") " Oct 31 00:44:22.137517 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.137198 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf-kube-api-access-qpp4l\" (UniqueName: \"kubernetes.io/projected/1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf-kube-api-access-qpp4l\") pod \"1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf\" (UID: \"1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf\") " Oct 31 00:44:22.137517 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.137263 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf-workload-socket" pod "1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf" (UID: "1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:44:22.137517 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.137265 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf-credential-socket\") pod \"1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf\" (UID: \"1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf\") " Oct 31 00:44:22.137517 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.137323 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf-istio-envoy\") pod \"1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf\" (UID: \"1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf\") " Oct 31 00:44:22.137517 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.137333 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf-istiod-ca-cert" pod "1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf" (UID: "1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:44:22.137517 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.137421 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fe80038-c64b-4114-ac0f-337a29f20204-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fe80038-c64b-4114-ac0f-337a29f20204-credential-socket\") pod \"2fe80038-c64b-4114-ac0f-337a29f20204\" (UID: \"2fe80038-c64b-4114-ac0f-337a29f20204\") " Oct 31 00:44:22.137517 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.137439 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf-istio-data" pod "1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf" (UID: "1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:44:22.137517 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.137448 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fe80038-c64b-4114-ac0f-337a29f20204-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fe80038-c64b-4114-ac0f-337a29f20204-istiod-ca-cert\") pod \"2fe80038-c64b-4114-ac0f-337a29f20204\" (UID: \"2fe80038-c64b-4114-ac0f-337a29f20204\") " Oct 31 00:44:22.137517 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.137500 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2fe80038-c64b-4114-ac0f-337a29f20204-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fe80038-c64b-4114-ac0f-337a29f20204-istio-podinfo\") pod \"2fe80038-c64b-4114-ac0f-337a29f20204\" (UID: \"2fe80038-c64b-4114-ac0f-337a29f20204\") " Oct 31 00:44:22.167473 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.138178 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dz85\" (UniqueName: \"kubernetes.io/projected/bc854b8d-c4e0-4355-a059-09b118cf87b7-kube-api-access-6dz85\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:22.167473 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.138204 5222 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6825ee8a-8ef9-4d40-8046-3e5bd337b6e0-istio-data\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:22.167473 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.138219 5222 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc854b8d-c4e0-4355-a059-09b118cf87b7-istio-token\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:22.167473 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.138233 5222 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc854b8d-c4e0-4355-a059-09b118cf87b7-istio-data\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:22.167473 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.138247 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89td9\" (UniqueName: \"kubernetes.io/projected/6825ee8a-8ef9-4d40-8046-3e5bd337b6e0-kube-api-access-89td9\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:22.167473 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.138263 5222 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc854b8d-c4e0-4355-a059-09b118cf87b7-workload-certs\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:22.167473 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.138277 5222 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6825ee8a-8ef9-4d40-8046-3e5bd337b6e0-istio-envoy\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:22.167473 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.138291 5222 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6825ee8a-8ef9-4d40-8046-3e5bd337b6e0-istiod-ca-cert\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:22.167473 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.138305 5222 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fe80038-c64b-4114-ac0f-337a29f20204-workload-socket\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:22.167473 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.138318 5222 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fe80038-c64b-4114-ac0f-337a29f20204-istio-data\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:22.167473 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.138331 5222 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc854b8d-c4e0-4355-a059-09b118cf87b7-credential-socket\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:22.167473 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.138346 5222 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf-istio-data\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:22.167473 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.138360 5222 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf-workload-socket\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:22.167473 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.138393 5222 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc854b8d-c4e0-4355-a059-09b118cf87b7-istio-envoy\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:22.167473 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.138408 5222 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc854b8d-c4e0-4355-a059-09b118cf87b7-istiod-ca-cert\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:22.167473 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.138421 5222 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fe80038-c64b-4114-ac0f-337a29f20204-workload-certs\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:22.167473 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.138435 5222 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf-istiod-ca-cert\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:22.167473 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.138448 5222 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6825ee8a-8ef9-4d40-8046-3e5bd337b6e0-workload-socket\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:22.167473 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.138462 5222 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6825ee8a-8ef9-4d40-8046-3e5bd337b6e0-workload-certs\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:22.167473 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.138475 5222 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6825ee8a-8ef9-4d40-8046-3e5bd337b6e0-istio-token\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:22.167473 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.138488 5222 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc854b8d-c4e0-4355-a059-09b118cf87b7-workload-socket\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:22.167473 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.138502 5222 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6825ee8a-8ef9-4d40-8046-3e5bd337b6e0-istio-podinfo\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:22.167473 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.138515 5222 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6825ee8a-8ef9-4d40-8046-3e5bd337b6e0-credential-socket\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:22.167473 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.138529 5222 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc854b8d-c4e0-4355-a059-09b118cf87b7-istio-podinfo\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:22.167473 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.163459 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf-workload-certs" pod "1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf" (UID: "1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:44:22.167473 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.166525 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf-credential-socket" pod "1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf" (UID: "1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:44:22.170360 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.168729 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fe80038-c64b-4114-ac0f-337a29f20204-credential-socket" pod "2fe80038-c64b-4114-ac0f-337a29f20204" (UID: "2fe80038-c64b-4114-ac0f-337a29f20204"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:44:22.170360 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.169045 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fe80038-c64b-4114-ac0f-337a29f20204-istiod-ca-cert" pod "2fe80038-c64b-4114-ac0f-337a29f20204" (UID: "2fe80038-c64b-4114-ac0f-337a29f20204"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:44:22.170884 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.170855 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf-istio-envoy" pod "1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf" (UID: "1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:44:22.171073 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.171051 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf-kube-api-access-qpp4l" pod "1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf" (UID: "1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf"). InnerVolumeSpecName "kube-api-access-qpp4l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:44:22.174181 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.174147 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf-istio-podinfo" pod "1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf" (UID: "1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:44:22.174518 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.174460 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf-istio-token" pod "1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf" (UID: "1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:44:22.175125 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.175098 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2fe80038-c64b-4114-ac0f-337a29f20204-istio-podinfo" pod "2fe80038-c64b-4114-ac0f-337a29f20204" (UID: "2fe80038-c64b-4114-ac0f-337a29f20204"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:44:22.176229 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.175876 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fe80038-c64b-4114-ac0f-337a29f20204-istio-envoy" pod "2fe80038-c64b-4114-ac0f-337a29f20204" (UID: "2fe80038-c64b-4114-ac0f-337a29f20204"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:44:22.179949 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.179910 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fe80038-c64b-4114-ac0f-337a29f20204-kube-api-access-9mrhl" pod "2fe80038-c64b-4114-ac0f-337a29f20204" (UID: "2fe80038-c64b-4114-ac0f-337a29f20204"). InnerVolumeSpecName "kube-api-access-9mrhl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:44:22.183293 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.183257 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fe80038-c64b-4114-ac0f-337a29f20204-istio-token" pod "2fe80038-c64b-4114-ac0f-337a29f20204" (UID: "2fe80038-c64b-4114-ac0f-337a29f20204"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:44:22.263631 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.263583 5222 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fe80038-c64b-4114-ac0f-337a29f20204-istio-token\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:22.263631 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.263622 5222 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf-istio-podinfo\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:22.263631 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.263632 5222 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fe80038-c64b-4114-ac0f-337a29f20204-istio-envoy\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:22.263631 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.263641 5222 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf-workload-certs\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:22.263922 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.263650 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mrhl\" (UniqueName: \"kubernetes.io/projected/2fe80038-c64b-4114-ac0f-337a29f20204-kube-api-access-9mrhl\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:22.263922 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.263662 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpp4l\" (UniqueName: \"kubernetes.io/projected/1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf-kube-api-access-qpp4l\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:22.263922 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.263670 5222 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf-credential-socket\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:22.263922 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.263680 5222 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf-istio-envoy\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:22.263922 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.263688 5222 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fe80038-c64b-4114-ac0f-337a29f20204-credential-socket\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:22.263922 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.263695 5222 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fe80038-c64b-4114-ac0f-337a29f20204-istiod-ca-cert\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:22.263922 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.263703 5222 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fe80038-c64b-4114-ac0f-337a29f20204-istio-podinfo\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:22.263922 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.263711 5222 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf-istio-token\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:22.703426 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.701276 5222 generic.go:356] "Generic (PLEG): container finished" podID="2fe80038-c64b-4114-ac0f-337a29f20204" containerID="fb22614abc89ee4bce35a3f610f5394eac15c2e48293e5d248f3b217798685ca" exitCode=0 Oct 31 00:44:22.703426 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.701362 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-2jrfg" Oct 31 00:44:22.703426 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.701397 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-2jrfg" event={"ID":"2fe80038-c64b-4114-ac0f-337a29f20204","Type":"ContainerDied","Data":"fb22614abc89ee4bce35a3f610f5394eac15c2e48293e5d248f3b217798685ca"} Oct 31 00:44:22.703426 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.701460 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-2jrfg" event={"ID":"2fe80038-c64b-4114-ac0f-337a29f20204","Type":"ContainerDied","Data":"9fdc833f4f2eebc9497590c6056fd9220fba2864b1dd01d2a321360cb5a2ff4d"} Oct 31 00:44:22.703426 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.701486 5222 scope.go:122] "RemoveContainer" containerID="fb22614abc89ee4bce35a3f610f5394eac15c2e48293e5d248f3b217798685ca" Oct 31 00:44:22.704643 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.704614 5222 generic.go:356] "Generic (PLEG): container finished" podID="1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf" containerID="727faaa62ad980a28fcacfec163bf748ad91159ed70e9b234c57594c0ac17834" exitCode=0 Oct 31 00:44:22.704811 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.704789 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vbgx5" Oct 31 00:44:22.704885 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.704790 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vbgx5" event={"ID":"1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf","Type":"ContainerDied","Data":"727faaa62ad980a28fcacfec163bf748ad91159ed70e9b234c57594c0ac17834"} Oct 31 00:44:22.704939 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.704919 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vbgx5" event={"ID":"1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf","Type":"ContainerDied","Data":"f27a9f499160e5f6fbe8249ec1f098c339c392bcc0147c2dd3d042cb313c5877"} Oct 31 00:44:22.706766 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.706225 5222 generic.go:356] "Generic (PLEG): container finished" podID="6825ee8a-8ef9-4d40-8046-3e5bd337b6e0" containerID="55271c1d80b8c03372e3b9481f2b215a19a47fbec4c26b0c808c432459732daf" exitCode=0 Oct 31 00:44:22.706766 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.706269 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-4hsbr" Oct 31 00:44:22.706766 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.706291 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-4hsbr" event={"ID":"6825ee8a-8ef9-4d40-8046-3e5bd337b6e0","Type":"ContainerDied","Data":"55271c1d80b8c03372e3b9481f2b215a19a47fbec4c26b0c808c432459732daf"} Oct 31 00:44:22.706766 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.706318 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-4hsbr" event={"ID":"6825ee8a-8ef9-4d40-8046-3e5bd337b6e0","Type":"ContainerDied","Data":"f76da1cccd0053be48c1d8b9923abe51c6b80d0e727efa90eb8546b23e2a6527"} Oct 31 00:44:22.708995 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.708965 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-bcc9n" Oct 31 00:44:22.709106 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.708962 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-bcc9n" event={"ID":"bc854b8d-c4e0-4355-a059-09b118cf87b7","Type":"ContainerDied","Data":"8488e4de7253cf5b53d2a7d5990df7ec2cff16805cca4b980d18fbde1bb53e98"} Oct 31 00:44:22.717831 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.717811 5222 scope.go:122] "RemoveContainer" containerID="51103936220c641723e6faed9902d3b5829355f32f6ab65da69cb81f5503a3e1" Oct 31 00:44:22.730169 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.730145 5222 scope.go:122] "RemoveContainer" containerID="fb22614abc89ee4bce35a3f610f5394eac15c2e48293e5d248f3b217798685ca" Oct 31 00:44:22.730652 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:44:22.730537 5222 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb22614abc89ee4bce35a3f610f5394eac15c2e48293e5d248f3b217798685ca\": container with ID starting with fb22614abc89ee4bce35a3f610f5394eac15c2e48293e5d248f3b217798685ca not found: ID does not exist" containerID="fb22614abc89ee4bce35a3f610f5394eac15c2e48293e5d248f3b217798685ca" Oct 31 00:44:22.730652 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.730569 5222 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb22614abc89ee4bce35a3f610f5394eac15c2e48293e5d248f3b217798685ca"} err="failed to get container status \"fb22614abc89ee4bce35a3f610f5394eac15c2e48293e5d248f3b217798685ca\": rpc error: code = NotFound desc = could not find container \"fb22614abc89ee4bce35a3f610f5394eac15c2e48293e5d248f3b217798685ca\": container with ID starting with fb22614abc89ee4bce35a3f610f5394eac15c2e48293e5d248f3b217798685ca not found: ID does not exist" Oct 31 00:44:22.730652 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.730587 5222 scope.go:122] "RemoveContainer" containerID="51103936220c641723e6faed9902d3b5829355f32f6ab65da69cb81f5503a3e1" Oct 31 00:44:22.730886 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:44:22.730807 5222 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51103936220c641723e6faed9902d3b5829355f32f6ab65da69cb81f5503a3e1\": container with ID starting with 51103936220c641723e6faed9902d3b5829355f32f6ab65da69cb81f5503a3e1 not found: ID does not exist" containerID="51103936220c641723e6faed9902d3b5829355f32f6ab65da69cb81f5503a3e1" Oct 31 00:44:22.730886 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.730830 5222 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51103936220c641723e6faed9902d3b5829355f32f6ab65da69cb81f5503a3e1"} err="failed to get container status \"51103936220c641723e6faed9902d3b5829355f32f6ab65da69cb81f5503a3e1\": rpc error: code = NotFound desc = could not find container \"51103936220c641723e6faed9902d3b5829355f32f6ab65da69cb81f5503a3e1\": container with ID starting with 51103936220c641723e6faed9902d3b5829355f32f6ab65da69cb81f5503a3e1 not found: ID does not exist" Oct 31 00:44:22.730886 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.730847 5222 scope.go:122] "RemoveContainer" containerID="727faaa62ad980a28fcacfec163bf748ad91159ed70e9b234c57594c0ac17834" Oct 31 00:44:22.765506 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.765475 5222 scope.go:122] "RemoveContainer" containerID="727faaa62ad980a28fcacfec163bf748ad91159ed70e9b234c57594c0ac17834" Oct 31 00:44:22.765867 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:44:22.765843 5222 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"727faaa62ad980a28fcacfec163bf748ad91159ed70e9b234c57594c0ac17834\": container with ID starting with 727faaa62ad980a28fcacfec163bf748ad91159ed70e9b234c57594c0ac17834 not found: ID does not exist" containerID="727faaa62ad980a28fcacfec163bf748ad91159ed70e9b234c57594c0ac17834" Oct 31 00:44:22.765955 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.765882 5222 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"727faaa62ad980a28fcacfec163bf748ad91159ed70e9b234c57594c0ac17834"} err="failed to get container status \"727faaa62ad980a28fcacfec163bf748ad91159ed70e9b234c57594c0ac17834\": rpc error: code = NotFound desc = could not find container \"727faaa62ad980a28fcacfec163bf748ad91159ed70e9b234c57594c0ac17834\": container with ID starting with 727faaa62ad980a28fcacfec163bf748ad91159ed70e9b234c57594c0ac17834 not found: ID does not exist" Oct 31 00:44:22.765955 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.765907 5222 scope.go:122] "RemoveContainer" containerID="55271c1d80b8c03372e3b9481f2b215a19a47fbec4c26b0c808c432459732daf" Oct 31 00:44:22.777268 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.777246 5222 scope.go:122] "RemoveContainer" containerID="55271c1d80b8c03372e3b9481f2b215a19a47fbec4c26b0c808c432459732daf" Oct 31 00:44:22.777604 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:44:22.777586 5222 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55271c1d80b8c03372e3b9481f2b215a19a47fbec4c26b0c808c432459732daf\": container with ID starting with 55271c1d80b8c03372e3b9481f2b215a19a47fbec4c26b0c808c432459732daf not found: ID does not exist" containerID="55271c1d80b8c03372e3b9481f2b215a19a47fbec4c26b0c808c432459732daf" Oct 31 00:44:22.777681 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.777610 5222 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55271c1d80b8c03372e3b9481f2b215a19a47fbec4c26b0c808c432459732daf"} err="failed to get container status \"55271c1d80b8c03372e3b9481f2b215a19a47fbec4c26b0c808c432459732daf\": rpc error: code = NotFound desc = could not find container \"55271c1d80b8c03372e3b9481f2b215a19a47fbec4c26b0c808c432459732daf\": container with ID starting with 55271c1d80b8c03372e3b9481f2b215a19a47fbec4c26b0c808c432459732daf not found: ID does not exist" Oct 31 00:44:22.777681 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.777630 5222 scope.go:122] "RemoveContainer" containerID="cf4387b7f512f38cc2ea4e1e3e8a585a653bee5dc49022d7a1a9b40bb40e3895" Oct 31 00:44:22.874850 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.874752 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-4hsbr"] Oct 31 00:44:22.878335 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.878301 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-4hsbr"] Oct 31 00:44:22.991832 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.991799 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vbgx5"] Oct 31 00:44:22.995031 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:22.995002 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vbgx5"] Oct 31 00:44:23.135715 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:23.135363 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-bcc9n"] Oct 31 00:44:23.139405 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:23.139354 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-bcc9n"] Oct 31 00:44:23.232333 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:23.232299 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-2jrfg"] Oct 31 00:44:23.235392 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:23.235353 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-2jrfg"] Oct 31 00:44:23.657104 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:23.657070 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf" path="/var/lib/kubelet/pods/1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf/volumes" Oct 31 00:44:23.657658 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:23.657643 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fe80038-c64b-4114-ac0f-337a29f20204" path="/var/lib/kubelet/pods/2fe80038-c64b-4114-ac0f-337a29f20204/volumes" Oct 31 00:44:23.658122 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:23.658107 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6825ee8a-8ef9-4d40-8046-3e5bd337b6e0" path="/var/lib/kubelet/pods/6825ee8a-8ef9-4d40-8046-3e5bd337b6e0/volumes" Oct 31 00:44:23.658909 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:23.658894 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc854b8d-c4e0-4355-a059-09b118cf87b7" path="/var/lib/kubelet/pods/bc854b8d-c4e0-4355-a059-09b118cf87b7/volumes" Oct 31 00:44:27.198469 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:27.198385 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-tf2dv"] Oct 31 00:44:27.199165 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:27.199148 5222 cpu_manager.go:414] "containerMap: removing container" podUID="6825ee8a-8ef9-4d40-8046-3e5bd337b6e0" containerName="istio-proxy" Oct 31 00:44:27.199231 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:27.199167 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6825ee8a-8ef9-4d40-8046-3e5bd337b6e0" containerName="istio-proxy" Oct 31 00:44:27.199231 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:27.199180 5222 cpu_manager.go:414] "containerMap: removing container" podUID="2fe80038-c64b-4114-ac0f-337a29f20204" containerName="istio-proxy" Oct 31 00:44:27.199231 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:27.199185 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fe80038-c64b-4114-ac0f-337a29f20204" containerName="istio-proxy" Oct 31 00:44:27.199231 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:27.199193 5222 cpu_manager.go:414] "containerMap: removing container" podUID="bc854b8d-c4e0-4355-a059-09b118cf87b7" containerName="istio-proxy" Oct 31 00:44:27.199231 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:27.199198 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc854b8d-c4e0-4355-a059-09b118cf87b7" containerName="istio-proxy" Oct 31 00:44:27.199231 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:27.199226 5222 cpu_manager.go:414] "containerMap: removing container" podUID="1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf" containerName="istio-proxy" Oct 31 00:44:27.199231 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:27.199231 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf" containerName="istio-proxy" Oct 31 00:44:27.199478 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:27.199241 5222 cpu_manager.go:414] "containerMap: removing container" podUID="267e2bb7-9215-4fc3-9874-ae4c2ad13749" containerName="istio-proxy" Oct 31 00:44:27.199478 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:27.199245 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="267e2bb7-9215-4fc3-9874-ae4c2ad13749" containerName="istio-proxy" Oct 31 00:44:27.199478 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:27.199337 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="6825ee8a-8ef9-4d40-8046-3e5bd337b6e0" containerName="istio-proxy" Oct 31 00:44:27.199478 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:27.199346 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fe80038-c64b-4114-ac0f-337a29f20204" containerName="istio-proxy" Oct 31 00:44:27.199478 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:27.199352 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc854b8d-c4e0-4355-a059-09b118cf87b7" containerName="istio-proxy" Oct 31 00:44:27.199478 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:27.199363 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a81dd45-ff89-4738-bf3d-0d3dc0d1feaf" containerName="istio-proxy" Oct 31 00:44:27.199478 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:27.199390 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="267e2bb7-9215-4fc3-9874-ae4c2ad13749" containerName="istio-proxy" Oct 31 00:44:27.201824 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:27.201809 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-tf2dv" Oct 31 00:44:27.255943 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:27.255912 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-5b4db8-openshift-default-dockercfg-tc2fm\"" Oct 31 00:44:27.389438 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:27.389409 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-tf2dv"] Oct 31 00:44:27.390818 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:27.390788 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b191ce67-4983-446c-bd11-394b1f008034-istiod-ca-cert\") pod \"e2e-api-a-5b4db8-openshift-default-5cc6998bbc-tf2dv\" (UID: \"b191ce67-4983-446c-bd11-394b1f008034\") " pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-tf2dv" Oct 31 00:44:27.390997 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:27.390859 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5b48\" (UniqueName: \"kubernetes.io/projected/b191ce67-4983-446c-bd11-394b1f008034-kube-api-access-v5b48\") pod \"e2e-api-a-5b4db8-openshift-default-5cc6998bbc-tf2dv\" (UID: \"b191ce67-4983-446c-bd11-394b1f008034\") " pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-tf2dv" Oct 31 00:44:27.390997 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:27.390900 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b191ce67-4983-446c-bd11-394b1f008034-istio-podinfo\") pod \"e2e-api-a-5b4db8-openshift-default-5cc6998bbc-tf2dv\" (UID: \"b191ce67-4983-446c-bd11-394b1f008034\") " pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-tf2dv" Oct 31 00:44:27.391183 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:27.391062 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b191ce67-4983-446c-bd11-394b1f008034-workload-socket\") pod \"e2e-api-a-5b4db8-openshift-default-5cc6998bbc-tf2dv\" (UID: \"b191ce67-4983-446c-bd11-394b1f008034\") " pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-tf2dv" Oct 31 00:44:27.391183 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:27.391136 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b191ce67-4983-446c-bd11-394b1f008034-workload-certs\") pod \"e2e-api-a-5b4db8-openshift-default-5cc6998bbc-tf2dv\" (UID: \"b191ce67-4983-446c-bd11-394b1f008034\") " pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-tf2dv" Oct 31 00:44:27.391183 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:27.391175 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b191ce67-4983-446c-bd11-394b1f008034-istio-envoy\") pod \"e2e-api-a-5b4db8-openshift-default-5cc6998bbc-tf2dv\" (UID: \"b191ce67-4983-446c-bd11-394b1f008034\") " pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-tf2dv" Oct 31 00:44:27.391458 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:27.391221 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b191ce67-4983-446c-bd11-394b1f008034-istio-data\") pod \"e2e-api-a-5b4db8-openshift-default-5cc6998bbc-tf2dv\" (UID: \"b191ce67-4983-446c-bd11-394b1f008034\") " pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-tf2dv" Oct 31 00:44:27.391458 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:27.391245 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b191ce67-4983-446c-bd11-394b1f008034-istio-token\") pod \"e2e-api-a-5b4db8-openshift-default-5cc6998bbc-tf2dv\" (UID: \"b191ce67-4983-446c-bd11-394b1f008034\") " pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-tf2dv" Oct 31 00:44:27.391458 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:27.391272 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b191ce67-4983-446c-bd11-394b1f008034-credential-socket\") pod \"e2e-api-a-5b4db8-openshift-default-5cc6998bbc-tf2dv\" (UID: \"b191ce67-4983-446c-bd11-394b1f008034\") " pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-tf2dv" Oct 31 00:44:27.498001 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:27.497872 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b191ce67-4983-446c-bd11-394b1f008034-istio-data\") pod \"e2e-api-a-5b4db8-openshift-default-5cc6998bbc-tf2dv\" (UID: \"b191ce67-4983-446c-bd11-394b1f008034\") " pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-tf2dv" Oct 31 00:44:27.498001 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:27.497929 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b191ce67-4983-446c-bd11-394b1f008034-istio-token\") pod \"e2e-api-a-5b4db8-openshift-default-5cc6998bbc-tf2dv\" (UID: \"b191ce67-4983-446c-bd11-394b1f008034\") " pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-tf2dv" Oct 31 00:44:27.498001 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:27.497960 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b191ce67-4983-446c-bd11-394b1f008034-credential-socket\") pod \"e2e-api-a-5b4db8-openshift-default-5cc6998bbc-tf2dv\" (UID: \"b191ce67-4983-446c-bd11-394b1f008034\") " pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-tf2dv" Oct 31 00:44:27.498299 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:27.498068 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b191ce67-4983-446c-bd11-394b1f008034-istiod-ca-cert\") pod \"e2e-api-a-5b4db8-openshift-default-5cc6998bbc-tf2dv\" (UID: \"b191ce67-4983-446c-bd11-394b1f008034\") " pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-tf2dv" Oct 31 00:44:27.498299 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:27.498126 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5b48\" (UniqueName: \"kubernetes.io/projected/b191ce67-4983-446c-bd11-394b1f008034-kube-api-access-v5b48\") pod \"e2e-api-a-5b4db8-openshift-default-5cc6998bbc-tf2dv\" (UID: \"b191ce67-4983-446c-bd11-394b1f008034\") " pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-tf2dv" Oct 31 00:44:27.498299 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:27.498163 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b191ce67-4983-446c-bd11-394b1f008034-istio-podinfo\") pod \"e2e-api-a-5b4db8-openshift-default-5cc6998bbc-tf2dv\" (UID: \"b191ce67-4983-446c-bd11-394b1f008034\") " pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-tf2dv" Oct 31 00:44:27.498535 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:27.498331 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b191ce67-4983-446c-bd11-394b1f008034-workload-socket\") pod \"e2e-api-a-5b4db8-openshift-default-5cc6998bbc-tf2dv\" (UID: \"b191ce67-4983-446c-bd11-394b1f008034\") " pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-tf2dv" Oct 31 00:44:27.498535 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:27.498471 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b191ce67-4983-446c-bd11-394b1f008034-workload-certs\") pod \"e2e-api-a-5b4db8-openshift-default-5cc6998bbc-tf2dv\" (UID: \"b191ce67-4983-446c-bd11-394b1f008034\") " pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-tf2dv" Oct 31 00:44:27.498535 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:27.498516 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b191ce67-4983-446c-bd11-394b1f008034-istio-envoy\") pod \"e2e-api-a-5b4db8-openshift-default-5cc6998bbc-tf2dv\" (UID: \"b191ce67-4983-446c-bd11-394b1f008034\") " pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-tf2dv" Oct 31 00:44:27.510016 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:27.509776 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b191ce67-4983-446c-bd11-394b1f008034-istio-data\") pod \"e2e-api-a-5b4db8-openshift-default-5cc6998bbc-tf2dv\" (UID: \"b191ce67-4983-446c-bd11-394b1f008034\") " pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-tf2dv" Oct 31 00:44:27.510251 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:27.510224 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b191ce67-4983-446c-bd11-394b1f008034-credential-socket\") pod \"e2e-api-a-5b4db8-openshift-default-5cc6998bbc-tf2dv\" (UID: \"b191ce67-4983-446c-bd11-394b1f008034\") " pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-tf2dv" Oct 31 00:44:27.516969 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:27.514947 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b191ce67-4983-446c-bd11-394b1f008034-istiod-ca-cert\") pod \"e2e-api-a-5b4db8-openshift-default-5cc6998bbc-tf2dv\" (UID: \"b191ce67-4983-446c-bd11-394b1f008034\") " pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-tf2dv" Oct 31 00:44:27.516969 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:27.515146 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b191ce67-4983-446c-bd11-394b1f008034-workload-socket\") pod \"e2e-api-a-5b4db8-openshift-default-5cc6998bbc-tf2dv\" (UID: \"b191ce67-4983-446c-bd11-394b1f008034\") " pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-tf2dv" Oct 31 00:44:27.516969 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:27.515242 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b191ce67-4983-446c-bd11-394b1f008034-workload-certs\") pod \"e2e-api-a-5b4db8-openshift-default-5cc6998bbc-tf2dv\" (UID: \"b191ce67-4983-446c-bd11-394b1f008034\") " pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-tf2dv" Oct 31 00:44:27.519207 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:27.519160 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b191ce67-4983-446c-bd11-394b1f008034-istio-podinfo\") pod \"e2e-api-a-5b4db8-openshift-default-5cc6998bbc-tf2dv\" (UID: \"b191ce67-4983-446c-bd11-394b1f008034\") " pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-tf2dv" Oct 31 00:44:27.524524 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:27.524482 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b191ce67-4983-446c-bd11-394b1f008034-istio-envoy\") pod \"e2e-api-a-5b4db8-openshift-default-5cc6998bbc-tf2dv\" (UID: \"b191ce67-4983-446c-bd11-394b1f008034\") " pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-tf2dv" Oct 31 00:44:27.573108 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:27.573076 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b191ce67-4983-446c-bd11-394b1f008034-istio-token\") pod \"e2e-api-a-5b4db8-openshift-default-5cc6998bbc-tf2dv\" (UID: \"b191ce67-4983-446c-bd11-394b1f008034\") " pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-tf2dv" Oct 31 00:44:27.577505 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:27.577479 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5b48\" (UniqueName: \"kubernetes.io/projected/b191ce67-4983-446c-bd11-394b1f008034-kube-api-access-v5b48\") pod \"e2e-api-a-5b4db8-openshift-default-5cc6998bbc-tf2dv\" (UID: \"b191ce67-4983-446c-bd11-394b1f008034\") " pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-tf2dv" Oct 31 00:44:27.751915 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:27.751833 5222 generic.go:356] "Generic (PLEG): container finished" podID="608e314b-554b-40a2-90f7-fbdefd2b27b6" containerID="ff6a9379b42cb98396f10d68f9835af0c5dbe25bc8a4554dd8bcfeb16eb7bbee" exitCode=0 Oct 31 00:44:27.751915 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:27.751901 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-d58cd3-openshift-default-d9c46776f-vklx7" event={"ID":"608e314b-554b-40a2-90f7-fbdefd2b27b6","Type":"ContainerDied","Data":"ff6a9379b42cb98396f10d68f9835af0c5dbe25bc8a4554dd8bcfeb16eb7bbee"} Oct 31 00:44:27.752432 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:27.752361 5222 scope.go:122] "RemoveContainer" containerID="ff6a9379b42cb98396f10d68f9835af0c5dbe25bc8a4554dd8bcfeb16eb7bbee" Oct 31 00:44:27.754479 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:27.754439 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:44:27.754581 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:27.754558 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:44:27.754620 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:27.754609 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:44:27.815631 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:27.815595 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-tf2dv" Oct 31 00:44:28.346308 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:28.346249 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:44:28.346766 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:28.346454 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:44:28.346766 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:28.346547 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:44:28.351084 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:28.351049 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-tf2dv"] Oct 31 00:44:28.775427 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:28.775389 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-d58cd3-openshift-default-d9c46776f-vklx7" event={"ID":"608e314b-554b-40a2-90f7-fbdefd2b27b6","Type":"ContainerStarted","Data":"ed21fb6353a1a845beb2ff9ed88d4338be5a28ac3df2de4357a2f56e0b70f15a"} Oct 31 00:44:28.777452 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:28.777418 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-tf2dv" event={"ID":"b191ce67-4983-446c-bd11-394b1f008034","Type":"ContainerStarted","Data":"be0e03299ea7d929c61339c68d60b33a9e7dc6dea816d5a994593d9ae2a5a510"} Oct 31 00:44:28.777597 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:28.777484 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-tf2dv" event={"ID":"b191ce67-4983-446c-bd11-394b1f008034","Type":"ContainerStarted","Data":"a6342241546876a8cbad3e544e655ba3339966e63171367924c9a8c1af401271"} Oct 31 00:44:28.816452 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:28.816364 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-tf2dv" Oct 31 00:44:29.028817 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:29.028703 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-tf2dv" podStartSLOduration=2.028688205 podStartE2EDuration="2.028688205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:44:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:44:29.026827755 +0000 UTC m=+1761.497997505" watchObservedRunningTime="2026-10-31 00:44:29.028688205 +0000 UTC m=+1761.499857955" Oct 31 00:44:29.399278 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:29.399177 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-d58cd3-openshift-default-d9c46776f-vklx7" Oct 31 00:44:29.400716 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:29.400682 5222 patch_prober.go:29] interesting pod/e2e-shared-a-w2-d58cd3-openshift-default-d9c46776f-vklx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Oct 31 00:44:29.400846 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:29.400737 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-d58cd3-openshift-default-d9c46776f-vklx7" podUID="608e314b-554b-40a2-90f7-fbdefd2b27b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Oct 31 00:44:29.821794 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:29.821762 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-tf2dv" Oct 31 00:44:30.398920 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:30.398883 5222 patch_prober.go:29] interesting pod/e2e-shared-a-w2-d58cd3-openshift-default-d9c46776f-vklx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Oct 31 00:44:30.399087 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:30.398943 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-d58cd3-openshift-default-d9c46776f-vklx7" podUID="608e314b-554b-40a2-90f7-fbdefd2b27b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Oct 31 00:44:30.657455 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:30.656547 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-d58cd3-openshift-default-d9c46776f-vklx7" Oct 31 00:44:30.791584 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:30.791552 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-tf2dv" Oct 31 00:44:30.792618 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:30.792600 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-tf2dv" Oct 31 00:44:31.399155 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:31.399116 5222 patch_prober.go:29] interesting pod/e2e-shared-a-w2-d58cd3-openshift-default-d9c46776f-vklx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Oct 31 00:44:31.399350 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:31.399175 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-d58cd3-openshift-default-d9c46776f-vklx7" podUID="608e314b-554b-40a2-90f7-fbdefd2b27b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Oct 31 00:44:32.399727 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:32.399674 5222 patch_prober.go:29] interesting pod/e2e-shared-a-w2-d58cd3-openshift-default-d9c46776f-vklx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Oct 31 00:44:32.400307 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:32.399757 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-d58cd3-openshift-default-d9c46776f-vklx7" podUID="608e314b-554b-40a2-90f7-fbdefd2b27b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Oct 31 00:44:33.400400 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:33.399762 5222 patch_prober.go:29] interesting pod/e2e-shared-a-w2-d58cd3-openshift-default-d9c46776f-vklx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Oct 31 00:44:33.400400 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:33.399837 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-d58cd3-openshift-default-d9c46776f-vklx7" podUID="608e314b-554b-40a2-90f7-fbdefd2b27b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Oct 31 00:44:34.403981 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:34.403948 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-d58cd3-openshift-default-d9c46776f-vklx7" Oct 31 00:44:34.405285 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:34.405265 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-d58cd3-openshift-default-d9c46776f-vklx7" Oct 31 00:44:35.891308 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:44:35.891262 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 00:44:36.517413 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:36.517364 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f0651cac-k697q"] Oct 31 00:44:36.568254 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:36.568216 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f0651cac-k697q"] Oct 31 00:44:37.658001 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:37.657954 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb5fa2a8-608d-4449-af5f-ab87a659ac07" path="/var/lib/kubelet/pods/bb5fa2a8-608d-4449-af5f-ab87a659ac07/volumes" Oct 31 00:44:39.058655 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:39.058624 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5b4db8-5c87d84775-wgg4h"] Oct 31 00:44:39.059263 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:39.059248 5222 cpu_manager.go:414] "containerMap: removing container" podUID="2fe80038-c64b-4114-ac0f-337a29f20204" containerName="istio-proxy" Oct 31 00:44:39.059316 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:39.059263 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fe80038-c64b-4114-ac0f-337a29f20204" containerName="istio-proxy" Oct 31 00:44:39.059445 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:39.059423 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fe80038-c64b-4114-ac0f-337a29f20204" containerName="istio-proxy" Oct 31 00:44:39.062229 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:39.062209 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5b4db8-5c87d84775-wgg4h" Oct 31 00:44:39.116891 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:39.116861 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5b4db8-5c87d84775-wgg4h"] Oct 31 00:44:39.118420 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:39.118355 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cc24a99-a137-4f53-aed6-6c914b1b9abb-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-5b4db8-5c87d84775-wgg4h\" (UID: \"5cc24a99-a137-4f53-aed6-6c914b1b9abb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5b4db8-5c87d84775-wgg4h" Oct 31 00:44:39.118570 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:39.118481 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmzls\" (UniqueName: \"kubernetes.io/projected/5cc24a99-a137-4f53-aed6-6c914b1b9abb-kube-api-access-nmzls\") pod \"payload-pre-processing-e2e-api-a-5b4db8-5c87d84775-wgg4h\" (UID: \"5cc24a99-a137-4f53-aed6-6c914b1b9abb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5b4db8-5c87d84775-wgg4h" Oct 31 00:44:39.164644 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:39.164611 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-5b4db8\"" Oct 31 00:44:39.164832 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:39.164749 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-5b4db8-dockercfg-wnb5z\"" Oct 31 00:44:39.220132 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:39.220089 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cc24a99-a137-4f53-aed6-6c914b1b9abb-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-5b4db8-5c87d84775-wgg4h\" (UID: \"5cc24a99-a137-4f53-aed6-6c914b1b9abb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5b4db8-5c87d84775-wgg4h" Oct 31 00:44:39.220301 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:39.220195 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmzls\" (UniqueName: \"kubernetes.io/projected/5cc24a99-a137-4f53-aed6-6c914b1b9abb-kube-api-access-nmzls\") pod \"payload-pre-processing-e2e-api-a-5b4db8-5c87d84775-wgg4h\" (UID: \"5cc24a99-a137-4f53-aed6-6c914b1b9abb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5b4db8-5c87d84775-wgg4h" Oct 31 00:44:39.220984 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:39.220962 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cc24a99-a137-4f53-aed6-6c914b1b9abb-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-5b4db8-5c87d84775-wgg4h\" (UID: \"5cc24a99-a137-4f53-aed6-6c914b1b9abb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5b4db8-5c87d84775-wgg4h" Oct 31 00:44:39.230119 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:39.230090 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5b4db8-78dcbd7945-mhhfh"] Oct 31 00:44:39.233543 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:39.233520 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-5b4db8-78dcbd7945-mhhfh" Oct 31 00:44:39.351466 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:39.341227 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5b4db8-78dcbd7945-mhhfh"] Oct 31 00:44:39.351466 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:39.343041 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3b6cede-ec74-40cf-998e-5ff4238cd411-plugins-config-volume\") pod \"payload-processing-e2e-api-a-5b4db8-78dcbd7945-mhhfh\" (UID: \"b3b6cede-ec74-40cf-998e-5ff4238cd411\") " pod="openshift-ingress/payload-processing-e2e-api-a-5b4db8-78dcbd7945-mhhfh" Oct 31 00:44:39.351466 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:39.343086 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjc4j\" (UniqueName: \"kubernetes.io/projected/b3b6cede-ec74-40cf-998e-5ff4238cd411-kube-api-access-cjc4j\") pod \"payload-processing-e2e-api-a-5b4db8-78dcbd7945-mhhfh\" (UID: \"b3b6cede-ec74-40cf-998e-5ff4238cd411\") " pod="openshift-ingress/payload-processing-e2e-api-a-5b4db8-78dcbd7945-mhhfh" Oct 31 00:44:39.453413 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:39.451059 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3b6cede-ec74-40cf-998e-5ff4238cd411-plugins-config-volume\") pod \"payload-processing-e2e-api-a-5b4db8-78dcbd7945-mhhfh\" (UID: \"b3b6cede-ec74-40cf-998e-5ff4238cd411\") " pod="openshift-ingress/payload-processing-e2e-api-a-5b4db8-78dcbd7945-mhhfh" Oct 31 00:44:39.453413 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:39.451113 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjc4j\" (UniqueName: \"kubernetes.io/projected/b3b6cede-ec74-40cf-998e-5ff4238cd411-kube-api-access-cjc4j\") pod \"payload-processing-e2e-api-a-5b4db8-78dcbd7945-mhhfh\" (UID: \"b3b6cede-ec74-40cf-998e-5ff4238cd411\") " pod="openshift-ingress/payload-processing-e2e-api-a-5b4db8-78dcbd7945-mhhfh" Oct 31 00:44:39.453413 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:39.452392 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3b6cede-ec74-40cf-998e-5ff4238cd411-plugins-config-volume\") pod \"payload-processing-e2e-api-a-5b4db8-78dcbd7945-mhhfh\" (UID: \"b3b6cede-ec74-40cf-998e-5ff4238cd411\") " pod="openshift-ingress/payload-processing-e2e-api-a-5b4db8-78dcbd7945-mhhfh" Oct 31 00:44:39.453413 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:39.452949 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmzls\" (UniqueName: \"kubernetes.io/projected/5cc24a99-a137-4f53-aed6-6c914b1b9abb-kube-api-access-nmzls\") pod \"payload-pre-processing-e2e-api-a-5b4db8-5c87d84775-wgg4h\" (UID: \"5cc24a99-a137-4f53-aed6-6c914b1b9abb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5b4db8-5c87d84775-wgg4h" Oct 31 00:44:39.516141 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:39.516106 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjc4j\" (UniqueName: \"kubernetes.io/projected/b3b6cede-ec74-40cf-998e-5ff4238cd411-kube-api-access-cjc4j\") pod \"payload-processing-e2e-api-a-5b4db8-78dcbd7945-mhhfh\" (UID: \"b3b6cede-ec74-40cf-998e-5ff4238cd411\") " pod="openshift-ingress/payload-processing-e2e-api-a-5b4db8-78dcbd7945-mhhfh" Oct 31 00:44:39.576892 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:39.576858 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-5b4db8-78dcbd7945-mhhfh" Oct 31 00:44:39.675613 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:39.675576 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5b4db8-5c87d84775-wgg4h" Oct 31 00:44:39.953750 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:44:39.953719 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3b6cede_ec74_40cf_998e_5ff4238cd411.slice/crio-5ddad855d11fdc896d3f147aa9fcada7c44c0770c7cf8fa113a3d5fd05ebc6ef WatchSource:0}: Error finding container 5ddad855d11fdc896d3f147aa9fcada7c44c0770c7cf8fa113a3d5fd05ebc6ef: Status 404 returned error can't find the container with id 5ddad855d11fdc896d3f147aa9fcada7c44c0770c7cf8fa113a3d5fd05ebc6ef Oct 31 00:44:39.957195 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:39.957150 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5b4db8-78dcbd7945-mhhfh"] Oct 31 00:44:40.059994 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:44:40.059951 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cc24a99_a137_4f53_aed6_6c914b1b9abb.slice/crio-088e4e21f0a93408d3df33863dd0023c9bcad66e2723cecf41fc04a2aeab4929 WatchSource:0}: Error finding container 088e4e21f0a93408d3df33863dd0023c9bcad66e2723cecf41fc04a2aeab4929: Status 404 returned error can't find the container with id 088e4e21f0a93408d3df33863dd0023c9bcad66e2723cecf41fc04a2aeab4929 Oct 31 00:44:40.064345 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:40.064319 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5b4db8-5c87d84775-wgg4h"] Oct 31 00:44:40.914861 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:40.914825 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-5b4db8-78dcbd7945-mhhfh" event={"ID":"b3b6cede-ec74-40cf-998e-5ff4238cd411","Type":"ContainerStarted","Data":"0f5fefa9728e0e47b5db0c1f4a2f96e2488d3e6885a10521b85e6b3c76df33ee"} Oct 31 00:44:40.915091 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:40.914879 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-5b4db8-78dcbd7945-mhhfh" event={"ID":"b3b6cede-ec74-40cf-998e-5ff4238cd411","Type":"ContainerStarted","Data":"5ddad855d11fdc896d3f147aa9fcada7c44c0770c7cf8fa113a3d5fd05ebc6ef"} Oct 31 00:44:40.915091 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:40.914910 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-5b4db8-78dcbd7945-mhhfh" Oct 31 00:44:40.916541 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:40.916513 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5b4db8-5c87d84775-wgg4h" event={"ID":"5cc24a99-a137-4f53-aed6-6c914b1b9abb","Type":"ContainerStarted","Data":"02d3057ec60584680437ecc2d5bfa6c2ad3aad953fe048dccc8cfef4394f9f3e"} Oct 31 00:44:40.916643 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:40.916564 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5b4db8-5c87d84775-wgg4h" event={"ID":"5cc24a99-a137-4f53-aed6-6c914b1b9abb","Type":"ContainerStarted","Data":"088e4e21f0a93408d3df33863dd0023c9bcad66e2723cecf41fc04a2aeab4929"} Oct 31 00:44:40.916643 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:40.916630 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5b4db8-5c87d84775-wgg4h" Oct 31 00:44:41.035402 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:41.035330 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-5b4db8-78dcbd7945-mhhfh" podStartSLOduration=2.035315322 podStartE2EDuration="2.035315322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:44:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:44:41.033061337 +0000 UTC m=+1773.504231089" watchObservedRunningTime="2026-10-31 00:44:41.035315322 +0000 UTC m=+1773.506485071" Oct 31 00:44:41.182761 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:41.182632 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5b4db8-5c87d84775-wgg4h" podStartSLOduration=2.182608113 podStartE2EDuration="2.182608113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:44:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:44:41.160833264 +0000 UTC m=+1773.632003015" watchObservedRunningTime="2026-10-31 00:44:41.182608113 +0000 UTC m=+1773.653777866" Oct 31 00:44:46.051433 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:46.051389 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b32e23d-mvndb"] Oct 31 00:44:46.102377 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:46.102329 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b32e23d-mvndb"] Oct 31 00:44:47.656932 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:47.656899 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e232ca5e-2013-4024-8118-5ecdb3088579" path="/var/lib/kubelet/pods/e232ca5e-2013-4024-8118-5ecdb3088579/volumes" Oct 31 00:44:49.058320 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:49.058283 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-f891e6-openshift-default-565fdf54d7-56xb6"] Oct 31 00:44:49.064141 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:49.064111 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f891e6-openshift-default-565fdf54d7-56xb6" Oct 31 00:44:49.164980 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:49.164940 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18761079-b183-45ca-9b61-cd9717b0195a-istio-envoy\") pod \"e2e-api-a-f891e6-openshift-default-565fdf54d7-56xb6\" (UID: \"18761079-b183-45ca-9b61-cd9717b0195a\") " pod="openshift-ingress/e2e-api-a-f891e6-openshift-default-565fdf54d7-56xb6" Oct 31 00:44:49.165159 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:49.164991 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18761079-b183-45ca-9b61-cd9717b0195a-workload-certs\") pod \"e2e-api-a-f891e6-openshift-default-565fdf54d7-56xb6\" (UID: \"18761079-b183-45ca-9b61-cd9717b0195a\") " pod="openshift-ingress/e2e-api-a-f891e6-openshift-default-565fdf54d7-56xb6" Oct 31 00:44:49.165159 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:49.165060 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18761079-b183-45ca-9b61-cd9717b0195a-istio-data\") pod \"e2e-api-a-f891e6-openshift-default-565fdf54d7-56xb6\" (UID: \"18761079-b183-45ca-9b61-cd9717b0195a\") " pod="openshift-ingress/e2e-api-a-f891e6-openshift-default-565fdf54d7-56xb6" Oct 31 00:44:49.165159 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:49.165121 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18761079-b183-45ca-9b61-cd9717b0195a-workload-socket\") pod \"e2e-api-a-f891e6-openshift-default-565fdf54d7-56xb6\" (UID: \"18761079-b183-45ca-9b61-cd9717b0195a\") " pod="openshift-ingress/e2e-api-a-f891e6-openshift-default-565fdf54d7-56xb6" Oct 31 00:44:49.165266 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:49.165231 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18761079-b183-45ca-9b61-cd9717b0195a-credential-socket\") pod \"e2e-api-a-f891e6-openshift-default-565fdf54d7-56xb6\" (UID: \"18761079-b183-45ca-9b61-cd9717b0195a\") " pod="openshift-ingress/e2e-api-a-f891e6-openshift-default-565fdf54d7-56xb6" Oct 31 00:44:49.165300 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:49.165281 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18761079-b183-45ca-9b61-cd9717b0195a-istio-podinfo\") pod \"e2e-api-a-f891e6-openshift-default-565fdf54d7-56xb6\" (UID: \"18761079-b183-45ca-9b61-cd9717b0195a\") " pod="openshift-ingress/e2e-api-a-f891e6-openshift-default-565fdf54d7-56xb6" Oct 31 00:44:49.165341 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:49.165327 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18761079-b183-45ca-9b61-cd9717b0195a-istiod-ca-cert\") pod \"e2e-api-a-f891e6-openshift-default-565fdf54d7-56xb6\" (UID: \"18761079-b183-45ca-9b61-cd9717b0195a\") " pod="openshift-ingress/e2e-api-a-f891e6-openshift-default-565fdf54d7-56xb6" Oct 31 00:44:49.165508 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:49.165493 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18761079-b183-45ca-9b61-cd9717b0195a-istio-token\") pod \"e2e-api-a-f891e6-openshift-default-565fdf54d7-56xb6\" (UID: \"18761079-b183-45ca-9b61-cd9717b0195a\") " pod="openshift-ingress/e2e-api-a-f891e6-openshift-default-565fdf54d7-56xb6" Oct 31 00:44:49.165597 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:49.165522 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jh2fd\" (UniqueName: \"kubernetes.io/projected/18761079-b183-45ca-9b61-cd9717b0195a-kube-api-access-jh2fd\") pod \"e2e-api-a-f891e6-openshift-default-565fdf54d7-56xb6\" (UID: \"18761079-b183-45ca-9b61-cd9717b0195a\") " pod="openshift-ingress/e2e-api-a-f891e6-openshift-default-565fdf54d7-56xb6" Oct 31 00:44:49.169883 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:49.169862 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-f891e6-openshift-default-dockercfg-gvq4f\"" Oct 31 00:44:49.173608 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:49.173588 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f891e6-openshift-default-565fdf54d7-56xb6"] Oct 31 00:44:49.269398 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:49.269315 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18761079-b183-45ca-9b61-cd9717b0195a-istio-envoy\") pod \"e2e-api-a-f891e6-openshift-default-565fdf54d7-56xb6\" (UID: \"18761079-b183-45ca-9b61-cd9717b0195a\") " pod="openshift-ingress/e2e-api-a-f891e6-openshift-default-565fdf54d7-56xb6" Oct 31 00:44:49.269653 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:49.269450 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18761079-b183-45ca-9b61-cd9717b0195a-workload-certs\") pod \"e2e-api-a-f891e6-openshift-default-565fdf54d7-56xb6\" (UID: \"18761079-b183-45ca-9b61-cd9717b0195a\") " pod="openshift-ingress/e2e-api-a-f891e6-openshift-default-565fdf54d7-56xb6" Oct 31 00:44:49.269653 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:49.269484 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18761079-b183-45ca-9b61-cd9717b0195a-istio-data\") pod \"e2e-api-a-f891e6-openshift-default-565fdf54d7-56xb6\" (UID: \"18761079-b183-45ca-9b61-cd9717b0195a\") " pod="openshift-ingress/e2e-api-a-f891e6-openshift-default-565fdf54d7-56xb6" Oct 31 00:44:49.269653 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:49.269571 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18761079-b183-45ca-9b61-cd9717b0195a-workload-socket\") pod \"e2e-api-a-f891e6-openshift-default-565fdf54d7-56xb6\" (UID: \"18761079-b183-45ca-9b61-cd9717b0195a\") " pod="openshift-ingress/e2e-api-a-f891e6-openshift-default-565fdf54d7-56xb6" Oct 31 00:44:49.269653 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:49.269640 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18761079-b183-45ca-9b61-cd9717b0195a-credential-socket\") pod \"e2e-api-a-f891e6-openshift-default-565fdf54d7-56xb6\" (UID: \"18761079-b183-45ca-9b61-cd9717b0195a\") " pod="openshift-ingress/e2e-api-a-f891e6-openshift-default-565fdf54d7-56xb6" Oct 31 00:44:49.269894 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:49.269698 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18761079-b183-45ca-9b61-cd9717b0195a-istio-podinfo\") pod \"e2e-api-a-f891e6-openshift-default-565fdf54d7-56xb6\" (UID: \"18761079-b183-45ca-9b61-cd9717b0195a\") " pod="openshift-ingress/e2e-api-a-f891e6-openshift-default-565fdf54d7-56xb6" Oct 31 00:44:49.269894 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:49.269727 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18761079-b183-45ca-9b61-cd9717b0195a-istiod-ca-cert\") pod \"e2e-api-a-f891e6-openshift-default-565fdf54d7-56xb6\" (UID: \"18761079-b183-45ca-9b61-cd9717b0195a\") " pod="openshift-ingress/e2e-api-a-f891e6-openshift-default-565fdf54d7-56xb6" Oct 31 00:44:49.270004 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:49.269899 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18761079-b183-45ca-9b61-cd9717b0195a-istio-token\") pod \"e2e-api-a-f891e6-openshift-default-565fdf54d7-56xb6\" (UID: \"18761079-b183-45ca-9b61-cd9717b0195a\") " pod="openshift-ingress/e2e-api-a-f891e6-openshift-default-565fdf54d7-56xb6" Oct 31 00:44:49.270004 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:49.269955 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jh2fd\" (UniqueName: \"kubernetes.io/projected/18761079-b183-45ca-9b61-cd9717b0195a-kube-api-access-jh2fd\") pod \"e2e-api-a-f891e6-openshift-default-565fdf54d7-56xb6\" (UID: \"18761079-b183-45ca-9b61-cd9717b0195a\") " pod="openshift-ingress/e2e-api-a-f891e6-openshift-default-565fdf54d7-56xb6" Oct 31 00:44:49.297426 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:49.273661 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18761079-b183-45ca-9b61-cd9717b0195a-istio-data\") pod \"e2e-api-a-f891e6-openshift-default-565fdf54d7-56xb6\" (UID: \"18761079-b183-45ca-9b61-cd9717b0195a\") " pod="openshift-ingress/e2e-api-a-f891e6-openshift-default-565fdf54d7-56xb6" Oct 31 00:44:49.297426 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:49.273924 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18761079-b183-45ca-9b61-cd9717b0195a-workload-certs\") pod \"e2e-api-a-f891e6-openshift-default-565fdf54d7-56xb6\" (UID: \"18761079-b183-45ca-9b61-cd9717b0195a\") " pod="openshift-ingress/e2e-api-a-f891e6-openshift-default-565fdf54d7-56xb6" Oct 31 00:44:49.297426 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:49.274148 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18761079-b183-45ca-9b61-cd9717b0195a-workload-socket\") pod \"e2e-api-a-f891e6-openshift-default-565fdf54d7-56xb6\" (UID: \"18761079-b183-45ca-9b61-cd9717b0195a\") " pod="openshift-ingress/e2e-api-a-f891e6-openshift-default-565fdf54d7-56xb6" Oct 31 00:44:49.297426 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:49.288793 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18761079-b183-45ca-9b61-cd9717b0195a-credential-socket\") pod \"e2e-api-a-f891e6-openshift-default-565fdf54d7-56xb6\" (UID: \"18761079-b183-45ca-9b61-cd9717b0195a\") " pod="openshift-ingress/e2e-api-a-f891e6-openshift-default-565fdf54d7-56xb6" Oct 31 00:44:49.300217 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:49.298517 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18761079-b183-45ca-9b61-cd9717b0195a-istiod-ca-cert\") pod \"e2e-api-a-f891e6-openshift-default-565fdf54d7-56xb6\" (UID: \"18761079-b183-45ca-9b61-cd9717b0195a\") " pod="openshift-ingress/e2e-api-a-f891e6-openshift-default-565fdf54d7-56xb6" Oct 31 00:44:49.313080 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:49.312997 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18761079-b183-45ca-9b61-cd9717b0195a-istio-envoy\") pod \"e2e-api-a-f891e6-openshift-default-565fdf54d7-56xb6\" (UID: \"18761079-b183-45ca-9b61-cd9717b0195a\") " pod="openshift-ingress/e2e-api-a-f891e6-openshift-default-565fdf54d7-56xb6" Oct 31 00:44:49.318404 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:49.316480 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18761079-b183-45ca-9b61-cd9717b0195a-istio-podinfo\") pod \"e2e-api-a-f891e6-openshift-default-565fdf54d7-56xb6\" (UID: \"18761079-b183-45ca-9b61-cd9717b0195a\") " pod="openshift-ingress/e2e-api-a-f891e6-openshift-default-565fdf54d7-56xb6" Oct 31 00:44:49.365481 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:49.365448 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jh2fd\" (UniqueName: \"kubernetes.io/projected/18761079-b183-45ca-9b61-cd9717b0195a-kube-api-access-jh2fd\") pod \"e2e-api-a-f891e6-openshift-default-565fdf54d7-56xb6\" (UID: \"18761079-b183-45ca-9b61-cd9717b0195a\") " pod="openshift-ingress/e2e-api-a-f891e6-openshift-default-565fdf54d7-56xb6" Oct 31 00:44:49.366324 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:49.366249 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18761079-b183-45ca-9b61-cd9717b0195a-istio-token\") pod \"e2e-api-a-f891e6-openshift-default-565fdf54d7-56xb6\" (UID: \"18761079-b183-45ca-9b61-cd9717b0195a\") " pod="openshift-ingress/e2e-api-a-f891e6-openshift-default-565fdf54d7-56xb6" Oct 31 00:44:49.385685 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:49.385647 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f891e6-openshift-default-565fdf54d7-56xb6" Oct 31 00:44:49.978592 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:49.978561 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f891e6-openshift-default-565fdf54d7-56xb6"] Oct 31 00:44:49.985484 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:49.981868 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:44:49.985742 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:49.985712 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:44:49.985884 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:49.985866 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:44:50.085636 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:50.085579 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f891e6-openshift-default-565fdf54d7-56xb6" event={"ID":"18761079-b183-45ca-9b61-cd9717b0195a","Type":"ContainerStarted","Data":"64f112b6f660b30d55fb28d7b2a1385fc252f2088e5d3db8357f9e292174f936"} Oct 31 00:44:51.094606 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:51.094569 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f891e6-openshift-default-565fdf54d7-56xb6" event={"ID":"18761079-b183-45ca-9b61-cd9717b0195a","Type":"ContainerStarted","Data":"17f34d286c445ba41db967e1cc0a46cb4287bfcc862122233af07db17a2e4fc3"} Oct 31 00:44:51.269596 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:51.269540 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-f891e6-openshift-default-565fdf54d7-56xb6" podStartSLOduration=2.269525513 podStartE2EDuration="2.269525513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:44:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:44:51.268514747 +0000 UTC m=+1783.739684497" watchObservedRunningTime="2026-10-31 00:44:51.269525513 +0000 UTC m=+1783.740695262" Oct 31 00:44:51.388717 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:51.388628 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-f891e6-openshift-default-565fdf54d7-56xb6" Oct 31 00:44:51.440929 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:51.440895 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-f891e6-openshift-default-565fdf54d7-56xb6" Oct 31 00:44:51.926452 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:51.926424 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-5b4db8-78dcbd7945-mhhfh" Oct 31 00:44:51.926629 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:51.926500 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5b4db8-5c87d84775-wgg4h" Oct 31 00:44:52.102919 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:52.102888 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f891e6-openshift-default-565fdf54d7-56xb6" Oct 31 00:44:52.103892 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:52.103868 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-f891e6-openshift-default-565fdf54d7-56xb6" Oct 31 00:44:57.068151 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:57.068107 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-dcdbd0-openshift-default-86cd956cc9-p8h4z"] Oct 31 00:44:57.072776 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:57.072753 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-dcdbd0-openshift-default-86cd956cc9-p8h4z" Oct 31 00:44:57.168883 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:57.167618 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-dcdbd0-openshift-default-dockercfg-8czsh\"" Oct 31 00:44:57.413478 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:57.412550 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0694975-c551-4a8a-b296-eaee47077b0c-workload-socket\") pod \"e2e-shared-b-w2-dcdbd0-openshift-default-86cd956cc9-p8h4z\" (UID: \"d0694975-c551-4a8a-b296-eaee47077b0c\") " pod="openshift-ingress/e2e-shared-b-w2-dcdbd0-openshift-default-86cd956cc9-p8h4z" Oct 31 00:44:57.413478 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:57.412611 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0694975-c551-4a8a-b296-eaee47077b0c-istiod-ca-cert\") pod \"e2e-shared-b-w2-dcdbd0-openshift-default-86cd956cc9-p8h4z\" (UID: \"d0694975-c551-4a8a-b296-eaee47077b0c\") " pod="openshift-ingress/e2e-shared-b-w2-dcdbd0-openshift-default-86cd956cc9-p8h4z" Oct 31 00:44:57.413478 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:57.412710 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0694975-c551-4a8a-b296-eaee47077b0c-credential-socket\") pod \"e2e-shared-b-w2-dcdbd0-openshift-default-86cd956cc9-p8h4z\" (UID: \"d0694975-c551-4a8a-b296-eaee47077b0c\") " pod="openshift-ingress/e2e-shared-b-w2-dcdbd0-openshift-default-86cd956cc9-p8h4z" Oct 31 00:44:57.413478 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:57.412757 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0694975-c551-4a8a-b296-eaee47077b0c-istio-envoy\") pod \"e2e-shared-b-w2-dcdbd0-openshift-default-86cd956cc9-p8h4z\" (UID: \"d0694975-c551-4a8a-b296-eaee47077b0c\") " pod="openshift-ingress/e2e-shared-b-w2-dcdbd0-openshift-default-86cd956cc9-p8h4z" Oct 31 00:44:57.413478 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:57.412864 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0694975-c551-4a8a-b296-eaee47077b0c-istio-podinfo\") pod \"e2e-shared-b-w2-dcdbd0-openshift-default-86cd956cc9-p8h4z\" (UID: \"d0694975-c551-4a8a-b296-eaee47077b0c\") " pod="openshift-ingress/e2e-shared-b-w2-dcdbd0-openshift-default-86cd956cc9-p8h4z" Oct 31 00:44:57.413478 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:57.412915 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0694975-c551-4a8a-b296-eaee47077b0c-istio-data\") pod \"e2e-shared-b-w2-dcdbd0-openshift-default-86cd956cc9-p8h4z\" (UID: \"d0694975-c551-4a8a-b296-eaee47077b0c\") " pod="openshift-ingress/e2e-shared-b-w2-dcdbd0-openshift-default-86cd956cc9-p8h4z" Oct 31 00:44:57.413478 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:57.412963 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0694975-c551-4a8a-b296-eaee47077b0c-workload-certs\") pod \"e2e-shared-b-w2-dcdbd0-openshift-default-86cd956cc9-p8h4z\" (UID: \"d0694975-c551-4a8a-b296-eaee47077b0c\") " pod="openshift-ingress/e2e-shared-b-w2-dcdbd0-openshift-default-86cd956cc9-p8h4z" Oct 31 00:44:57.413478 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:57.413010 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0694975-c551-4a8a-b296-eaee47077b0c-istio-token\") pod \"e2e-shared-b-w2-dcdbd0-openshift-default-86cd956cc9-p8h4z\" (UID: \"d0694975-c551-4a8a-b296-eaee47077b0c\") " pod="openshift-ingress/e2e-shared-b-w2-dcdbd0-openshift-default-86cd956cc9-p8h4z" Oct 31 00:44:57.413478 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:57.413076 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d66gr\" (UniqueName: \"kubernetes.io/projected/d0694975-c551-4a8a-b296-eaee47077b0c-kube-api-access-d66gr\") pod \"e2e-shared-b-w2-dcdbd0-openshift-default-86cd956cc9-p8h4z\" (UID: \"d0694975-c551-4a8a-b296-eaee47077b0c\") " pod="openshift-ingress/e2e-shared-b-w2-dcdbd0-openshift-default-86cd956cc9-p8h4z" Oct 31 00:44:57.419013 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:57.418850 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-dcdbd0-openshift-default-86cd956cc9-p8h4z"] Oct 31 00:44:57.516594 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:57.516541 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d66gr\" (UniqueName: \"kubernetes.io/projected/d0694975-c551-4a8a-b296-eaee47077b0c-kube-api-access-d66gr\") pod \"e2e-shared-b-w2-dcdbd0-openshift-default-86cd956cc9-p8h4z\" (UID: \"d0694975-c551-4a8a-b296-eaee47077b0c\") " pod="openshift-ingress/e2e-shared-b-w2-dcdbd0-openshift-default-86cd956cc9-p8h4z" Oct 31 00:44:57.517044 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:57.517018 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0694975-c551-4a8a-b296-eaee47077b0c-workload-socket\") pod \"e2e-shared-b-w2-dcdbd0-openshift-default-86cd956cc9-p8h4z\" (UID: \"d0694975-c551-4a8a-b296-eaee47077b0c\") " pod="openshift-ingress/e2e-shared-b-w2-dcdbd0-openshift-default-86cd956cc9-p8h4z" Oct 31 00:44:57.517223 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:57.517206 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0694975-c551-4a8a-b296-eaee47077b0c-istiod-ca-cert\") pod \"e2e-shared-b-w2-dcdbd0-openshift-default-86cd956cc9-p8h4z\" (UID: \"d0694975-c551-4a8a-b296-eaee47077b0c\") " pod="openshift-ingress/e2e-shared-b-w2-dcdbd0-openshift-default-86cd956cc9-p8h4z" Oct 31 00:44:57.517456 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:57.517436 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0694975-c551-4a8a-b296-eaee47077b0c-credential-socket\") pod \"e2e-shared-b-w2-dcdbd0-openshift-default-86cd956cc9-p8h4z\" (UID: \"d0694975-c551-4a8a-b296-eaee47077b0c\") " pod="openshift-ingress/e2e-shared-b-w2-dcdbd0-openshift-default-86cd956cc9-p8h4z" Oct 31 00:44:57.517623 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:57.517606 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0694975-c551-4a8a-b296-eaee47077b0c-istio-envoy\") pod \"e2e-shared-b-w2-dcdbd0-openshift-default-86cd956cc9-p8h4z\" (UID: \"d0694975-c551-4a8a-b296-eaee47077b0c\") " pod="openshift-ingress/e2e-shared-b-w2-dcdbd0-openshift-default-86cd956cc9-p8h4z" Oct 31 00:44:57.517815 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:57.517798 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0694975-c551-4a8a-b296-eaee47077b0c-istio-podinfo\") pod \"e2e-shared-b-w2-dcdbd0-openshift-default-86cd956cc9-p8h4z\" (UID: \"d0694975-c551-4a8a-b296-eaee47077b0c\") " pod="openshift-ingress/e2e-shared-b-w2-dcdbd0-openshift-default-86cd956cc9-p8h4z" Oct 31 00:44:57.517955 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:57.517940 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0694975-c551-4a8a-b296-eaee47077b0c-istio-data\") pod \"e2e-shared-b-w2-dcdbd0-openshift-default-86cd956cc9-p8h4z\" (UID: \"d0694975-c551-4a8a-b296-eaee47077b0c\") " pod="openshift-ingress/e2e-shared-b-w2-dcdbd0-openshift-default-86cd956cc9-p8h4z" Oct 31 00:44:57.518116 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:57.518098 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0694975-c551-4a8a-b296-eaee47077b0c-workload-certs\") pod \"e2e-shared-b-w2-dcdbd0-openshift-default-86cd956cc9-p8h4z\" (UID: \"d0694975-c551-4a8a-b296-eaee47077b0c\") " pod="openshift-ingress/e2e-shared-b-w2-dcdbd0-openshift-default-86cd956cc9-p8h4z" Oct 31 00:44:57.518260 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:57.518244 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0694975-c551-4a8a-b296-eaee47077b0c-istio-token\") pod \"e2e-shared-b-w2-dcdbd0-openshift-default-86cd956cc9-p8h4z\" (UID: \"d0694975-c551-4a8a-b296-eaee47077b0c\") " pod="openshift-ingress/e2e-shared-b-w2-dcdbd0-openshift-default-86cd956cc9-p8h4z" Oct 31 00:44:57.519705 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:57.519681 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0694975-c551-4a8a-b296-eaee47077b0c-workload-socket\") pod \"e2e-shared-b-w2-dcdbd0-openshift-default-86cd956cc9-p8h4z\" (UID: \"d0694975-c551-4a8a-b296-eaee47077b0c\") " pod="openshift-ingress/e2e-shared-b-w2-dcdbd0-openshift-default-86cd956cc9-p8h4z" Oct 31 00:44:57.520563 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:57.520542 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0694975-c551-4a8a-b296-eaee47077b0c-istiod-ca-cert\") pod \"e2e-shared-b-w2-dcdbd0-openshift-default-86cd956cc9-p8h4z\" (UID: \"d0694975-c551-4a8a-b296-eaee47077b0c\") " pod="openshift-ingress/e2e-shared-b-w2-dcdbd0-openshift-default-86cd956cc9-p8h4z" Oct 31 00:44:57.520969 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:57.520942 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0694975-c551-4a8a-b296-eaee47077b0c-credential-socket\") pod \"e2e-shared-b-w2-dcdbd0-openshift-default-86cd956cc9-p8h4z\" (UID: \"d0694975-c551-4a8a-b296-eaee47077b0c\") " pod="openshift-ingress/e2e-shared-b-w2-dcdbd0-openshift-default-86cd956cc9-p8h4z" Oct 31 00:44:57.586763 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:57.586723 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0694975-c551-4a8a-b296-eaee47077b0c-istio-envoy\") pod \"e2e-shared-b-w2-dcdbd0-openshift-default-86cd956cc9-p8h4z\" (UID: \"d0694975-c551-4a8a-b296-eaee47077b0c\") " pod="openshift-ingress/e2e-shared-b-w2-dcdbd0-openshift-default-86cd956cc9-p8h4z" Oct 31 00:44:57.588358 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:57.588120 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0694975-c551-4a8a-b296-eaee47077b0c-istio-data\") pod \"e2e-shared-b-w2-dcdbd0-openshift-default-86cd956cc9-p8h4z\" (UID: \"d0694975-c551-4a8a-b296-eaee47077b0c\") " pod="openshift-ingress/e2e-shared-b-w2-dcdbd0-openshift-default-86cd956cc9-p8h4z" Oct 31 00:44:57.589027 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:57.588775 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0694975-c551-4a8a-b296-eaee47077b0c-workload-certs\") pod \"e2e-shared-b-w2-dcdbd0-openshift-default-86cd956cc9-p8h4z\" (UID: \"d0694975-c551-4a8a-b296-eaee47077b0c\") " pod="openshift-ingress/e2e-shared-b-w2-dcdbd0-openshift-default-86cd956cc9-p8h4z" Oct 31 00:44:57.652397 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:57.644735 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d66gr\" (UniqueName: \"kubernetes.io/projected/d0694975-c551-4a8a-b296-eaee47077b0c-kube-api-access-d66gr\") pod \"e2e-shared-b-w2-dcdbd0-openshift-default-86cd956cc9-p8h4z\" (UID: \"d0694975-c551-4a8a-b296-eaee47077b0c\") " pod="openshift-ingress/e2e-shared-b-w2-dcdbd0-openshift-default-86cd956cc9-p8h4z" Oct 31 00:44:57.746613 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:57.746448 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0694975-c551-4a8a-b296-eaee47077b0c-istio-token\") pod \"e2e-shared-b-w2-dcdbd0-openshift-default-86cd956cc9-p8h4z\" (UID: \"d0694975-c551-4a8a-b296-eaee47077b0c\") " pod="openshift-ingress/e2e-shared-b-w2-dcdbd0-openshift-default-86cd956cc9-p8h4z" Oct 31 00:44:57.751409 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:57.746947 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0694975-c551-4a8a-b296-eaee47077b0c-istio-podinfo\") pod \"e2e-shared-b-w2-dcdbd0-openshift-default-86cd956cc9-p8h4z\" (UID: \"d0694975-c551-4a8a-b296-eaee47077b0c\") " pod="openshift-ingress/e2e-shared-b-w2-dcdbd0-openshift-default-86cd956cc9-p8h4z" Oct 31 00:44:57.799402 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:57.798591 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f891e6-5cf4b56747-st4dd"] Oct 31 00:44:57.805553 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:57.805477 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f891e6-5cf4b56747-st4dd" Oct 31 00:44:57.824575 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:57.824542 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f891e6-5cf4b56747-st4dd"] Oct 31 00:44:57.868649 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:57.860308 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-f891e6\"" Oct 31 00:44:57.868649 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:57.860544 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-f891e6-dockercfg-xhjws\"" Oct 31 00:44:57.929506 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:57.924937 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xlfc\" (UniqueName: \"kubernetes.io/projected/815b9a5c-72ac-447b-bd5f-bb0df408c5f3-kube-api-access-8xlfc\") pod \"payload-pre-processing-e2e-api-a-f891e6-5cf4b56747-st4dd\" (UID: \"815b9a5c-72ac-447b-bd5f-bb0df408c5f3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f891e6-5cf4b56747-st4dd" Oct 31 00:44:57.929506 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:57.925290 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/815b9a5c-72ac-447b-bd5f-bb0df408c5f3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-f891e6-5cf4b56747-st4dd\" (UID: \"815b9a5c-72ac-447b-bd5f-bb0df408c5f3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f891e6-5cf4b56747-st4dd" Oct 31 00:44:57.953435 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:57.953361 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f891e6-5b8bd8d58f-wpn4q"] Oct 31 00:44:57.998115 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:57.997761 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-dcdbd0-openshift-default-86cd956cc9-p8h4z" Oct 31 00:44:58.042934 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:58.042887 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/815b9a5c-72ac-447b-bd5f-bb0df408c5f3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-f891e6-5cf4b56747-st4dd\" (UID: \"815b9a5c-72ac-447b-bd5f-bb0df408c5f3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f891e6-5cf4b56747-st4dd" Oct 31 00:44:58.043127 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:58.043000 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xlfc\" (UniqueName: \"kubernetes.io/projected/815b9a5c-72ac-447b-bd5f-bb0df408c5f3-kube-api-access-8xlfc\") pod \"payload-pre-processing-e2e-api-a-f891e6-5cf4b56747-st4dd\" (UID: \"815b9a5c-72ac-447b-bd5f-bb0df408c5f3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f891e6-5cf4b56747-st4dd" Oct 31 00:44:58.044926 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:58.044364 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/815b9a5c-72ac-447b-bd5f-bb0df408c5f3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-f891e6-5cf4b56747-st4dd\" (UID: \"815b9a5c-72ac-447b-bd5f-bb0df408c5f3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f891e6-5cf4b56747-st4dd" Oct 31 00:44:58.047663 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:58.047631 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f891e6-5b8bd8d58f-wpn4q" Oct 31 00:44:58.073500 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:58.073462 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f891e6-5b8bd8d58f-wpn4q"] Oct 31 00:44:58.105738 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:58.105708 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xlfc\" (UniqueName: \"kubernetes.io/projected/815b9a5c-72ac-447b-bd5f-bb0df408c5f3-kube-api-access-8xlfc\") pod \"payload-pre-processing-e2e-api-a-f891e6-5cf4b56747-st4dd\" (UID: \"815b9a5c-72ac-447b-bd5f-bb0df408c5f3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f891e6-5cf4b56747-st4dd" Oct 31 00:44:58.127685 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:58.127658 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f891e6-5cf4b56747-st4dd" Oct 31 00:44:58.148678 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:58.148600 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2rfc\" (UniqueName: \"kubernetes.io/projected/2b11eda2-8622-4e58-ad66-e13985b1dd36-kube-api-access-m2rfc\") pod \"payload-processing-e2e-api-a-f891e6-5b8bd8d58f-wpn4q\" (UID: \"2b11eda2-8622-4e58-ad66-e13985b1dd36\") " pod="openshift-ingress/payload-processing-e2e-api-a-f891e6-5b8bd8d58f-wpn4q" Oct 31 00:44:58.149438 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:58.148853 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b11eda2-8622-4e58-ad66-e13985b1dd36-plugins-config-volume\") pod \"payload-processing-e2e-api-a-f891e6-5b8bd8d58f-wpn4q\" (UID: \"2b11eda2-8622-4e58-ad66-e13985b1dd36\") " pod="openshift-ingress/payload-processing-e2e-api-a-f891e6-5b8bd8d58f-wpn4q" Oct 31 00:44:58.250878 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:58.250781 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b11eda2-8622-4e58-ad66-e13985b1dd36-plugins-config-volume\") pod \"payload-processing-e2e-api-a-f891e6-5b8bd8d58f-wpn4q\" (UID: \"2b11eda2-8622-4e58-ad66-e13985b1dd36\") " pod="openshift-ingress/payload-processing-e2e-api-a-f891e6-5b8bd8d58f-wpn4q" Oct 31 00:44:58.251050 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:58.251016 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2rfc\" (UniqueName: \"kubernetes.io/projected/2b11eda2-8622-4e58-ad66-e13985b1dd36-kube-api-access-m2rfc\") pod \"payload-processing-e2e-api-a-f891e6-5b8bd8d58f-wpn4q\" (UID: \"2b11eda2-8622-4e58-ad66-e13985b1dd36\") " pod="openshift-ingress/payload-processing-e2e-api-a-f891e6-5b8bd8d58f-wpn4q" Oct 31 00:44:58.251532 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:58.251513 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b11eda2-8622-4e58-ad66-e13985b1dd36-plugins-config-volume\") pod \"payload-processing-e2e-api-a-f891e6-5b8bd8d58f-wpn4q\" (UID: \"2b11eda2-8622-4e58-ad66-e13985b1dd36\") " pod="openshift-ingress/payload-processing-e2e-api-a-f891e6-5b8bd8d58f-wpn4q" Oct 31 00:44:58.315961 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:58.315927 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2rfc\" (UniqueName: \"kubernetes.io/projected/2b11eda2-8622-4e58-ad66-e13985b1dd36-kube-api-access-m2rfc\") pod \"payload-processing-e2e-api-a-f891e6-5b8bd8d58f-wpn4q\" (UID: \"2b11eda2-8622-4e58-ad66-e13985b1dd36\") " pod="openshift-ingress/payload-processing-e2e-api-a-f891e6-5b8bd8d58f-wpn4q" Oct 31 00:44:58.387780 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:58.387740 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f891e6-5b8bd8d58f-wpn4q" Oct 31 00:44:58.499954 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:58.499925 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-dcdbd0-openshift-default-86cd956cc9-p8h4z"] Oct 31 00:44:58.503194 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:58.503147 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:44:58.503317 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:58.503249 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:44:58.503317 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:58.503295 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:44:58.542777 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:44:58.542742 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod815b9a5c_72ac_447b_bd5f_bb0df408c5f3.slice/crio-4cf5575c34faa8a9f46c557a7d2f839e9b531878db418a325588fd6734614554 WatchSource:0}: Error finding container 4cf5575c34faa8a9f46c557a7d2f839e9b531878db418a325588fd6734614554: Status 404 returned error can't find the container with id 4cf5575c34faa8a9f46c557a7d2f839e9b531878db418a325588fd6734614554 Oct 31 00:44:58.547721 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:58.546510 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f891e6-5cf4b56747-st4dd"] Oct 31 00:44:58.662770 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:58.662733 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f891e6-5cf4b56747-st4dd" event={"ID":"815b9a5c-72ac-447b-bd5f-bb0df408c5f3","Type":"ContainerStarted","Data":"cc59c8674e4d5ee2bad47707cc6c3feded1509019fb145c10934697f4db595aa"} Oct 31 00:44:58.662959 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:58.662803 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f891e6-5cf4b56747-st4dd" Oct 31 00:44:58.662959 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:58.662822 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f891e6-5cf4b56747-st4dd" event={"ID":"815b9a5c-72ac-447b-bd5f-bb0df408c5f3","Type":"ContainerStarted","Data":"4cf5575c34faa8a9f46c557a7d2f839e9b531878db418a325588fd6734614554"} Oct 31 00:44:58.664890 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:58.664857 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-dcdbd0-openshift-default-86cd956cc9-p8h4z" event={"ID":"d0694975-c551-4a8a-b296-eaee47077b0c","Type":"ContainerStarted","Data":"ce3d7e380246894abac01542c0fdcd9b01ae85322ca675c9d5d7ac62e7e8f8ce"} Oct 31 00:44:58.665006 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:58.664921 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-dcdbd0-openshift-default-86cd956cc9-p8h4z" event={"ID":"d0694975-c551-4a8a-b296-eaee47077b0c","Type":"ContainerStarted","Data":"a53d563fe9f049b4392e680e5c3f8bd711e499b10fa2fa0742df1776658c6497"} Oct 31 00:44:58.766110 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:44:58.766077 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b11eda2_8622_4e58_ad66_e13985b1dd36.slice/crio-88531a398b76cbea7bb96fe1bf0ad4917e0518336af53845125d95ed8795ef45 WatchSource:0}: Error finding container 88531a398b76cbea7bb96fe1bf0ad4917e0518336af53845125d95ed8795ef45: Status 404 returned error can't find the container with id 88531a398b76cbea7bb96fe1bf0ad4917e0518336af53845125d95ed8795ef45 Oct 31 00:44:58.769736 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:58.769704 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f891e6-5b8bd8d58f-wpn4q"] Oct 31 00:44:58.783755 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:58.783709 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f891e6-5cf4b56747-st4dd" podStartSLOduration=1.783695061 podStartE2EDuration="1.783695061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:44:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:44:58.780690645 +0000 UTC m=+1791.251860394" watchObservedRunningTime="2026-10-31 00:44:58.783695061 +0000 UTC m=+1791.254864807" Oct 31 00:44:58.942395 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:58.939476 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-dcdbd0-openshift-default-86cd956cc9-p8h4z" podStartSLOduration=1.939454649 podStartE2EDuration="1.939454649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:44:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:44:58.903943913 +0000 UTC m=+1791.375113660" watchObservedRunningTime="2026-10-31 00:44:58.939454649 +0000 UTC m=+1791.410624399" Oct 31 00:44:59.000473 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:59.000435 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-dcdbd0-openshift-default-86cd956cc9-p8h4z" Oct 31 00:44:59.001897 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:59.001860 5222 patch_prober.go:29] interesting pod/e2e-shared-b-w2-dcdbd0-openshift-default-86cd956cc9-p8h4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Oct 31 00:44:59.002026 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:59.001926 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-dcdbd0-openshift-default-86cd956cc9-p8h4z" podUID="d0694975-c551-4a8a-b296-eaee47077b0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Oct 31 00:44:59.673861 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:59.673821 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f891e6-5b8bd8d58f-wpn4q" event={"ID":"2b11eda2-8622-4e58-ad66-e13985b1dd36","Type":"ContainerStarted","Data":"0cf16ed2e7b751144ec3a3b69c45e986d1aed2a88fc35c9bb399ca87bab7d2e7"} Oct 31 00:44:59.674324 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:59.673878 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f891e6-5b8bd8d58f-wpn4q" event={"ID":"2b11eda2-8622-4e58-ad66-e13985b1dd36","Type":"ContainerStarted","Data":"88531a398b76cbea7bb96fe1bf0ad4917e0518336af53845125d95ed8795ef45"} Oct 31 00:44:59.794163 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:44:59.794099 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-f891e6-5b8bd8d58f-wpn4q" podStartSLOduration=2.79408101 podStartE2EDuration="2.79408101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:44:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:44:59.793231744 +0000 UTC m=+1792.264401485" watchObservedRunningTime="2026-10-31 00:44:59.79408101 +0000 UTC m=+1792.265250757" Oct 31 00:45:00.004984 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:00.004910 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-dcdbd0-openshift-default-86cd956cc9-p8h4z" Oct 31 00:45:00.188018 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:00.187952 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-e030e0-29890125-88clm"] Oct 31 00:45:00.192686 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:00.192661 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-e030e0-29890125-88clm" Oct 31 00:45:00.195874 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:00.195841 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-d58cd3-29890125-5z9v5"] Oct 31 00:45:00.200291 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:00.200238 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-0da668-29890125-jfbgd"] Oct 31 00:45:00.201196 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:00.201177 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-d58cd3-29890125-5z9v5" Oct 31 00:45:00.204869 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:00.204851 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-7d5f4de7-29890125-wn9gz"] Oct 31 00:45:00.205008 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:00.204983 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-0da668-29890125-jfbgd" Oct 31 00:45:00.208946 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:00.208918 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-7d5f4de7-29890125-wn9gz" Oct 31 00:45:00.226499 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:00.226474 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-101b9e-29890125-8899q"] Oct 31 00:45:00.231013 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:00.230992 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-101b9e-29890125-8899q" Oct 31 00:45:00.247078 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:00.247047 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1e5e75-29890125-5xl6t"] Oct 31 00:45:00.251547 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:00.251519 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890125-4cnvj"] Oct 31 00:45:00.251705 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:00.251596 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1e5e75-29890125-5xl6t" Oct 31 00:45:00.253578 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:00.253553 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:45:00.253578 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:00.253583 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:45:00.253812 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:00.253671 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfz9z\"" Oct 31 00:45:00.256661 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:00.256606 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-e030e0-29890125-88clm"] Oct 31 00:45:00.256747 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:00.256693 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890125-4cnvj" Oct 31 00:45:00.280238 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:00.280189 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da9d07-29890125-q7hqd"] Oct 31 00:45:00.291258 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:00.290340 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0edd0ed-32e0-4ba2-8ae0-958c6c530645-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-d58cd3-29890125-5z9v5\" (UID: \"e0edd0ed-32e0-4ba2-8ae0-958c6c530645\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-d58cd3-29890125-5z9v5" Oct 31 00:45:00.291258 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:00.290443 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee2582c4-1fff-43ff-b101-98a384127187-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-e030e0-29890125-88clm\" (UID: \"ee2582c4-1fff-43ff-b101-98a384127187\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-e030e0-29890125-88clm" Oct 31 00:45:00.291258 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:00.290533 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ns9t\" (UniqueName: \"kubernetes.io/projected/e0edd0ed-32e0-4ba2-8ae0-958c6c530645-kube-api-access-8ns9t\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-d58cd3-29890125-5z9v5\" (UID: \"e0edd0ed-32e0-4ba2-8ae0-958c6c530645\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-d58cd3-29890125-5z9v5" Oct 31 00:45:00.291258 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:00.290600 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jh2dt\" (UniqueName: \"kubernetes.io/projected/ee2582c4-1fff-43ff-b101-98a384127187-kube-api-access-jh2dt\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-e030e0-29890125-88clm\" (UID: \"ee2582c4-1fff-43ff-b101-98a384127187\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-e030e0-29890125-88clm" Oct 31 00:45:00.292961 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:00.292918 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-d58cd3-29890125-5z9v5"] Oct 31 00:45:00.293226 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:00.293210 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-0da668-29890125-jfbgd"] Oct 31 00:45:00.293439 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:00.293415 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da9d07-29890125-q7hqd" Oct 31 00:45:00.300675 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:00.300656 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-982ae3-29890125-mjwrq"] Oct 31 00:45:00.313585 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:00.313557 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-101b9e-29890125-8899q"] Oct 31 00:45:00.313728 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:00.313603 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1e5e75-29890125-5xl6t"] Oct 31 00:45:00.313728 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:00.313687 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-982ae3-29890125-mjwrq" Oct 31 00:45:00.341039 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:00.340940 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890125-4cnvj"] Oct 31 00:45:00.346048 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:00.346019 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-7d5f4de7-29890125-wn9gz"] Oct 31 00:45:00.349912 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:00.349880 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da9d07-29890125-q7hqd"] Oct 31 00:45:00.392423 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:00.392349 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvr2v\" (UniqueName: \"kubernetes.io/projected/e64541b5-6dc8-4677-aaf3-95422b177fce-kube-api-access-bvr2v\") pod \"maas-api-key-cleanup-29890125-4cnvj\" (UID: \"e64541b5-6dc8-4677-aaf3-95422b177fce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890125-4cnvj" Oct 31 00:45:00.392591 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:00.392535 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b644185-b04a-4508-ae1a-61bb035c796e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-0da668-29890125-jfbgd\" (UID: \"0b644185-b04a-4508-ae1a-61bb035c796e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-0da668-29890125-jfbgd" Oct 31 00:45:00.392591 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:00.392582 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e8880e8-58ba-4e14-b50b-1d66c4765ebd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-7d5f4de7-29890125-wn9gz\" (UID: \"9e8880e8-58ba-4e14-b50b-1d66c4765ebd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-7d5f4de7-29890125-wn9gz" Oct 31 00:45:00.392682 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:00.392668 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pc5w\" (UniqueName: \"kubernetes.io/projected/bf7409ba-8c48-494c-92ac-a1686ce34ad2-kube-api-access-4pc5w\") pod \"maas-api-key-cleanup-e2e-worker-w4-da9d07-29890125-q7hqd\" (UID: \"bf7409ba-8c48-494c-92ac-a1686ce34ad2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da9d07-29890125-q7hqd" Oct 31 00:45:00.392726 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:00.392712 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4sgr\" (UniqueName: \"kubernetes.io/projected/17065afe-2927-4aa9-9dce-9c6c55315d1e-kube-api-access-r4sgr\") pod \"maas-api-key-cleanup-e2e-worker-w0-101b9e-29890125-8899q\" (UID: \"17065afe-2927-4aa9-9dce-9c6c55315d1e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-101b9e-29890125-8899q" Oct 31 00:45:00.392780 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:00.392737 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0edd0ed-32e0-4ba2-8ae0-958c6c530645-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-d58cd3-29890125-5z9v5\" (UID: \"e0edd0ed-32e0-4ba2-8ae0-958c6c530645\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-d58cd3-29890125-5z9v5" Oct 31 00:45:00.392944 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:00.392902 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee2582c4-1fff-43ff-b101-98a384127187-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-e030e0-29890125-88clm\" (UID: \"ee2582c4-1fff-43ff-b101-98a384127187\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-e030e0-29890125-88clm" Oct 31 00:45:00.392944 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:00.392926 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88t6f\" (UniqueName: \"kubernetes.io/projected/0b644185-b04a-4508-ae1a-61bb035c796e-kube-api-access-88t6f\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-0da668-29890125-jfbgd\" (UID: \"0b644185-b04a-4508-ae1a-61bb035c796e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-0da668-29890125-jfbgd" Oct 31 00:45:00.393079 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:00.392957 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e64541b5-6dc8-4677-aaf3-95422b177fce-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890125-4cnvj\" (UID: \"e64541b5-6dc8-4677-aaf3-95422b177fce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890125-4cnvj" Oct 31 00:45:00.393079 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:00.392990 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf7409ba-8c48-494c-92ac-a1686ce34ad2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-da9d07-29890125-q7hqd\" (UID: \"bf7409ba-8c48-494c-92ac-a1686ce34ad2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da9d07-29890125-q7hqd" Oct 31 00:45:00.393190 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:00.393129 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05aae0c1-03f7-4a08-978e-adcbbd0d1a60-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-1e5e75-29890125-5xl6t\" (UID: \"05aae0c1-03f7-4a08-978e-adcbbd0d1a60\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1e5e75-29890125-5xl6t" Oct 31 00:45:00.393190 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:00.393172 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ns9t\" (UniqueName: \"kubernetes.io/projected/e0edd0ed-32e0-4ba2-8ae0-958c6c530645-kube-api-access-8ns9t\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-d58cd3-29890125-5z9v5\" (UID: \"e0edd0ed-32e0-4ba2-8ae0-958c6c530645\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-d58cd3-29890125-5z9v5" Oct 31 00:45:00.393297 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:00.393249 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jh2dt\" (UniqueName: \"kubernetes.io/projected/ee2582c4-1fff-43ff-b101-98a384127187-kube-api-access-jh2dt\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-e030e0-29890125-88clm\" (UID: \"ee2582c4-1fff-43ff-b101-98a384127187\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-e030e0-29890125-88clm" Oct 31 00:45:00.393297 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:00.393279 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17065afe-2927-4aa9-9dce-9c6c55315d1e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-101b9e-29890125-8899q\" (UID: \"17065afe-2927-4aa9-9dce-9c6c55315d1e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-101b9e-29890125-8899q" Oct 31 00:45:00.393423 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:00.393313 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsncp\" (UniqueName: \"kubernetes.io/projected/05aae0c1-03f7-4a08-978e-adcbbd0d1a60-kube-api-access-gsncp\") pod \"maas-api-key-cleanup-e2e-worker-w3-1e5e75-29890125-5xl6t\" (UID: \"05aae0c1-03f7-4a08-978e-adcbbd0d1a60\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1e5e75-29890125-5xl6t" Oct 31 00:45:00.393423 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:00.393333 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psk6t\" (UniqueName: \"kubernetes.io/projected/9e8880e8-58ba-4e14-b50b-1d66c4765ebd-kube-api-access-psk6t\") pod \"maas-api-key-cleanup-e2e-mt-7d5f4de7-29890125-wn9gz\" (UID: \"9e8880e8-58ba-4e14-b50b-1d66c4765ebd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-7d5f4de7-29890125-wn9gz" Oct 31 00:45:00.393615 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:00.393595 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0edd0ed-32e0-4ba2-8ae0-958c6c530645-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-d58cd3-29890125-5z9v5\" (UID: \"e0edd0ed-32e0-4ba2-8ae0-958c6c530645\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-d58cd3-29890125-5z9v5" Oct 31 00:45:00.393728 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:00.393709 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee2582c4-1fff-43ff-b101-98a384127187-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-e030e0-29890125-88clm\" (UID: \"ee2582c4-1fff-43ff-b101-98a384127187\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-e030e0-29890125-88clm" Oct 31 00:45:00.415426 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:00.415391 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-982ae3-29890125-mjwrq"] Oct 31 00:45:00.456237 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:00.456201 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ns9t\" (UniqueName: \"kubernetes.io/projected/e0edd0ed-32e0-4ba2-8ae0-958c6c530645-kube-api-access-8ns9t\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-d58cd3-29890125-5z9v5\" (UID: \"e0edd0ed-32e0-4ba2-8ae0-958c6c530645\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-d58cd3-29890125-5z9v5" Oct 31 00:45:00.458306 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:00.458277 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jh2dt\" (UniqueName: \"kubernetes.io/projected/ee2582c4-1fff-43ff-b101-98a384127187-kube-api-access-jh2dt\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-e030e0-29890125-88clm\" (UID: \"ee2582c4-1fff-43ff-b101-98a384127187\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-e030e0-29890125-88clm" Oct 31 00:45:00.495207 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:00.495153 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e8880e8-58ba-4e14-b50b-1d66c4765ebd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-7d5f4de7-29890125-wn9gz\" (UID: \"9e8880e8-58ba-4e14-b50b-1d66c4765ebd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-7d5f4de7-29890125-wn9gz" Oct 31 00:45:00.495437 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:00.495225 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pc5w\" (UniqueName: \"kubernetes.io/projected/bf7409ba-8c48-494c-92ac-a1686ce34ad2-kube-api-access-4pc5w\") pod \"maas-api-key-cleanup-e2e-worker-w4-da9d07-29890125-q7hqd\" (UID: \"bf7409ba-8c48-494c-92ac-a1686ce34ad2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da9d07-29890125-q7hqd" Oct 31 00:45:00.495437 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:00.495265 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4sgr\" (UniqueName: \"kubernetes.io/projected/17065afe-2927-4aa9-9dce-9c6c55315d1e-kube-api-access-r4sgr\") pod \"maas-api-key-cleanup-e2e-worker-w0-101b9e-29890125-8899q\" (UID: \"17065afe-2927-4aa9-9dce-9c6c55315d1e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-101b9e-29890125-8899q" Oct 31 00:45:00.495437 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:00.495292 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9753f84f-09ef-4897-b46b-aec7dd93ee82-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-982ae3-29890125-mjwrq\" (UID: \"9753f84f-09ef-4897-b46b-aec7dd93ee82\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-982ae3-29890125-mjwrq" Oct 31 00:45:00.495437 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:00.495346 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88t6f\" (UniqueName: \"kubernetes.io/projected/0b644185-b04a-4508-ae1a-61bb035c796e-kube-api-access-88t6f\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-0da668-29890125-jfbgd\" (UID: \"0b644185-b04a-4508-ae1a-61bb035c796e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-0da668-29890125-jfbgd" Oct 31 00:45:00.495437 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:00.495399 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e64541b5-6dc8-4677-aaf3-95422b177fce-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890125-4cnvj\" (UID: \"e64541b5-6dc8-4677-aaf3-95422b177fce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890125-4cnvj" Oct 31 00:45:00.495437 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:00.495416 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzmdc\" (UniqueName: \"kubernetes.io/projected/9753f84f-09ef-4897-b46b-aec7dd93ee82-kube-api-access-tzmdc\") pod \"maas-api-key-cleanup-e2e-worker-w1-982ae3-29890125-mjwrq\" (UID: \"9753f84f-09ef-4897-b46b-aec7dd93ee82\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-982ae3-29890125-mjwrq" Oct 31 00:45:00.495437 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:00.495439 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf7409ba-8c48-494c-92ac-a1686ce34ad2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-da9d07-29890125-q7hqd\" (UID: \"bf7409ba-8c48-494c-92ac-a1686ce34ad2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da9d07-29890125-q7hqd" Oct 31 00:45:00.495825 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:00.495596 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05aae0c1-03f7-4a08-978e-adcbbd0d1a60-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-1e5e75-29890125-5xl6t\" (UID: \"05aae0c1-03f7-4a08-978e-adcbbd0d1a60\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1e5e75-29890125-5xl6t" Oct 31 00:45:00.495825 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:00.495741 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17065afe-2927-4aa9-9dce-9c6c55315d1e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-101b9e-29890125-8899q\" (UID: \"17065afe-2927-4aa9-9dce-9c6c55315d1e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-101b9e-29890125-8899q" Oct 31 00:45:00.495825 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:00.495799 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsncp\" (UniqueName: \"kubernetes.io/projected/05aae0c1-03f7-4a08-978e-adcbbd0d1a60-kube-api-access-gsncp\") pod \"maas-api-key-cleanup-e2e-worker-w3-1e5e75-29890125-5xl6t\" (UID: \"05aae0c1-03f7-4a08-978e-adcbbd0d1a60\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1e5e75-29890125-5xl6t" Oct 31 00:45:00.495984 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:00.495825 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psk6t\" (UniqueName: \"kubernetes.io/projected/9e8880e8-58ba-4e14-b50b-1d66c4765ebd-kube-api-access-psk6t\") pod \"maas-api-key-cleanup-e2e-mt-7d5f4de7-29890125-wn9gz\" (UID: \"9e8880e8-58ba-4e14-b50b-1d66c4765ebd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-7d5f4de7-29890125-wn9gz" Oct 31 00:45:00.495984 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:00.495869 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvr2v\" (UniqueName: \"kubernetes.io/projected/e64541b5-6dc8-4677-aaf3-95422b177fce-kube-api-access-bvr2v\") pod \"maas-api-key-cleanup-29890125-4cnvj\" (UID: \"e64541b5-6dc8-4677-aaf3-95422b177fce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890125-4cnvj" Oct 31 00:45:00.496089 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:00.495981 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b644185-b04a-4508-ae1a-61bb035c796e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-0da668-29890125-jfbgd\" (UID: \"0b644185-b04a-4508-ae1a-61bb035c796e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-0da668-29890125-jfbgd" Oct 31 00:45:00.496089 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:00.496077 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e8880e8-58ba-4e14-b50b-1d66c4765ebd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-7d5f4de7-29890125-wn9gz\" (UID: \"9e8880e8-58ba-4e14-b50b-1d66c4765ebd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-7d5f4de7-29890125-wn9gz" Oct 31 00:45:00.496197 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:00.496096 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf7409ba-8c48-494c-92ac-a1686ce34ad2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-da9d07-29890125-q7hqd\" (UID: \"bf7409ba-8c48-494c-92ac-a1686ce34ad2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da9d07-29890125-q7hqd" Oct 31 00:45:00.496197 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:00.496153 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e64541b5-6dc8-4677-aaf3-95422b177fce-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890125-4cnvj\" (UID: \"e64541b5-6dc8-4677-aaf3-95422b177fce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890125-4cnvj" Oct 31 00:45:00.496407 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:00.496383 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05aae0c1-03f7-4a08-978e-adcbbd0d1a60-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-1e5e75-29890125-5xl6t\" (UID: \"05aae0c1-03f7-4a08-978e-adcbbd0d1a60\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1e5e75-29890125-5xl6t" Oct 31 00:45:00.496763 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:00.496689 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b644185-b04a-4508-ae1a-61bb035c796e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-0da668-29890125-jfbgd\" (UID: \"0b644185-b04a-4508-ae1a-61bb035c796e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-0da668-29890125-jfbgd" Oct 31 00:45:00.496763 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:00.496740 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17065afe-2927-4aa9-9dce-9c6c55315d1e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-101b9e-29890125-8899q\" (UID: \"17065afe-2927-4aa9-9dce-9c6c55315d1e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-101b9e-29890125-8899q" Oct 31 00:45:00.510262 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:00.510198 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-e030e0-29890125-88clm" Oct 31 00:45:00.518081 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:00.518058 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-d58cd3-29890125-5z9v5" Oct 31 00:45:00.565114 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:00.565068 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvr2v\" (UniqueName: \"kubernetes.io/projected/e64541b5-6dc8-4677-aaf3-95422b177fce-kube-api-access-bvr2v\") pod \"maas-api-key-cleanup-29890125-4cnvj\" (UID: \"e64541b5-6dc8-4677-aaf3-95422b177fce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890125-4cnvj" Oct 31 00:45:00.565283 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:00.565178 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4sgr\" (UniqueName: \"kubernetes.io/projected/17065afe-2927-4aa9-9dce-9c6c55315d1e-kube-api-access-r4sgr\") pod \"maas-api-key-cleanup-e2e-worker-w0-101b9e-29890125-8899q\" (UID: \"17065afe-2927-4aa9-9dce-9c6c55315d1e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-101b9e-29890125-8899q" Oct 31 00:45:00.565819 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:00.565793 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88t6f\" (UniqueName: \"kubernetes.io/projected/0b644185-b04a-4508-ae1a-61bb035c796e-kube-api-access-88t6f\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-0da668-29890125-jfbgd\" (UID: \"0b644185-b04a-4508-ae1a-61bb035c796e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-0da668-29890125-jfbgd" Oct 31 00:45:00.566169 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:00.566149 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsncp\" (UniqueName: \"kubernetes.io/projected/05aae0c1-03f7-4a08-978e-adcbbd0d1a60-kube-api-access-gsncp\") pod \"maas-api-key-cleanup-e2e-worker-w3-1e5e75-29890125-5xl6t\" (UID: \"05aae0c1-03f7-4a08-978e-adcbbd0d1a60\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1e5e75-29890125-5xl6t" Oct 31 00:45:00.571501 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:00.569958 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-psk6t\" (UniqueName: \"kubernetes.io/projected/9e8880e8-58ba-4e14-b50b-1d66c4765ebd-kube-api-access-psk6t\") pod \"maas-api-key-cleanup-e2e-mt-7d5f4de7-29890125-wn9gz\" (UID: \"9e8880e8-58ba-4e14-b50b-1d66c4765ebd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-7d5f4de7-29890125-wn9gz" Oct 31 00:45:00.571649 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:00.571554 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pc5w\" (UniqueName: \"kubernetes.io/projected/bf7409ba-8c48-494c-92ac-a1686ce34ad2-kube-api-access-4pc5w\") pod \"maas-api-key-cleanup-e2e-worker-w4-da9d07-29890125-q7hqd\" (UID: \"bf7409ba-8c48-494c-92ac-a1686ce34ad2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da9d07-29890125-q7hqd" Oct 31 00:45:00.573690 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:00.573666 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1e5e75-29890125-5xl6t" Oct 31 00:45:00.587892 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:00.587633 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890125-4cnvj" Oct 31 00:45:00.605712 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:00.605599 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzmdc\" (UniqueName: \"kubernetes.io/projected/9753f84f-09ef-4897-b46b-aec7dd93ee82-kube-api-access-tzmdc\") pod \"maas-api-key-cleanup-e2e-worker-w1-982ae3-29890125-mjwrq\" (UID: \"9753f84f-09ef-4897-b46b-aec7dd93ee82\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-982ae3-29890125-mjwrq" Oct 31 00:45:00.606710 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:00.606085 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9753f84f-09ef-4897-b46b-aec7dd93ee82-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-982ae3-29890125-mjwrq\" (UID: \"9753f84f-09ef-4897-b46b-aec7dd93ee82\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-982ae3-29890125-mjwrq" Oct 31 00:45:00.607261 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:00.607175 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9753f84f-09ef-4897-b46b-aec7dd93ee82-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-982ae3-29890125-mjwrq\" (UID: \"9753f84f-09ef-4897-b46b-aec7dd93ee82\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-982ae3-29890125-mjwrq" Oct 31 00:45:00.618453 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:00.618426 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da9d07-29890125-q7hqd" Oct 31 00:45:00.704605 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:00.701167 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzmdc\" (UniqueName: \"kubernetes.io/projected/9753f84f-09ef-4897-b46b-aec7dd93ee82-kube-api-access-tzmdc\") pod \"maas-api-key-cleanup-e2e-worker-w1-982ae3-29890125-mjwrq\" (UID: \"9753f84f-09ef-4897-b46b-aec7dd93ee82\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-982ae3-29890125-mjwrq" Oct 31 00:45:00.722322 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:00.722278 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-dcdbd0-openshift-default-86cd956cc9-p8h4z" Oct 31 00:45:00.722608 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:00.722579 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-f891e6-5b8bd8d58f-wpn4q" Oct 31 00:45:00.748778 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:00.748698 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-dcdbd0-openshift-default-86cd956cc9-p8h4z" Oct 31 00:45:00.841163 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:00.840913 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-7d5f4de7-29890125-wn9gz" Oct 31 00:45:00.846067 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:00.845861 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-0da668-29890125-jfbgd" Oct 31 00:45:00.889782 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:00.848812 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-101b9e-29890125-8899q" Oct 31 00:45:00.942939 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:00.942904 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-982ae3-29890125-mjwrq" Oct 31 00:45:01.119975 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:45:01.119942 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee2582c4_1fff_43ff_b101_98a384127187.slice/crio-8f7ac05afa20dfa9cd459c5d66acf32fcf3e16dcfb99c0290f2bd2edbc0e0232 WatchSource:0}: Error finding container 8f7ac05afa20dfa9cd459c5d66acf32fcf3e16dcfb99c0290f2bd2edbc0e0232: Status 404 returned error can't find the container with id 8f7ac05afa20dfa9cd459c5d66acf32fcf3e16dcfb99c0290f2bd2edbc0e0232 Oct 31 00:45:01.129184 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:01.129156 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-e030e0-29890125-88clm"] Oct 31 00:45:01.195415 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:45:01.195365 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode64541b5_6dc8_4677_aaf3_95422b177fce.slice/crio-cc989a09e83b3b7e117d27d6b26886577ce95239b20b49c4482b0d0d67ce2f34 WatchSource:0}: Error finding container cc989a09e83b3b7e117d27d6b26886577ce95239b20b49c4482b0d0d67ce2f34: Status 404 returned error can't find the container with id cc989a09e83b3b7e117d27d6b26886577ce95239b20b49c4482b0d0d67ce2f34 Oct 31 00:45:01.197795 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:01.197769 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-d58cd3-29890125-5z9v5"] Oct 31 00:45:01.204110 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:01.204084 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890125-4cnvj"] Oct 31 00:45:01.209852 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:01.209808 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1e5e75-29890125-5xl6t"] Oct 31 00:45:01.288579 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:01.288548 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da9d07-29890125-q7hqd"] Oct 31 00:45:01.347189 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:45:01.346832 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf7409ba_8c48_494c_92ac_a1686ce34ad2.slice/crio-832b49bb758530d0f335271ef8be89c6588ebd2dcdc0347a59ef74882c4b1b9f WatchSource:0}: Error finding container 832b49bb758530d0f335271ef8be89c6588ebd2dcdc0347a59ef74882c4b1b9f: Status 404 returned error can't find the container with id 832b49bb758530d0f335271ef8be89c6588ebd2dcdc0347a59ef74882c4b1b9f Oct 31 00:45:01.355468 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:01.355443 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-101b9e-29890125-8899q"] Oct 31 00:45:01.359895 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:01.359869 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-7d5f4de7-29890125-wn9gz"] Oct 31 00:45:01.361920 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:45:01.361891 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17065afe_2927_4aa9_9dce_9c6c55315d1e.slice/crio-291f378118834f972cba06a1f7048f5a01c88c71668a06e49a917b4d32f2dd15 WatchSource:0}: Error finding container 291f378118834f972cba06a1f7048f5a01c88c71668a06e49a917b4d32f2dd15: Status 404 returned error can't find the container with id 291f378118834f972cba06a1f7048f5a01c88c71668a06e49a917b4d32f2dd15 Oct 31 00:45:01.396825 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:45:01.396778 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b644185_b04a_4508_ae1a_61bb035c796e.slice/crio-d49c2d902aab9240d922858bca071f974ea22638cba9c91f293874d77dc7d1aa WatchSource:0}: Error finding container d49c2d902aab9240d922858bca071f974ea22638cba9c91f293874d77dc7d1aa: Status 404 returned error can't find the container with id d49c2d902aab9240d922858bca071f974ea22638cba9c91f293874d77dc7d1aa Oct 31 00:45:01.401353 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:01.401322 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-0da668-29890125-jfbgd"] Oct 31 00:45:01.450488 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:01.450458 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-982ae3-29890125-mjwrq"] Oct 31 00:45:01.644041 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:45:01.643960 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9753f84f_09ef_4897_b46b_aec7dd93ee82.slice/crio-3326881aa00685634c94e8510eb614f65c7467c2601b5b764b989cda5ee95c27 WatchSource:0}: Error finding container 3326881aa00685634c94e8510eb614f65c7467c2601b5b764b989cda5ee95c27: Status 404 returned error can't find the container with id 3326881aa00685634c94e8510eb614f65c7467c2601b5b764b989cda5ee95c27 Oct 31 00:45:01.764955 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:01.764924 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1e5e75-29890125-5xl6t" event={"ID":"05aae0c1-03f7-4a08-978e-adcbbd0d1a60","Type":"ContainerStarted","Data":"9212566c28862df3e5845b8b6bd8d02baba4d1d52155382cd13f2061b03b8add"} Oct 31 00:45:01.767032 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:01.766998 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-e030e0-29890125-88clm" event={"ID":"ee2582c4-1fff-43ff-b101-98a384127187","Type":"ContainerStarted","Data":"8f7ac05afa20dfa9cd459c5d66acf32fcf3e16dcfb99c0290f2bd2edbc0e0232"} Oct 31 00:45:01.777696 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:01.773656 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890125-4cnvj" event={"ID":"e64541b5-6dc8-4677-aaf3-95422b177fce","Type":"ContainerStarted","Data":"cc989a09e83b3b7e117d27d6b26886577ce95239b20b49c4482b0d0d67ce2f34"} Oct 31 00:45:01.777696 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:01.776898 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-101b9e-29890125-8899q" event={"ID":"17065afe-2927-4aa9-9dce-9c6c55315d1e","Type":"ContainerStarted","Data":"291f378118834f972cba06a1f7048f5a01c88c71668a06e49a917b4d32f2dd15"} Oct 31 00:45:01.785696 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:01.785612 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da9d07-29890125-q7hqd" event={"ID":"bf7409ba-8c48-494c-92ac-a1686ce34ad2","Type":"ContainerStarted","Data":"832b49bb758530d0f335271ef8be89c6588ebd2dcdc0347a59ef74882c4b1b9f"} Oct 31 00:45:01.788207 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:01.788071 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-982ae3-29890125-mjwrq" event={"ID":"9753f84f-09ef-4897-b46b-aec7dd93ee82","Type":"ContainerStarted","Data":"3326881aa00685634c94e8510eb614f65c7467c2601b5b764b989cda5ee95c27"} Oct 31 00:45:01.790608 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:01.790574 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-0da668-29890125-jfbgd" event={"ID":"0b644185-b04a-4508-ae1a-61bb035c796e","Type":"ContainerStarted","Data":"d49c2d902aab9240d922858bca071f974ea22638cba9c91f293874d77dc7d1aa"} Oct 31 00:45:01.792301 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:01.792274 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-d58cd3-29890125-5z9v5" event={"ID":"e0edd0ed-32e0-4ba2-8ae0-958c6c530645","Type":"ContainerStarted","Data":"09c968c71ea5708a26c819435654aae9079557259402020f17d7a6725af38446"} Oct 31 00:45:01.793740 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:01.793713 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-7d5f4de7-29890125-wn9gz" event={"ID":"9e8880e8-58ba-4e14-b50b-1d66c4765ebd","Type":"ContainerStarted","Data":"5f0ac6b89ad7afee9505e129448678a820fb8c11f906e67043f33f93c79b6da9"} Oct 31 00:45:02.805192 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:02.805100 5222 generic.go:356] "Generic (PLEG): container finished" podID="0b644185-b04a-4508-ae1a-61bb035c796e" containerID="5f41c3da312c25169c271f8638b49d3fd174f366cd45f18e3fc8d200201a6ed3" exitCode=60 Oct 31 00:45:02.805629 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:02.805184 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-0da668-29890125-jfbgd" event={"ID":"0b644185-b04a-4508-ae1a-61bb035c796e","Type":"ContainerDied","Data":"5f41c3da312c25169c271f8638b49d3fd174f366cd45f18e3fc8d200201a6ed3"} Oct 31 00:45:02.805720 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:02.805670 5222 scope.go:122] "RemoveContainer" containerID="5f41c3da312c25169c271f8638b49d3fd174f366cd45f18e3fc8d200201a6ed3" Oct 31 00:45:02.807041 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:02.807016 5222 generic.go:356] "Generic (PLEG): container finished" podID="e0edd0ed-32e0-4ba2-8ae0-958c6c530645" containerID="e74ea503f8510331198f326b0849ea61cbffe60158be4399e2f3ef4c6f6d5a33" exitCode=60 Oct 31 00:45:02.807162 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:02.807098 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-d58cd3-29890125-5z9v5" event={"ID":"e0edd0ed-32e0-4ba2-8ae0-958c6c530645","Type":"ContainerDied","Data":"e74ea503f8510331198f326b0849ea61cbffe60158be4399e2f3ef4c6f6d5a33"} Oct 31 00:45:02.807400 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:02.807350 5222 scope.go:122] "RemoveContainer" containerID="e74ea503f8510331198f326b0849ea61cbffe60158be4399e2f3ef4c6f6d5a33" Oct 31 00:45:02.809185 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:02.809158 5222 generic.go:356] "Generic (PLEG): container finished" podID="9e8880e8-58ba-4e14-b50b-1d66c4765ebd" containerID="ceb0c0934468b4fa1e464aab3f1322f02867989c81e9e26f5eff02ff02551fa0" exitCode=60 Oct 31 00:45:02.809289 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:02.809245 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-7d5f4de7-29890125-wn9gz" event={"ID":"9e8880e8-58ba-4e14-b50b-1d66c4765ebd","Type":"ContainerDied","Data":"ceb0c0934468b4fa1e464aab3f1322f02867989c81e9e26f5eff02ff02551fa0"} Oct 31 00:45:02.809593 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:02.809577 5222 scope.go:122] "RemoveContainer" containerID="ceb0c0934468b4fa1e464aab3f1322f02867989c81e9e26f5eff02ff02551fa0" Oct 31 00:45:02.811253 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:02.811232 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1e5e75-29890125-5xl6t" event={"ID":"05aae0c1-03f7-4a08-978e-adcbbd0d1a60","Type":"ContainerStarted","Data":"e0bca9c0f9c8a29d4114ae08bdaf0a48fe90ff1d36e1be27eda04d00865c1228"} Oct 31 00:45:02.813088 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:02.813068 5222 generic.go:356] "Generic (PLEG): container finished" podID="ee2582c4-1fff-43ff-b101-98a384127187" containerID="1360b4b0096729513b245de25981ab39a92483b61bfe015578e5e5afd670ccf6" exitCode=60 Oct 31 00:45:02.813189 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:02.813130 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-e030e0-29890125-88clm" event={"ID":"ee2582c4-1fff-43ff-b101-98a384127187","Type":"ContainerDied","Data":"1360b4b0096729513b245de25981ab39a92483b61bfe015578e5e5afd670ccf6"} Oct 31 00:45:02.813342 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:02.813329 5222 scope.go:122] "RemoveContainer" containerID="1360b4b0096729513b245de25981ab39a92483b61bfe015578e5e5afd670ccf6" Oct 31 00:45:02.814765 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:02.814745 5222 generic.go:356] "Generic (PLEG): container finished" podID="e64541b5-6dc8-4677-aaf3-95422b177fce" containerID="8303ac8103d8b4ad4eb7279cb2561481976e2d04e8c0a22f698cff2a476fb2d0" exitCode=60 Oct 31 00:45:02.814859 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:02.814824 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890125-4cnvj" event={"ID":"e64541b5-6dc8-4677-aaf3-95422b177fce","Type":"ContainerDied","Data":"8303ac8103d8b4ad4eb7279cb2561481976e2d04e8c0a22f698cff2a476fb2d0"} Oct 31 00:45:02.815027 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:02.815013 5222 scope.go:122] "RemoveContainer" containerID="8303ac8103d8b4ad4eb7279cb2561481976e2d04e8c0a22f698cff2a476fb2d0" Oct 31 00:45:02.816740 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:02.816716 5222 generic.go:356] "Generic (PLEG): container finished" podID="17065afe-2927-4aa9-9dce-9c6c55315d1e" containerID="e275385a0676a26b334a866014fe390cebbb070afc46d95a82fbdb83e1f6e5c2" exitCode=60 Oct 31 00:45:02.816832 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:02.816803 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-101b9e-29890125-8899q" event={"ID":"17065afe-2927-4aa9-9dce-9c6c55315d1e","Type":"ContainerDied","Data":"e275385a0676a26b334a866014fe390cebbb070afc46d95a82fbdb83e1f6e5c2"} Oct 31 00:45:02.817026 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:02.817008 5222 scope.go:122] "RemoveContainer" containerID="e275385a0676a26b334a866014fe390cebbb070afc46d95a82fbdb83e1f6e5c2" Oct 31 00:45:02.820051 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:02.820024 5222 generic.go:356] "Generic (PLEG): container finished" podID="bf7409ba-8c48-494c-92ac-a1686ce34ad2" containerID="4507a54ccaf2ac2b96103a61cfbda9abcf5b376a4678c8531d103e02c2718c60" exitCode=60 Oct 31 00:45:02.820161 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:02.820121 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da9d07-29890125-q7hqd" event={"ID":"bf7409ba-8c48-494c-92ac-a1686ce34ad2","Type":"ContainerDied","Data":"4507a54ccaf2ac2b96103a61cfbda9abcf5b376a4678c8531d103e02c2718c60"} Oct 31 00:45:02.820459 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:02.820443 5222 scope.go:122] "RemoveContainer" containerID="4507a54ccaf2ac2b96103a61cfbda9abcf5b376a4678c8531d103e02c2718c60" Oct 31 00:45:02.821923 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:02.821898 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-982ae3-29890125-mjwrq" event={"ID":"9753f84f-09ef-4897-b46b-aec7dd93ee82","Type":"ContainerStarted","Data":"58bb9a00e831235529e6fe9d295ed4fbd648ca28d387639af23aecabf8c0e2af"} Oct 31 00:45:03.162157 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:03.162093 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-982ae3-29890125-mjwrq" podStartSLOduration=3.162078044 podStartE2EDuration="3.162078044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:45:03.159679036 +0000 UTC m=+1795.630848786" watchObservedRunningTime="2026-10-31 00:45:03.162078044 +0000 UTC m=+1795.633247793" Oct 31 00:45:03.768171 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:03.768116 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1e5e75-29890125-5xl6t" podStartSLOduration=3.768098633 podStartE2EDuration="3.768098633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:45:03.763212256 +0000 UTC m=+1796.234382007" watchObservedRunningTime="2026-10-31 00:45:03.768098633 +0000 UTC m=+1796.239268379" Oct 31 00:45:03.831246 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:03.831209 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-7d5f4de7-29890125-wn9gz" event={"ID":"9e8880e8-58ba-4e14-b50b-1d66c4765ebd","Type":"ContainerStarted","Data":"25918655d66ed10b57e03cc3d0368bdc11dc8cefcdd258f7ac78d239dd93b3f8"} Oct 31 00:45:03.833116 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:03.833088 5222 generic.go:356] "Generic (PLEG): container finished" podID="05aae0c1-03f7-4a08-978e-adcbbd0d1a60" containerID="e0bca9c0f9c8a29d4114ae08bdaf0a48fe90ff1d36e1be27eda04d00865c1228" exitCode=60 Oct 31 00:45:03.833269 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:03.833175 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1e5e75-29890125-5xl6t" event={"ID":"05aae0c1-03f7-4a08-978e-adcbbd0d1a60","Type":"ContainerDied","Data":"e0bca9c0f9c8a29d4114ae08bdaf0a48fe90ff1d36e1be27eda04d00865c1228"} Oct 31 00:45:03.833495 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:03.833481 5222 scope.go:122] "RemoveContainer" containerID="e0bca9c0f9c8a29d4114ae08bdaf0a48fe90ff1d36e1be27eda04d00865c1228" Oct 31 00:45:03.835448 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:03.835310 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-e030e0-29890125-88clm" event={"ID":"ee2582c4-1fff-43ff-b101-98a384127187","Type":"ContainerStarted","Data":"5e94176ca674fc42958f359ee0fd80474c2e82ee64174de6373ad1c5a8bcab60"} Oct 31 00:45:03.837777 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:03.837455 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890125-4cnvj" event={"ID":"e64541b5-6dc8-4677-aaf3-95422b177fce","Type":"ContainerStarted","Data":"2e680bf9b6cd854bcd7dc0dbf26122758ed5b83b3b2cca33e916929a8a20bc8a"} Oct 31 00:45:03.851328 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:03.851289 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-101b9e-29890125-8899q" event={"ID":"17065afe-2927-4aa9-9dce-9c6c55315d1e","Type":"ContainerStarted","Data":"bf42f9442b067c67e6dbd22224f37838f08787a2a2561e4795ea7fcd2eec49cc"} Oct 31 00:45:03.854847 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:03.854817 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da9d07-29890125-q7hqd" event={"ID":"bf7409ba-8c48-494c-92ac-a1686ce34ad2","Type":"ContainerStarted","Data":"d37f289a8b1d3f953537ba8865786123d83d0b93b962c85e6df8ad6af56dd2bc"} Oct 31 00:45:03.857220 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:03.857159 5222 generic.go:356] "Generic (PLEG): container finished" podID="9753f84f-09ef-4897-b46b-aec7dd93ee82" containerID="58bb9a00e831235529e6fe9d295ed4fbd648ca28d387639af23aecabf8c0e2af" exitCode=60 Oct 31 00:45:03.857297 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:03.857217 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-982ae3-29890125-mjwrq" event={"ID":"9753f84f-09ef-4897-b46b-aec7dd93ee82","Type":"ContainerDied","Data":"58bb9a00e831235529e6fe9d295ed4fbd648ca28d387639af23aecabf8c0e2af"} Oct 31 00:45:03.857685 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:03.857665 5222 scope.go:122] "RemoveContainer" containerID="58bb9a00e831235529e6fe9d295ed4fbd648ca28d387639af23aecabf8c0e2af" Oct 31 00:45:04.009867 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:04.009816 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890125-4cnvj" podStartSLOduration=4.00980041 podStartE2EDuration="4.00980041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:45:04.006797322 +0000 UTC m=+1796.477967068" watchObservedRunningTime="2026-10-31 00:45:04.00980041 +0000 UTC m=+1796.480970159" Oct 31 00:45:04.131342 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:04.131285 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-101b9e-29890125-8899q" podStartSLOduration=4.131272442 podStartE2EDuration="4.131272442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:45:04.129874887 +0000 UTC m=+1796.601044637" watchObservedRunningTime="2026-10-31 00:45:04.131272442 +0000 UTC m=+1796.602442191" Oct 31 00:45:04.365363 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:04.365313 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-e030e0-29890125-88clm" podStartSLOduration=4.365298507 podStartE2EDuration="4.365298507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:45:04.364875058 +0000 UTC m=+1796.836044807" watchObservedRunningTime="2026-10-31 00:45:04.365298507 +0000 UTC m=+1796.836468307" Oct 31 00:45:04.483770 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:04.483715 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-7d5f4de7-29890125-wn9gz" podStartSLOduration=4.483701215 podStartE2EDuration="4.483701215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:45:04.482246987 +0000 UTC m=+1796.953416737" watchObservedRunningTime="2026-10-31 00:45:04.483701215 +0000 UTC m=+1796.954870999" Oct 31 00:45:04.720064 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:04.720005 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da9d07-29890125-q7hqd" podStartSLOduration=4.71999116 podStartE2EDuration="4.71999116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:45:04.718192924 +0000 UTC m=+1797.189362672" watchObservedRunningTime="2026-10-31 00:45:04.71999116 +0000 UTC m=+1797.191160909" Oct 31 00:45:04.867923 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:04.867887 5222 generic.go:356] "Generic (PLEG): container finished" podID="bf7409ba-8c48-494c-92ac-a1686ce34ad2" containerID="d37f289a8b1d3f953537ba8865786123d83d0b93b962c85e6df8ad6af56dd2bc" exitCode=60 Oct 31 00:45:04.868411 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:04.867954 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da9d07-29890125-q7hqd" event={"ID":"bf7409ba-8c48-494c-92ac-a1686ce34ad2","Type":"ContainerDied","Data":"d37f289a8b1d3f953537ba8865786123d83d0b93b962c85e6df8ad6af56dd2bc"} Oct 31 00:45:04.868411 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:04.868013 5222 scope.go:122] "RemoveContainer" containerID="4507a54ccaf2ac2b96103a61cfbda9abcf5b376a4678c8531d103e02c2718c60" Oct 31 00:45:04.868575 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:04.868434 5222 scope.go:122] "RemoveContainer" containerID="d37f289a8b1d3f953537ba8865786123d83d0b93b962c85e6df8ad6af56dd2bc" Oct 31 00:45:04.868924 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:45:04.868900 5222 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-da9d07-29890125-q7hqd_odh-ai-gateway-infra(bf7409ba-8c48-494c-92ac-a1686ce34ad2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da9d07-29890125-q7hqd" podUID="bf7409ba-8c48-494c-92ac-a1686ce34ad2" Oct 31 00:45:04.870457 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:04.870434 5222 generic.go:356] "Generic (PLEG): container finished" podID="9753f84f-09ef-4897-b46b-aec7dd93ee82" containerID="dbf989036ad3c81f61f595ecbcfc4fcf12cadbed0fb473fe538854de0b5ed818" exitCode=60 Oct 31 00:45:04.870548 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:04.870515 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-982ae3-29890125-mjwrq" event={"ID":"9753f84f-09ef-4897-b46b-aec7dd93ee82","Type":"ContainerDied","Data":"dbf989036ad3c81f61f595ecbcfc4fcf12cadbed0fb473fe538854de0b5ed818"} Oct 31 00:45:04.870874 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:04.870854 5222 scope.go:122] "RemoveContainer" containerID="dbf989036ad3c81f61f595ecbcfc4fcf12cadbed0fb473fe538854de0b5ed818" Oct 31 00:45:04.871270 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:45:04.871247 5222 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-982ae3-29890125-mjwrq_odh-ai-gateway-infra(9753f84f-09ef-4897-b46b-aec7dd93ee82)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-982ae3-29890125-mjwrq" podUID="9753f84f-09ef-4897-b46b-aec7dd93ee82" Oct 31 00:45:04.872478 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:04.872456 5222 generic.go:356] "Generic (PLEG): container finished" podID="0b644185-b04a-4508-ae1a-61bb035c796e" containerID="552fc656ebb8c96e664c216d41b56ffe5789491597f632fbcc99468cfb02fa42" exitCode=60 Oct 31 00:45:04.872586 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:04.872525 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-0da668-29890125-jfbgd" event={"ID":"0b644185-b04a-4508-ae1a-61bb035c796e","Type":"ContainerDied","Data":"552fc656ebb8c96e664c216d41b56ffe5789491597f632fbcc99468cfb02fa42"} Oct 31 00:45:04.872842 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:04.872821 5222 scope.go:122] "RemoveContainer" containerID="552fc656ebb8c96e664c216d41b56ffe5789491597f632fbcc99468cfb02fa42" Oct 31 00:45:04.873186 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:45:04.873166 5222 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-0da668-29890125-jfbgd_odh-ai-gateway-infra(0b644185-b04a-4508-ae1a-61bb035c796e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-0da668-29890125-jfbgd" podUID="0b644185-b04a-4508-ae1a-61bb035c796e" Oct 31 00:45:04.874606 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:04.874581 5222 generic.go:356] "Generic (PLEG): container finished" podID="e0edd0ed-32e0-4ba2-8ae0-958c6c530645" containerID="4f8fae5465be21ca0061cd69e92270a66cb995ba574ff589adf342ba12dd5b0a" exitCode=60 Oct 31 00:45:04.874719 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:04.874664 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-d58cd3-29890125-5z9v5" event={"ID":"e0edd0ed-32e0-4ba2-8ae0-958c6c530645","Type":"ContainerDied","Data":"4f8fae5465be21ca0061cd69e92270a66cb995ba574ff589adf342ba12dd5b0a"} Oct 31 00:45:04.874923 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:04.874906 5222 scope.go:122] "RemoveContainer" containerID="4f8fae5465be21ca0061cd69e92270a66cb995ba574ff589adf342ba12dd5b0a" Oct 31 00:45:04.875236 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:45:04.875215 5222 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-d58cd3-29890125-5z9v5_odh-ai-gateway-infra(e0edd0ed-32e0-4ba2-8ae0-958c6c530645)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-d58cd3-29890125-5z9v5" podUID="e0edd0ed-32e0-4ba2-8ae0-958c6c530645" Oct 31 00:45:04.876699 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:04.876677 5222 generic.go:356] "Generic (PLEG): container finished" podID="9e8880e8-58ba-4e14-b50b-1d66c4765ebd" containerID="25918655d66ed10b57e03cc3d0368bdc11dc8cefcdd258f7ac78d239dd93b3f8" exitCode=60 Oct 31 00:45:04.876790 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:04.876754 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-7d5f4de7-29890125-wn9gz" event={"ID":"9e8880e8-58ba-4e14-b50b-1d66c4765ebd","Type":"ContainerDied","Data":"25918655d66ed10b57e03cc3d0368bdc11dc8cefcdd258f7ac78d239dd93b3f8"} Oct 31 00:45:04.877010 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:04.876994 5222 scope.go:122] "RemoveContainer" containerID="25918655d66ed10b57e03cc3d0368bdc11dc8cefcdd258f7ac78d239dd93b3f8" Oct 31 00:45:04.877300 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:45:04.877278 5222 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-7d5f4de7-29890125-wn9gz_odh-ai-gateway-infra(9e8880e8-58ba-4e14-b50b-1d66c4765ebd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-7d5f4de7-29890125-wn9gz" podUID="9e8880e8-58ba-4e14-b50b-1d66c4765ebd" Oct 31 00:45:04.878898 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:04.878876 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1e5e75-29890125-5xl6t" event={"ID":"05aae0c1-03f7-4a08-978e-adcbbd0d1a60","Type":"ContainerStarted","Data":"59855b1f4a9823241b6676e0d5643b0cdb142b69f4a3a8401af3729844307976"} Oct 31 00:45:04.880795 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:04.880765 5222 generic.go:356] "Generic (PLEG): container finished" podID="ee2582c4-1fff-43ff-b101-98a384127187" containerID="5e94176ca674fc42958f359ee0fd80474c2e82ee64174de6373ad1c5a8bcab60" exitCode=60 Oct 31 00:45:04.880876 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:04.880817 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-e030e0-29890125-88clm" event={"ID":"ee2582c4-1fff-43ff-b101-98a384127187","Type":"ContainerDied","Data":"5e94176ca674fc42958f359ee0fd80474c2e82ee64174de6373ad1c5a8bcab60"} Oct 31 00:45:04.881093 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:04.881075 5222 scope.go:122] "RemoveContainer" containerID="5e94176ca674fc42958f359ee0fd80474c2e82ee64174de6373ad1c5a8bcab60" Oct 31 00:45:04.881401 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:45:04.881359 5222 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-e030e0-29890125-88clm_odh-ai-gateway-infra(ee2582c4-1fff-43ff-b101-98a384127187)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-e030e0-29890125-88clm" podUID="ee2582c4-1fff-43ff-b101-98a384127187" Oct 31 00:45:04.882985 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:04.882961 5222 generic.go:356] "Generic (PLEG): container finished" podID="e64541b5-6dc8-4677-aaf3-95422b177fce" containerID="2e680bf9b6cd854bcd7dc0dbf26122758ed5b83b3b2cca33e916929a8a20bc8a" exitCode=60 Oct 31 00:45:04.883075 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:04.883040 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890125-4cnvj" event={"ID":"e64541b5-6dc8-4677-aaf3-95422b177fce","Type":"ContainerDied","Data":"2e680bf9b6cd854bcd7dc0dbf26122758ed5b83b3b2cca33e916929a8a20bc8a"} Oct 31 00:45:04.883272 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:04.883258 5222 scope.go:122] "RemoveContainer" containerID="2e680bf9b6cd854bcd7dc0dbf26122758ed5b83b3b2cca33e916929a8a20bc8a" Oct 31 00:45:04.883601 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:45:04.883569 5222 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890125-4cnvj_odh-ai-gateway-infra(e64541b5-6dc8-4677-aaf3-95422b177fce)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890125-4cnvj" podUID="e64541b5-6dc8-4677-aaf3-95422b177fce" Oct 31 00:45:04.883601 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:04.883586 5222 scope.go:122] "RemoveContainer" containerID="58bb9a00e831235529e6fe9d295ed4fbd648ca28d387639af23aecabf8c0e2af" Oct 31 00:45:04.884904 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:04.884886 5222 generic.go:356] "Generic (PLEG): container finished" podID="17065afe-2927-4aa9-9dce-9c6c55315d1e" containerID="bf42f9442b067c67e6dbd22224f37838f08787a2a2561e4795ea7fcd2eec49cc" exitCode=60 Oct 31 00:45:04.884976 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:04.884935 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-101b9e-29890125-8899q" event={"ID":"17065afe-2927-4aa9-9dce-9c6c55315d1e","Type":"ContainerDied","Data":"bf42f9442b067c67e6dbd22224f37838f08787a2a2561e4795ea7fcd2eec49cc"} Oct 31 00:45:04.885229 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:04.885211 5222 scope.go:122] "RemoveContainer" containerID="bf42f9442b067c67e6dbd22224f37838f08787a2a2561e4795ea7fcd2eec49cc" Oct 31 00:45:04.885518 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:45:04.885500 5222 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-101b9e-29890125-8899q_odh-ai-gateway-infra(17065afe-2927-4aa9-9dce-9c6c55315d1e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-101b9e-29890125-8899q" podUID="17065afe-2927-4aa9-9dce-9c6c55315d1e" Oct 31 00:45:04.898551 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:04.898520 5222 scope.go:122] "RemoveContainer" containerID="5f41c3da312c25169c271f8638b49d3fd174f366cd45f18e3fc8d200201a6ed3" Oct 31 00:45:04.913833 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:04.913803 5222 scope.go:122] "RemoveContainer" containerID="e74ea503f8510331198f326b0849ea61cbffe60158be4399e2f3ef4c6f6d5a33" Oct 31 00:45:04.928396 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:04.928358 5222 scope.go:122] "RemoveContainer" containerID="ceb0c0934468b4fa1e464aab3f1322f02867989c81e9e26f5eff02ff02551fa0" Oct 31 00:45:04.943913 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:04.943889 5222 scope.go:122] "RemoveContainer" containerID="1360b4b0096729513b245de25981ab39a92483b61bfe015578e5e5afd670ccf6" Oct 31 00:45:05.053439 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:05.053412 5222 scope.go:122] "RemoveContainer" containerID="8303ac8103d8b4ad4eb7279cb2561481976e2d04e8c0a22f698cff2a476fb2d0" Oct 31 00:45:05.068038 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:05.068014 5222 scope.go:122] "RemoveContainer" containerID="e275385a0676a26b334a866014fe390cebbb070afc46d95a82fbdb83e1f6e5c2" Oct 31 00:45:05.893007 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:45:05.892969 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 00:45:05.896199 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:05.896172 5222 scope.go:122] "RemoveContainer" containerID="552fc656ebb8c96e664c216d41b56ffe5789491597f632fbcc99468cfb02fa42" Oct 31 00:45:05.896641 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:45:05.896607 5222 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-0da668-29890125-jfbgd_odh-ai-gateway-infra(0b644185-b04a-4508-ae1a-61bb035c796e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-0da668-29890125-jfbgd" podUID="0b644185-b04a-4508-ae1a-61bb035c796e" Oct 31 00:45:05.898093 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:05.898073 5222 scope.go:122] "RemoveContainer" containerID="4f8fae5465be21ca0061cd69e92270a66cb995ba574ff589adf342ba12dd5b0a" Oct 31 00:45:05.898448 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:45:05.898424 5222 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-d58cd3-29890125-5z9v5_odh-ai-gateway-infra(e0edd0ed-32e0-4ba2-8ae0-958c6c530645)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-d58cd3-29890125-5z9v5" podUID="e0edd0ed-32e0-4ba2-8ae0-958c6c530645" Oct 31 00:45:05.899822 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:05.899807 5222 scope.go:122] "RemoveContainer" containerID="25918655d66ed10b57e03cc3d0368bdc11dc8cefcdd258f7ac78d239dd93b3f8" Oct 31 00:45:05.900101 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:45:05.900079 5222 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-7d5f4de7-29890125-wn9gz_odh-ai-gateway-infra(9e8880e8-58ba-4e14-b50b-1d66c4765ebd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-7d5f4de7-29890125-wn9gz" podUID="9e8880e8-58ba-4e14-b50b-1d66c4765ebd" Oct 31 00:45:05.901592 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:05.901570 5222 generic.go:356] "Generic (PLEG): container finished" podID="05aae0c1-03f7-4a08-978e-adcbbd0d1a60" containerID="59855b1f4a9823241b6676e0d5643b0cdb142b69f4a3a8401af3729844307976" exitCode=60 Oct 31 00:45:05.901677 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:05.901658 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1e5e75-29890125-5xl6t" event={"ID":"05aae0c1-03f7-4a08-978e-adcbbd0d1a60","Type":"ContainerDied","Data":"59855b1f4a9823241b6676e0d5643b0cdb142b69f4a3a8401af3729844307976"} Oct 31 00:45:05.901735 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:05.901713 5222 scope.go:122] "RemoveContainer" containerID="e0bca9c0f9c8a29d4114ae08bdaf0a48fe90ff1d36e1be27eda04d00865c1228" Oct 31 00:45:05.901854 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:05.901832 5222 scope.go:122] "RemoveContainer" containerID="59855b1f4a9823241b6676e0d5643b0cdb142b69f4a3a8401af3729844307976" Oct 31 00:45:05.902108 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:45:05.902089 5222 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-1e5e75-29890125-5xl6t_odh-ai-gateway-infra(05aae0c1-03f7-4a08-978e-adcbbd0d1a60)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1e5e75-29890125-5xl6t" podUID="05aae0c1-03f7-4a08-978e-adcbbd0d1a60" Oct 31 00:45:05.903703 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:05.903682 5222 scope.go:122] "RemoveContainer" containerID="5e94176ca674fc42958f359ee0fd80474c2e82ee64174de6373ad1c5a8bcab60" Oct 31 00:45:05.903977 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:45:05.903953 5222 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-e030e0-29890125-88clm_odh-ai-gateway-infra(ee2582c4-1fff-43ff-b101-98a384127187)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-e030e0-29890125-88clm" podUID="ee2582c4-1fff-43ff-b101-98a384127187" Oct 31 00:45:05.905353 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:05.905337 5222 scope.go:122] "RemoveContainer" containerID="2e680bf9b6cd854bcd7dc0dbf26122758ed5b83b3b2cca33e916929a8a20bc8a" Oct 31 00:45:05.905695 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:45:05.905672 5222 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890125-4cnvj_odh-ai-gateway-infra(e64541b5-6dc8-4677-aaf3-95422b177fce)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890125-4cnvj" podUID="e64541b5-6dc8-4677-aaf3-95422b177fce" Oct 31 00:45:05.906930 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:05.906913 5222 scope.go:122] "RemoveContainer" containerID="bf42f9442b067c67e6dbd22224f37838f08787a2a2561e4795ea7fcd2eec49cc" Oct 31 00:45:05.907197 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:45:05.907178 5222 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-101b9e-29890125-8899q_odh-ai-gateway-infra(17065afe-2927-4aa9-9dce-9c6c55315d1e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-101b9e-29890125-8899q" podUID="17065afe-2927-4aa9-9dce-9c6c55315d1e" Oct 31 00:45:05.908656 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:05.908638 5222 scope.go:122] "RemoveContainer" containerID="d37f289a8b1d3f953537ba8865786123d83d0b93b962c85e6df8ad6af56dd2bc" Oct 31 00:45:05.908884 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:45:05.908863 5222 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-da9d07-29890125-q7hqd_odh-ai-gateway-infra(bf7409ba-8c48-494c-92ac-a1686ce34ad2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da9d07-29890125-q7hqd" podUID="bf7409ba-8c48-494c-92ac-a1686ce34ad2" Oct 31 00:45:05.910253 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:05.910235 5222 scope.go:122] "RemoveContainer" containerID="dbf989036ad3c81f61f595ecbcfc4fcf12cadbed0fb473fe538854de0b5ed818" Oct 31 00:45:05.910481 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:45:05.910464 5222 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-982ae3-29890125-mjwrq_odh-ai-gateway-infra(9753f84f-09ef-4897-b46b-aec7dd93ee82)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-982ae3-29890125-mjwrq" podUID="9753f84f-09ef-4897-b46b-aec7dd93ee82" Oct 31 00:45:06.958727 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:06.958693 5222 scope.go:122] "RemoveContainer" containerID="59855b1f4a9823241b6676e0d5643b0cdb142b69f4a3a8401af3729844307976" Oct 31 00:45:06.959228 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:45:06.959108 5222 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-1e5e75-29890125-5xl6t_odh-ai-gateway-infra(05aae0c1-03f7-4a08-978e-adcbbd0d1a60)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1e5e75-29890125-5xl6t" podUID="05aae0c1-03f7-4a08-978e-adcbbd0d1a60" Oct 31 00:45:07.095288 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:07.095256 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-dcdbd0-7cc8947c99-k4hnc"] Oct 31 00:45:07.101243 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:07.101222 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-dcdbd0-7cc8947c99-k4hnc" Oct 31 00:45:07.153004 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:07.152953 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjlwb\" (UniqueName: \"kubernetes.io/projected/bb6ff052-f89c-4b7f-945f-01ffd909308c-kube-api-access-bjlwb\") pod \"payload-pre-processing-e2e-shared-b-w2-dcdbd0-7cc8947c99-k4hnc\" (UID: \"bb6ff052-f89c-4b7f-945f-01ffd909308c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-dcdbd0-7cc8947c99-k4hnc" Oct 31 00:45:07.153202 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:07.153185 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb6ff052-f89c-4b7f-945f-01ffd909308c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-dcdbd0-7cc8947c99-k4hnc\" (UID: \"bb6ff052-f89c-4b7f-945f-01ffd909308c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-dcdbd0-7cc8947c99-k4hnc" Oct 31 00:45:07.156247 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:07.156216 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-dcdbd0-7cc8947c99-k4hnc"] Oct 31 00:45:07.168270 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:07.157678 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-dcdbd0\"" Oct 31 00:45:07.168270 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:07.157815 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-dcdbd0-dockercfg-vxh6s\"" Oct 31 00:45:07.286446 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:07.275468 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb6ff052-f89c-4b7f-945f-01ffd909308c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-dcdbd0-7cc8947c99-k4hnc\" (UID: \"bb6ff052-f89c-4b7f-945f-01ffd909308c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-dcdbd0-7cc8947c99-k4hnc" Oct 31 00:45:07.286446 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:07.275697 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjlwb\" (UniqueName: \"kubernetes.io/projected/bb6ff052-f89c-4b7f-945f-01ffd909308c-kube-api-access-bjlwb\") pod \"payload-pre-processing-e2e-shared-b-w2-dcdbd0-7cc8947c99-k4hnc\" (UID: \"bb6ff052-f89c-4b7f-945f-01ffd909308c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-dcdbd0-7cc8947c99-k4hnc" Oct 31 00:45:07.286446 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:07.276822 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb6ff052-f89c-4b7f-945f-01ffd909308c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-dcdbd0-7cc8947c99-k4hnc\" (UID: \"bb6ff052-f89c-4b7f-945f-01ffd909308c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-dcdbd0-7cc8947c99-k4hnc" Oct 31 00:45:07.296556 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:07.296516 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-dcdbd0-bc7d86cfc-rs4qx"] Oct 31 00:45:07.315306 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:07.315277 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-dcdbd0-bc7d86cfc-rs4qx" Oct 31 00:45:07.387434 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:07.384030 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjlwb\" (UniqueName: \"kubernetes.io/projected/bb6ff052-f89c-4b7f-945f-01ffd909308c-kube-api-access-bjlwb\") pod \"payload-pre-processing-e2e-shared-b-w2-dcdbd0-7cc8947c99-k4hnc\" (UID: \"bb6ff052-f89c-4b7f-945f-01ffd909308c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-dcdbd0-7cc8947c99-k4hnc" Oct 31 00:45:07.400409 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:07.389556 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-dcdbd0-bc7d86cfc-rs4qx"] Oct 31 00:45:07.417290 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:07.417257 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-dcdbd0-7cc8947c99-k4hnc" Oct 31 00:45:07.481988 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:07.481949 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8899ca28-d521-4591-b372-dcef03e946e1-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-dcdbd0-bc7d86cfc-rs4qx\" (UID: \"8899ca28-d521-4591-b372-dcef03e946e1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-dcdbd0-bc7d86cfc-rs4qx" Oct 31 00:45:07.482269 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:07.482250 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwd7h\" (UniqueName: \"kubernetes.io/projected/8899ca28-d521-4591-b372-dcef03e946e1-kube-api-access-nwd7h\") pod \"payload-processing-e2e-shared-b-w2-dcdbd0-bc7d86cfc-rs4qx\" (UID: \"8899ca28-d521-4591-b372-dcef03e946e1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-dcdbd0-bc7d86cfc-rs4qx" Oct 31 00:45:07.583639 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:07.583541 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwd7h\" (UniqueName: \"kubernetes.io/projected/8899ca28-d521-4591-b372-dcef03e946e1-kube-api-access-nwd7h\") pod \"payload-processing-e2e-shared-b-w2-dcdbd0-bc7d86cfc-rs4qx\" (UID: \"8899ca28-d521-4591-b372-dcef03e946e1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-dcdbd0-bc7d86cfc-rs4qx" Oct 31 00:45:07.583818 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:07.583725 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8899ca28-d521-4591-b372-dcef03e946e1-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-dcdbd0-bc7d86cfc-rs4qx\" (UID: \"8899ca28-d521-4591-b372-dcef03e946e1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-dcdbd0-bc7d86cfc-rs4qx" Oct 31 00:45:07.584630 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:07.584607 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8899ca28-d521-4591-b372-dcef03e946e1-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-dcdbd0-bc7d86cfc-rs4qx\" (UID: \"8899ca28-d521-4591-b372-dcef03e946e1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-dcdbd0-bc7d86cfc-rs4qx" Oct 31 00:45:07.647380 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:07.647334 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwd7h\" (UniqueName: \"kubernetes.io/projected/8899ca28-d521-4591-b372-dcef03e946e1-kube-api-access-nwd7h\") pod \"payload-processing-e2e-shared-b-w2-dcdbd0-bc7d86cfc-rs4qx\" (UID: \"8899ca28-d521-4591-b372-dcef03e946e1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-dcdbd0-bc7d86cfc-rs4qx" Oct 31 00:45:07.680535 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:07.680501 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-dcdbd0-bc7d86cfc-rs4qx" Oct 31 00:45:07.822527 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:45:07.822486 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb6ff052_f89c_4b7f_945f_01ffd909308c.slice/crio-95fd6f16f6ffa572c4cb4024218c79d1ad35b92f67df3cf1d45d57117524ed6e WatchSource:0}: Error finding container 95fd6f16f6ffa572c4cb4024218c79d1ad35b92f67df3cf1d45d57117524ed6e: Status 404 returned error can't find the container with id 95fd6f16f6ffa572c4cb4024218c79d1ad35b92f67df3cf1d45d57117524ed6e Oct 31 00:45:07.826713 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:07.826683 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-dcdbd0-7cc8947c99-k4hnc"] Oct 31 00:45:07.925625 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:07.925596 5222 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h9hsf_0f8c3c03-a1fc-4376-98c6-feeeefa08068/ovn-acl-logging/0.log" Oct 31 00:45:07.931938 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:07.931915 5222 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h9hsf_0f8c3c03-a1fc-4376-98c6-feeeefa08068/ovn-acl-logging/0.log" Oct 31 00:45:07.967905 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:07.967873 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-dcdbd0-7cc8947c99-k4hnc" event={"ID":"bb6ff052-f89c-4b7f-945f-01ffd909308c","Type":"ContainerStarted","Data":"1d1a4dc4adedb20f0377dc7d82e5807701afda3799a04c7dd514c5f4d6cdf98c"} Oct 31 00:45:07.968302 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:07.967924 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-dcdbd0-7cc8947c99-k4hnc" event={"ID":"bb6ff052-f89c-4b7f-945f-01ffd909308c","Type":"ContainerStarted","Data":"95fd6f16f6ffa572c4cb4024218c79d1ad35b92f67df3cf1d45d57117524ed6e"} Oct 31 00:45:07.968302 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:07.967985 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-dcdbd0-7cc8947c99-k4hnc" Oct 31 00:45:08.077735 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:45:08.077694 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8899ca28_d521_4591_b372_dcef03e946e1.slice/crio-fa290622518adecc96355a6282f549a8e33a8629f1129e2a8fec58ad27b6cfcb WatchSource:0}: Error finding container fa290622518adecc96355a6282f549a8e33a8629f1129e2a8fec58ad27b6cfcb: Status 404 returned error can't find the container with id fa290622518adecc96355a6282f549a8e33a8629f1129e2a8fec58ad27b6cfcb Oct 31 00:45:08.080912 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:08.080888 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-dcdbd0-bc7d86cfc-rs4qx"] Oct 31 00:45:08.977222 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:08.977185 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-dcdbd0-bc7d86cfc-rs4qx" event={"ID":"8899ca28-d521-4591-b372-dcef03e946e1","Type":"ContainerStarted","Data":"b311048e31633670fe1f6caffdfe4629820a8ca6413b72583f6476e109a0444a"} Oct 31 00:45:08.977741 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:08.977245 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-dcdbd0-bc7d86cfc-rs4qx" event={"ID":"8899ca28-d521-4591-b372-dcef03e946e1","Type":"ContainerStarted","Data":"fa290622518adecc96355a6282f549a8e33a8629f1129e2a8fec58ad27b6cfcb"} Oct 31 00:45:08.977741 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:08.977311 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-dcdbd0-bc7d86cfc-rs4qx" Oct 31 00:45:09.100870 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:09.100819 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-dcdbd0-bc7d86cfc-rs4qx" podStartSLOduration=2.100802749 podStartE2EDuration="2.100802749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:45:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:45:09.097648233 +0000 UTC m=+1801.568817983" watchObservedRunningTime="2026-10-31 00:45:09.100802749 +0000 UTC m=+1801.571972498" Oct 31 00:45:09.101260 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:09.101240 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-dcdbd0-7cc8947c99-k4hnc" podStartSLOduration=2.101233089 podStartE2EDuration="2.101233089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:45:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:45:08.092089797 +0000 UTC m=+1800.563259547" watchObservedRunningTime="2026-10-31 00:45:09.101233089 +0000 UTC m=+1801.572402838" Oct 31 00:45:09.677212 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:09.677182 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f891e6-5cf4b56747-st4dd" Oct 31 00:45:10.822530 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:10.822497 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-7d5f4de7/e2e-lifecycle-model-7d5f4de7-kserve-799dd698b9-p8scb"] Oct 31 00:45:10.829087 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:10.829065 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-7d5f4de7/e2e-lifecycle-model-7d5f4de7-kserve-799dd698b9-p8scb" Oct 31 00:45:10.883662 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:10.883623 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-7d5f4de7/e2e-lifecycle-model-7d5f4de7-kserve-799dd698b9-p8scb"] Oct 31 00:45:10.930616 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:10.930581 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-7d5f4de7\"/\"kube-root-ca.crt\"" Oct 31 00:45:10.930854 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:10.930833 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-7d5f4de7\"/\"openshift-service-ca.crt\"" Oct 31 00:45:10.934229 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:10.934208 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-7d5f4de7\"/\"e2e-lifecycle-model-7d5f4de7-kserve-self-signed-certs\"" Oct 31 00:45:10.934229 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:10.934207 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-7d5f4de7\"/\"default-dockercfg-f8gwc\"" Oct 31 00:45:10.940079 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:10.940054 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8849902-5dbf-4045-97e8-94cd53c64a43-home\") pod \"e2e-lifecycle-model-7d5f4de7-kserve-799dd698b9-p8scb\" (UID: \"e8849902-5dbf-4045-97e8-94cd53c64a43\") " pod="ai-tenant-e2e-mt-7d5f4de7/e2e-lifecycle-model-7d5f4de7-kserve-799dd698b9-p8scb" Oct 31 00:45:10.940161 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:10.940105 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8849902-5dbf-4045-97e8-94cd53c64a43-tls-certs\") pod \"e2e-lifecycle-model-7d5f4de7-kserve-799dd698b9-p8scb\" (UID: \"e8849902-5dbf-4045-97e8-94cd53c64a43\") " pod="ai-tenant-e2e-mt-7d5f4de7/e2e-lifecycle-model-7d5f4de7-kserve-799dd698b9-p8scb" Oct 31 00:45:10.940240 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:10.940223 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8849902-5dbf-4045-97e8-94cd53c64a43-tmp-dir\") pod \"e2e-lifecycle-model-7d5f4de7-kserve-799dd698b9-p8scb\" (UID: \"e8849902-5dbf-4045-97e8-94cd53c64a43\") " pod="ai-tenant-e2e-mt-7d5f4de7/e2e-lifecycle-model-7d5f4de7-kserve-799dd698b9-p8scb" Oct 31 00:45:10.940317 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:10.940304 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8849902-5dbf-4045-97e8-94cd53c64a43-dshm\") pod \"e2e-lifecycle-model-7d5f4de7-kserve-799dd698b9-p8scb\" (UID: \"e8849902-5dbf-4045-97e8-94cd53c64a43\") " pod="ai-tenant-e2e-mt-7d5f4de7/e2e-lifecycle-model-7d5f4de7-kserve-799dd698b9-p8scb" Oct 31 00:45:10.940351 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:10.940331 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lctf\" (UniqueName: \"kubernetes.io/projected/e8849902-5dbf-4045-97e8-94cd53c64a43-kube-api-access-5lctf\") pod \"e2e-lifecycle-model-7d5f4de7-kserve-799dd698b9-p8scb\" (UID: \"e8849902-5dbf-4045-97e8-94cd53c64a43\") " pod="ai-tenant-e2e-mt-7d5f4de7/e2e-lifecycle-model-7d5f4de7-kserve-799dd698b9-p8scb" Oct 31 00:45:10.940486 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:10.940472 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8849902-5dbf-4045-97e8-94cd53c64a43-model-cache\") pod \"e2e-lifecycle-model-7d5f4de7-kserve-799dd698b9-p8scb\" (UID: \"e8849902-5dbf-4045-97e8-94cd53c64a43\") " pod="ai-tenant-e2e-mt-7d5f4de7/e2e-lifecycle-model-7d5f4de7-kserve-799dd698b9-p8scb" Oct 31 00:45:11.042472 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:11.042433 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8849902-5dbf-4045-97e8-94cd53c64a43-model-cache\") pod \"e2e-lifecycle-model-7d5f4de7-kserve-799dd698b9-p8scb\" (UID: \"e8849902-5dbf-4045-97e8-94cd53c64a43\") " pod="ai-tenant-e2e-mt-7d5f4de7/e2e-lifecycle-model-7d5f4de7-kserve-799dd698b9-p8scb" Oct 31 00:45:11.042703 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:11.042683 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8849902-5dbf-4045-97e8-94cd53c64a43-home\") pod \"e2e-lifecycle-model-7d5f4de7-kserve-799dd698b9-p8scb\" (UID: \"e8849902-5dbf-4045-97e8-94cd53c64a43\") " pod="ai-tenant-e2e-mt-7d5f4de7/e2e-lifecycle-model-7d5f4de7-kserve-799dd698b9-p8scb" Oct 31 00:45:11.042766 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:11.042746 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8849902-5dbf-4045-97e8-94cd53c64a43-tls-certs\") pod \"e2e-lifecycle-model-7d5f4de7-kserve-799dd698b9-p8scb\" (UID: \"e8849902-5dbf-4045-97e8-94cd53c64a43\") " pod="ai-tenant-e2e-mt-7d5f4de7/e2e-lifecycle-model-7d5f4de7-kserve-799dd698b9-p8scb" Oct 31 00:45:11.042833 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:11.042818 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8849902-5dbf-4045-97e8-94cd53c64a43-tmp-dir\") pod \"e2e-lifecycle-model-7d5f4de7-kserve-799dd698b9-p8scb\" (UID: \"e8849902-5dbf-4045-97e8-94cd53c64a43\") " pod="ai-tenant-e2e-mt-7d5f4de7/e2e-lifecycle-model-7d5f4de7-kserve-799dd698b9-p8scb" Oct 31 00:45:11.042897 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:11.042882 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8849902-5dbf-4045-97e8-94cd53c64a43-dshm\") pod \"e2e-lifecycle-model-7d5f4de7-kserve-799dd698b9-p8scb\" (UID: \"e8849902-5dbf-4045-97e8-94cd53c64a43\") " pod="ai-tenant-e2e-mt-7d5f4de7/e2e-lifecycle-model-7d5f4de7-kserve-799dd698b9-p8scb" Oct 31 00:45:11.042953 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:11.042897 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8849902-5dbf-4045-97e8-94cd53c64a43-model-cache\") pod \"e2e-lifecycle-model-7d5f4de7-kserve-799dd698b9-p8scb\" (UID: \"e8849902-5dbf-4045-97e8-94cd53c64a43\") " pod="ai-tenant-e2e-mt-7d5f4de7/e2e-lifecycle-model-7d5f4de7-kserve-799dd698b9-p8scb" Oct 31 00:45:11.042953 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:11.042918 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lctf\" (UniqueName: \"kubernetes.io/projected/e8849902-5dbf-4045-97e8-94cd53c64a43-kube-api-access-5lctf\") pod \"e2e-lifecycle-model-7d5f4de7-kserve-799dd698b9-p8scb\" (UID: \"e8849902-5dbf-4045-97e8-94cd53c64a43\") " pod="ai-tenant-e2e-mt-7d5f4de7/e2e-lifecycle-model-7d5f4de7-kserve-799dd698b9-p8scb" Oct 31 00:45:11.043121 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:11.043097 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8849902-5dbf-4045-97e8-94cd53c64a43-home\") pod \"e2e-lifecycle-model-7d5f4de7-kserve-799dd698b9-p8scb\" (UID: \"e8849902-5dbf-4045-97e8-94cd53c64a43\") " pod="ai-tenant-e2e-mt-7d5f4de7/e2e-lifecycle-model-7d5f4de7-kserve-799dd698b9-p8scb" Oct 31 00:45:11.043176 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:11.043117 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8849902-5dbf-4045-97e8-94cd53c64a43-tmp-dir\") pod \"e2e-lifecycle-model-7d5f4de7-kserve-799dd698b9-p8scb\" (UID: \"e8849902-5dbf-4045-97e8-94cd53c64a43\") " pod="ai-tenant-e2e-mt-7d5f4de7/e2e-lifecycle-model-7d5f4de7-kserve-799dd698b9-p8scb" Oct 31 00:45:11.045244 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:11.045219 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8849902-5dbf-4045-97e8-94cd53c64a43-dshm\") pod \"e2e-lifecycle-model-7d5f4de7-kserve-799dd698b9-p8scb\" (UID: \"e8849902-5dbf-4045-97e8-94cd53c64a43\") " pod="ai-tenant-e2e-mt-7d5f4de7/e2e-lifecycle-model-7d5f4de7-kserve-799dd698b9-p8scb" Oct 31 00:45:11.045577 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:11.045559 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8849902-5dbf-4045-97e8-94cd53c64a43-tls-certs\") pod \"e2e-lifecycle-model-7d5f4de7-kserve-799dd698b9-p8scb\" (UID: \"e8849902-5dbf-4045-97e8-94cd53c64a43\") " pod="ai-tenant-e2e-mt-7d5f4de7/e2e-lifecycle-model-7d5f4de7-kserve-799dd698b9-p8scb" Oct 31 00:45:11.109238 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:11.109152 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lctf\" (UniqueName: \"kubernetes.io/projected/e8849902-5dbf-4045-97e8-94cd53c64a43-kube-api-access-5lctf\") pod \"e2e-lifecycle-model-7d5f4de7-kserve-799dd698b9-p8scb\" (UID: \"e8849902-5dbf-4045-97e8-94cd53c64a43\") " pod="ai-tenant-e2e-mt-7d5f4de7/e2e-lifecycle-model-7d5f4de7-kserve-799dd698b9-p8scb" Oct 31 00:45:11.145379 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:11.145341 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-7d5f4de7/e2e-lifecycle-model-7d5f4de7-kserve-799dd698b9-p8scb" Oct 31 00:45:11.536094 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:45:11.536059 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode8849902_5dbf_4045_97e8_94cd53c64a43.slice/crio-5ed1a83a9c4f6e53338c56b6c60108e2cf8e9c408ebc096c426a2e35a02753b1 WatchSource:0}: Error finding container 5ed1a83a9c4f6e53338c56b6c60108e2cf8e9c408ebc096c426a2e35a02753b1: Status 404 returned error can't find the container with id 5ed1a83a9c4f6e53338c56b6c60108e2cf8e9c408ebc096c426a2e35a02753b1 Oct 31 00:45:11.540925 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:11.540891 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-7d5f4de7/e2e-lifecycle-model-7d5f4de7-kserve-799dd698b9-p8scb"] Oct 31 00:45:11.797083 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:11.796985 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-f891e6-5b8bd8d58f-wpn4q" Oct 31 00:45:12.043327 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:12.043290 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-7d5f4de7/e2e-lifecycle-model-7d5f4de7-kserve-799dd698b9-p8scb" event={"ID":"e8849902-5dbf-4045-97e8-94cd53c64a43","Type":"ContainerStarted","Data":"7d2209ee2f6f5208d4e3f19d427e32681a81a4719441f8f33705cff08dd5656f"} Oct 31 00:45:12.043815 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:12.043339 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-7d5f4de7/e2e-lifecycle-model-7d5f4de7-kserve-799dd698b9-p8scb" event={"ID":"e8849902-5dbf-4045-97e8-94cd53c64a43","Type":"ContainerStarted","Data":"5ed1a83a9c4f6e53338c56b6c60108e2cf8e9c408ebc096c426a2e35a02753b1"} Oct 31 00:45:12.146470 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:12.146381 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-7d5f4de7/e2e-lifecycle-model-7d5f4de7-kserve-799dd698b9-p8scb" Oct 31 00:45:12.170096 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:12.170030 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-7d5f4de7/e2e-lifecycle-model-7d5f4de7-kserve-799dd698b9-p8scb" podStartSLOduration=2.170016337 podStartE2EDuration="2.170016337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:45:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:45:12.164275359 +0000 UTC m=+1804.635445108" watchObservedRunningTime="2026-10-31 00:45:12.170016337 +0000 UTC m=+1804.641186086" Oct 31 00:45:15.648901 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:15.648824 5222 scope.go:122] "RemoveContainer" containerID="2e680bf9b6cd854bcd7dc0dbf26122758ed5b83b3b2cca33e916929a8a20bc8a" Oct 31 00:45:16.081592 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:16.081555 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890125-4cnvj" event={"ID":"e64541b5-6dc8-4677-aaf3-95422b177fce","Type":"ContainerStarted","Data":"cfde1bf6fadea1efca0cac6b7b724f4878c2d654531972b7b9e526edee9e9cfe"} Oct 31 00:45:16.648746 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:16.648701 5222 scope.go:122] "RemoveContainer" containerID="552fc656ebb8c96e664c216d41b56ffe5789491597f632fbcc99468cfb02fa42" Oct 31 00:45:16.648924 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:16.648890 5222 scope.go:122] "RemoveContainer" containerID="4f8fae5465be21ca0061cd69e92270a66cb995ba574ff589adf342ba12dd5b0a" Oct 31 00:45:16.649328 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:16.649044 5222 scope.go:122] "RemoveContainer" containerID="bf42f9442b067c67e6dbd22224f37838f08787a2a2561e4795ea7fcd2eec49cc" Oct 31 00:45:17.100176 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:17.100145 5222 generic.go:356] "Generic (PLEG): container finished" podID="e64541b5-6dc8-4677-aaf3-95422b177fce" containerID="cfde1bf6fadea1efca0cac6b7b724f4878c2d654531972b7b9e526edee9e9cfe" exitCode=60 Oct 31 00:45:17.100495 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:17.100223 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890125-4cnvj" event={"ID":"e64541b5-6dc8-4677-aaf3-95422b177fce","Type":"ContainerDied","Data":"cfde1bf6fadea1efca0cac6b7b724f4878c2d654531972b7b9e526edee9e9cfe"} Oct 31 00:45:17.100495 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:17.100325 5222 scope.go:122] "RemoveContainer" containerID="2e680bf9b6cd854bcd7dc0dbf26122758ed5b83b3b2cca33e916929a8a20bc8a" Oct 31 00:45:17.100799 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:17.100784 5222 scope.go:122] "RemoveContainer" containerID="cfde1bf6fadea1efca0cac6b7b724f4878c2d654531972b7b9e526edee9e9cfe" Oct 31 00:45:17.101154 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:45:17.101135 5222 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29890125-4cnvj_odh-ai-gateway-infra(e64541b5-6dc8-4677-aaf3-95422b177fce)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890125-4cnvj" podUID="e64541b5-6dc8-4677-aaf3-95422b177fce" Oct 31 00:45:17.171776 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:17.171746 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-79tk8"] Oct 31 00:45:17.213487 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:17.213449 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890125-4cnvj"] Oct 31 00:45:18.110452 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:18.110329 5222 generic.go:356] "Generic (PLEG): container finished" podID="0b644185-b04a-4508-ae1a-61bb035c796e" containerID="c0a40dfdfc61d9fe5f30c87eb76c588b058b45c64b08caa8853b1044e7fa8ecb" exitCode=60 Oct 31 00:45:18.110452 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:18.110429 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-0da668-29890125-jfbgd" event={"ID":"0b644185-b04a-4508-ae1a-61bb035c796e","Type":"ContainerDied","Data":"c0a40dfdfc61d9fe5f30c87eb76c588b058b45c64b08caa8853b1044e7fa8ecb"} Oct 31 00:45:18.110939 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:18.110502 5222 scope.go:122] "RemoveContainer" containerID="552fc656ebb8c96e664c216d41b56ffe5789491597f632fbcc99468cfb02fa42" Oct 31 00:45:18.113106 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:18.113076 5222 generic.go:356] "Generic (PLEG): container finished" podID="e0edd0ed-32e0-4ba2-8ae0-958c6c530645" containerID="ff35e5de5a5c4778c329c252b5b6ed46b12146f6beb50b9c1782006d4eaff698" exitCode=60 Oct 31 00:45:18.113221 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:18.113152 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-d58cd3-29890125-5z9v5" event={"ID":"e0edd0ed-32e0-4ba2-8ae0-958c6c530645","Type":"ContainerDied","Data":"ff35e5de5a5c4778c329c252b5b6ed46b12146f6beb50b9c1782006d4eaff698"} Oct 31 00:45:18.113803 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:18.113783 5222 scope.go:122] "RemoveContainer" containerID="c0a40dfdfc61d9fe5f30c87eb76c588b058b45c64b08caa8853b1044e7fa8ecb" Oct 31 00:45:18.114012 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:18.113994 5222 scope.go:122] "RemoveContainer" containerID="ff35e5de5a5c4778c329c252b5b6ed46b12146f6beb50b9c1782006d4eaff698" Oct 31 00:45:18.114239 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:45:18.114204 5222 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-0da668-29890125-jfbgd_odh-ai-gateway-infra(0b644185-b04a-4508-ae1a-61bb035c796e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-0da668-29890125-jfbgd" podUID="0b644185-b04a-4508-ae1a-61bb035c796e" Oct 31 00:45:18.114409 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:45:18.114333 5222 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-d58cd3-29890125-5z9v5_odh-ai-gateway-infra(e0edd0ed-32e0-4ba2-8ae0-958c6c530645)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-d58cd3-29890125-5z9v5" podUID="e0edd0ed-32e0-4ba2-8ae0-958c6c530645" Oct 31 00:45:18.129178 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:18.127211 5222 generic.go:356] "Generic (PLEG): container finished" podID="17065afe-2927-4aa9-9dce-9c6c55315d1e" containerID="ba3e17139ae06ba599b17d58ddebc8b78ef2a4bb10e44540abafb1f7c72a72ef" exitCode=60 Oct 31 00:45:18.129178 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:18.127349 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-101b9e-29890125-8899q" event={"ID":"17065afe-2927-4aa9-9dce-9c6c55315d1e","Type":"ContainerDied","Data":"ba3e17139ae06ba599b17d58ddebc8b78ef2a4bb10e44540abafb1f7c72a72ef"} Oct 31 00:45:18.129178 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:18.129044 5222 scope.go:122] "RemoveContainer" containerID="ba3e17139ae06ba599b17d58ddebc8b78ef2a4bb10e44540abafb1f7c72a72ef" Oct 31 00:45:18.134229 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:45:18.129791 5222 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-101b9e-29890125-8899q_odh-ai-gateway-infra(17065afe-2927-4aa9-9dce-9c6c55315d1e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-101b9e-29890125-8899q" podUID="17065afe-2927-4aa9-9dce-9c6c55315d1e" Oct 31 00:45:18.228688 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:18.225955 5222 scope.go:122] "RemoveContainer" containerID="4f8fae5465be21ca0061cd69e92270a66cb995ba574ff589adf342ba12dd5b0a" Oct 31 00:45:18.278916 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:18.278882 5222 scope.go:122] "RemoveContainer" containerID="bf42f9442b067c67e6dbd22224f37838f08787a2a2561e4795ea7fcd2eec49cc" Oct 31 00:45:18.472557 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:18.472532 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890125-4cnvj" Oct 31 00:45:18.525760 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:18.525725 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e64541b5-6dc8-4677-aaf3-95422b177fce-kube-api-access-bvr2v\" (UniqueName: \"kubernetes.io/projected/e64541b5-6dc8-4677-aaf3-95422b177fce-kube-api-access-bvr2v\") pod \"e64541b5-6dc8-4677-aaf3-95422b177fce\" (UID: \"e64541b5-6dc8-4677-aaf3-95422b177fce\") " Oct 31 00:45:18.525929 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:18.525766 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e64541b5-6dc8-4677-aaf3-95422b177fce-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e64541b5-6dc8-4677-aaf3-95422b177fce-maas-api-service-ca\") pod \"e64541b5-6dc8-4677-aaf3-95422b177fce\" (UID: \"e64541b5-6dc8-4677-aaf3-95422b177fce\") " Oct 31 00:45:18.526219 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:18.526189 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e64541b5-6dc8-4677-aaf3-95422b177fce-maas-api-service-ca" pod "e64541b5-6dc8-4677-aaf3-95422b177fce" (UID: "e64541b5-6dc8-4677-aaf3-95422b177fce"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:45:18.526451 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:18.526432 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e64541b5-6dc8-4677-aaf3-95422b177fce-maas-api-service-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:18.527988 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:18.527965 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e64541b5-6dc8-4677-aaf3-95422b177fce-kube-api-access-bvr2v" pod "e64541b5-6dc8-4677-aaf3-95422b177fce" (UID: "e64541b5-6dc8-4677-aaf3-95422b177fce"). InnerVolumeSpecName "kube-api-access-bvr2v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:45:18.628691 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:18.628657 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvr2v\" (UniqueName: \"kubernetes.io/projected/e64541b5-6dc8-4677-aaf3-95422b177fce-kube-api-access-bvr2v\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:18.663589 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:18.663491 5222 scope.go:122] "RemoveContainer" containerID="25918655d66ed10b57e03cc3d0368bdc11dc8cefcdd258f7ac78d239dd93b3f8" Oct 31 00:45:18.663772 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:18.663640 5222 scope.go:122] "RemoveContainer" containerID="dbf989036ad3c81f61f595ecbcfc4fcf12cadbed0fb473fe538854de0b5ed818" Oct 31 00:45:18.663912 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:18.663889 5222 scope.go:122] "RemoveContainer" containerID="5e94176ca674fc42958f359ee0fd80474c2e82ee64174de6373ad1c5a8bcab60" Oct 31 00:45:18.664583 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:18.664545 5222 scope.go:122] "RemoveContainer" containerID="d37f289a8b1d3f953537ba8865786123d83d0b93b962c85e6df8ad6af56dd2bc" Oct 31 00:45:18.981266 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:18.981189 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-dcdbd0-7cc8947c99-k4hnc" Oct 31 00:45:19.181437 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:19.180179 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890125-4cnvj" Oct 31 00:45:19.181437 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:19.180222 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890125-4cnvj" event={"ID":"e64541b5-6dc8-4677-aaf3-95422b177fce","Type":"ContainerDied","Data":"cc989a09e83b3b7e117d27d6b26886577ce95239b20b49c4482b0d0d67ce2f34"} Oct 31 00:45:19.181437 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:19.180279 5222 scope.go:122] "RemoveContainer" containerID="cfde1bf6fadea1efca0cac6b7b724f4878c2d654531972b7b9e526edee9e9cfe" Oct 31 00:45:19.273271 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:19.273214 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-0da668-29890125-jfbgd"] Oct 31 00:45:19.380217 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:19.380185 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890125-4cnvj"] Oct 31 00:45:19.392231 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:19.392184 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890125-4cnvj"] Oct 31 00:45:19.404275 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:19.404128 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-101b9e-29890125-8899q"] Oct 31 00:45:19.515551 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:19.515512 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-d58cd3-29890125-5z9v5"] Oct 31 00:45:19.579573 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:19.579039 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-0da668-29890125-jfbgd" Oct 31 00:45:19.681638 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:19.681607 5222 scope.go:122] "RemoveContainer" containerID="59855b1f4a9823241b6676e0d5643b0cdb142b69f4a3a8401af3729844307976" Oct 31 00:45:19.712448 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:19.700398 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b644185-b04a-4508-ae1a-61bb035c796e-kube-api-access-88t6f\" (UniqueName: \"kubernetes.io/projected/0b644185-b04a-4508-ae1a-61bb035c796e-kube-api-access-88t6f\") pod \"0b644185-b04a-4508-ae1a-61bb035c796e\" (UID: \"0b644185-b04a-4508-ae1a-61bb035c796e\") " Oct 31 00:45:19.712448 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:19.700582 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b644185-b04a-4508-ae1a-61bb035c796e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b644185-b04a-4508-ae1a-61bb035c796e-maas-api-service-ca\") pod \"0b644185-b04a-4508-ae1a-61bb035c796e\" (UID: \"0b644185-b04a-4508-ae1a-61bb035c796e\") " Oct 31 00:45:19.712448 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:19.702650 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b644185-b04a-4508-ae1a-61bb035c796e-maas-api-service-ca" pod "0b644185-b04a-4508-ae1a-61bb035c796e" (UID: "0b644185-b04a-4508-ae1a-61bb035c796e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:45:19.712448 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:19.710530 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b644185-b04a-4508-ae1a-61bb035c796e-kube-api-access-88t6f" pod "0b644185-b04a-4508-ae1a-61bb035c796e" (UID: "0b644185-b04a-4508-ae1a-61bb035c796e"). InnerVolumeSpecName "kube-api-access-88t6f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:45:19.732645 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:19.732613 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e64541b5-6dc8-4677-aaf3-95422b177fce" path="/var/lib/kubelet/pods/e64541b5-6dc8-4677-aaf3-95422b177fce/volumes" Oct 31 00:45:19.764590 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:19.764548 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-101b9e-29890125-8899q" Oct 31 00:45:19.804273 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:19.804180 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88t6f\" (UniqueName: \"kubernetes.io/projected/0b644185-b04a-4508-ae1a-61bb035c796e-kube-api-access-88t6f\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:19.804273 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:19.804212 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b644185-b04a-4508-ae1a-61bb035c796e-maas-api-service-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:19.905275 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:19.905241 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17065afe-2927-4aa9-9dce-9c6c55315d1e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17065afe-2927-4aa9-9dce-9c6c55315d1e-maas-api-service-ca\") pod \"17065afe-2927-4aa9-9dce-9c6c55315d1e\" (UID: \"17065afe-2927-4aa9-9dce-9c6c55315d1e\") " Oct 31 00:45:19.905484 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:19.905423 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17065afe-2927-4aa9-9dce-9c6c55315d1e-kube-api-access-r4sgr\" (UniqueName: \"kubernetes.io/projected/17065afe-2927-4aa9-9dce-9c6c55315d1e-kube-api-access-r4sgr\") pod \"17065afe-2927-4aa9-9dce-9c6c55315d1e\" (UID: \"17065afe-2927-4aa9-9dce-9c6c55315d1e\") " Oct 31 00:45:19.905719 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:19.905689 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17065afe-2927-4aa9-9dce-9c6c55315d1e-maas-api-service-ca" pod "17065afe-2927-4aa9-9dce-9c6c55315d1e" (UID: "17065afe-2927-4aa9-9dce-9c6c55315d1e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:45:19.905993 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:19.905975 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17065afe-2927-4aa9-9dce-9c6c55315d1e-maas-api-service-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:19.907693 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:19.907672 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17065afe-2927-4aa9-9dce-9c6c55315d1e-kube-api-access-r4sgr" pod "17065afe-2927-4aa9-9dce-9c6c55315d1e" (UID: "17065afe-2927-4aa9-9dce-9c6c55315d1e"). InnerVolumeSpecName "kube-api-access-r4sgr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:45:19.992209 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:19.992184 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-dcdbd0-bc7d86cfc-rs4qx" Oct 31 00:45:20.007525 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:20.007491 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4sgr\" (UniqueName: \"kubernetes.io/projected/17065afe-2927-4aa9-9dce-9c6c55315d1e-kube-api-access-r4sgr\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:20.164068 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:20.164037 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-d58cd3-29890125-5z9v5" Oct 31 00:45:20.196250 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:20.196221 5222 generic.go:356] "Generic (PLEG): container finished" podID="bf7409ba-8c48-494c-92ac-a1686ce34ad2" containerID="0fd1bb3d93d795087d7cef13cf82f78738b7c24ff2ee362683da382416445f79" exitCode=60 Oct 31 00:45:20.196776 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:20.196292 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da9d07-29890125-q7hqd" event={"ID":"bf7409ba-8c48-494c-92ac-a1686ce34ad2","Type":"ContainerDied","Data":"0fd1bb3d93d795087d7cef13cf82f78738b7c24ff2ee362683da382416445f79"} Oct 31 00:45:20.196776 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:20.196355 5222 scope.go:122] "RemoveContainer" containerID="d37f289a8b1d3f953537ba8865786123d83d0b93b962c85e6df8ad6af56dd2bc" Oct 31 00:45:20.196901 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:20.196875 5222 scope.go:122] "RemoveContainer" containerID="0fd1bb3d93d795087d7cef13cf82f78738b7c24ff2ee362683da382416445f79" Oct 31 00:45:20.197615 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:45:20.197292 5222 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-da9d07-29890125-q7hqd_odh-ai-gateway-infra(bf7409ba-8c48-494c-92ac-a1686ce34ad2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da9d07-29890125-q7hqd" podUID="bf7409ba-8c48-494c-92ac-a1686ce34ad2" Oct 31 00:45:20.198650 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:20.198629 5222 generic.go:356] "Generic (PLEG): container finished" podID="9753f84f-09ef-4897-b46b-aec7dd93ee82" containerID="81bc9165a96cc70111de79f3ebcaf9260c7611dcd0a65a85e02a09e6547b7617" exitCode=60 Oct 31 00:45:20.198745 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:20.198653 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-982ae3-29890125-mjwrq" event={"ID":"9753f84f-09ef-4897-b46b-aec7dd93ee82","Type":"ContainerDied","Data":"81bc9165a96cc70111de79f3ebcaf9260c7611dcd0a65a85e02a09e6547b7617"} Oct 31 00:45:20.199041 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:20.199024 5222 scope.go:122] "RemoveContainer" containerID="81bc9165a96cc70111de79f3ebcaf9260c7611dcd0a65a85e02a09e6547b7617" Oct 31 00:45:20.199484 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:45:20.199461 5222 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-982ae3-29890125-mjwrq_odh-ai-gateway-infra(9753f84f-09ef-4897-b46b-aec7dd93ee82)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-982ae3-29890125-mjwrq" podUID="9753f84f-09ef-4897-b46b-aec7dd93ee82" Oct 31 00:45:20.200152 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:20.200133 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-0da668-29890125-jfbgd" event={"ID":"0b644185-b04a-4508-ae1a-61bb035c796e","Type":"ContainerDied","Data":"d49c2d902aab9240d922858bca071f974ea22638cba9c91f293874d77dc7d1aa"} Oct 31 00:45:20.200255 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:20.200141 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-0da668-29890125-jfbgd" Oct 31 00:45:20.201807 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:20.201776 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-d58cd3-29890125-5z9v5" Oct 31 00:45:20.201915 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:20.201819 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-d58cd3-29890125-5z9v5" event={"ID":"e0edd0ed-32e0-4ba2-8ae0-958c6c530645","Type":"ContainerDied","Data":"09c968c71ea5708a26c819435654aae9079557259402020f17d7a6725af38446"} Oct 31 00:45:20.204541 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:20.204520 5222 generic.go:356] "Generic (PLEG): container finished" podID="9e8880e8-58ba-4e14-b50b-1d66c4765ebd" containerID="e40268c6c61a996b421978e4d722a27dcc0440ec99893c1af540f1a5e098a4cc" exitCode=60 Oct 31 00:45:20.204641 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:20.204588 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-7d5f4de7-29890125-wn9gz" event={"ID":"9e8880e8-58ba-4e14-b50b-1d66c4765ebd","Type":"ContainerDied","Data":"e40268c6c61a996b421978e4d722a27dcc0440ec99893c1af540f1a5e098a4cc"} Oct 31 00:45:20.205051 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:20.205016 5222 scope.go:122] "RemoveContainer" containerID="e40268c6c61a996b421978e4d722a27dcc0440ec99893c1af540f1a5e098a4cc" Oct 31 00:45:20.205396 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:45:20.205347 5222 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-7d5f4de7-29890125-wn9gz_odh-ai-gateway-infra(9e8880e8-58ba-4e14-b50b-1d66c4765ebd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-7d5f4de7-29890125-wn9gz" podUID="9e8880e8-58ba-4e14-b50b-1d66c4765ebd" Oct 31 00:45:20.206652 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:20.206634 5222 generic.go:356] "Generic (PLEG): container finished" podID="ee2582c4-1fff-43ff-b101-98a384127187" containerID="76f67f62aa10db9f348e135f037dde5769b2608da3b3735fd6d680c06c3dd18a" exitCode=60 Oct 31 00:45:20.206735 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:20.206709 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-e030e0-29890125-88clm" event={"ID":"ee2582c4-1fff-43ff-b101-98a384127187","Type":"ContainerDied","Data":"76f67f62aa10db9f348e135f037dde5769b2608da3b3735fd6d680c06c3dd18a"} Oct 31 00:45:20.207114 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:20.207096 5222 scope.go:122] "RemoveContainer" containerID="76f67f62aa10db9f348e135f037dde5769b2608da3b3735fd6d680c06c3dd18a" Oct 31 00:45:20.207422 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:45:20.207400 5222 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-e030e0-29890125-88clm_odh-ai-gateway-infra(ee2582c4-1fff-43ff-b101-98a384127187)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-e030e0-29890125-88clm" podUID="ee2582c4-1fff-43ff-b101-98a384127187" Oct 31 00:45:20.209026 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:20.209007 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-101b9e-29890125-8899q" event={"ID":"17065afe-2927-4aa9-9dce-9c6c55315d1e","Type":"ContainerDied","Data":"291f378118834f972cba06a1f7048f5a01c88c71668a06e49a917b4d32f2dd15"} Oct 31 00:45:20.209109 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:20.209041 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-101b9e-29890125-8899q" Oct 31 00:45:20.212615 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:20.212593 5222 scope.go:122] "RemoveContainer" containerID="dbf989036ad3c81f61f595ecbcfc4fcf12cadbed0fb473fe538854de0b5ed818" Oct 31 00:45:20.227607 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:20.227550 5222 scope.go:122] "RemoveContainer" containerID="c0a40dfdfc61d9fe5f30c87eb76c588b058b45c64b08caa8853b1044e7fa8ecb" Oct 31 00:45:20.264920 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:20.264883 5222 scope.go:122] "RemoveContainer" containerID="ff35e5de5a5c4778c329c252b5b6ed46b12146f6beb50b9c1782006d4eaff698" Oct 31 00:45:20.282093 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:20.282064 5222 scope.go:122] "RemoveContainer" containerID="25918655d66ed10b57e03cc3d0368bdc11dc8cefcdd258f7ac78d239dd93b3f8" Oct 31 00:45:20.298532 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:20.298504 5222 scope.go:122] "RemoveContainer" containerID="5e94176ca674fc42958f359ee0fd80474c2e82ee64174de6373ad1c5a8bcab60" Oct 31 00:45:20.311805 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:20.311773 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0edd0ed-32e0-4ba2-8ae0-958c6c530645-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0edd0ed-32e0-4ba2-8ae0-958c6c530645-maas-api-service-ca\") pod \"e0edd0ed-32e0-4ba2-8ae0-958c6c530645\" (UID: \"e0edd0ed-32e0-4ba2-8ae0-958c6c530645\") " Oct 31 00:45:20.311984 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:20.311963 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0edd0ed-32e0-4ba2-8ae0-958c6c530645-kube-api-access-8ns9t\" (UniqueName: \"kubernetes.io/projected/e0edd0ed-32e0-4ba2-8ae0-958c6c530645-kube-api-access-8ns9t\") pod \"e0edd0ed-32e0-4ba2-8ae0-958c6c530645\" (UID: \"e0edd0ed-32e0-4ba2-8ae0-958c6c530645\") " Oct 31 00:45:20.312425 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:20.312393 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0edd0ed-32e0-4ba2-8ae0-958c6c530645-maas-api-service-ca" pod "e0edd0ed-32e0-4ba2-8ae0-958c6c530645" (UID: "e0edd0ed-32e0-4ba2-8ae0-958c6c530645"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:45:20.313100 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:20.313077 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0edd0ed-32e0-4ba2-8ae0-958c6c530645-maas-api-service-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:20.314923 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:20.314911 5222 scope.go:122] "RemoveContainer" containerID="ba3e17139ae06ba599b17d58ddebc8b78ef2a4bb10e44540abafb1f7c72a72ef" Oct 31 00:45:20.319823 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:20.319755 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0edd0ed-32e0-4ba2-8ae0-958c6c530645-kube-api-access-8ns9t" pod "e0edd0ed-32e0-4ba2-8ae0-958c6c530645" (UID: "e0edd0ed-32e0-4ba2-8ae0-958c6c530645"). InnerVolumeSpecName "kube-api-access-8ns9t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:45:20.378951 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:20.378747 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-101b9e-29890125-8899q"] Oct 31 00:45:20.382696 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:20.382665 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-101b9e-29890125-8899q"] Oct 31 00:45:20.414571 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:20.414542 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ns9t\" (UniqueName: \"kubernetes.io/projected/e0edd0ed-32e0-4ba2-8ae0-958c6c530645-kube-api-access-8ns9t\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:20.999076 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:20.999044 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-0da668-29890125-jfbgd"] Oct 31 00:45:21.002732 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:21.002703 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-0da668-29890125-jfbgd"] Oct 31 00:45:21.150744 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:21.149762 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-7d5f4de7/e2e-lifecycle-model-7d5f4de7-kserve-799dd698b9-p8scb" Oct 31 00:45:21.179147 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:21.179121 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-7d5f4de7/e2e-lifecycle-model-7d5f4de7-kserve-799dd698b9-p8scb" Oct 31 00:45:21.222521 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:21.222490 5222 generic.go:356] "Generic (PLEG): container finished" podID="05aae0c1-03f7-4a08-978e-adcbbd0d1a60" containerID="24d26610892a5bcfb9d60e06de962b4b6e62ec28f5bf142040be1e8efeadea0a" exitCode=60 Oct 31 00:45:21.222521 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:21.222558 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1e5e75-29890125-5xl6t" event={"ID":"05aae0c1-03f7-4a08-978e-adcbbd0d1a60","Type":"ContainerDied","Data":"24d26610892a5bcfb9d60e06de962b4b6e62ec28f5bf142040be1e8efeadea0a"} Oct 31 00:45:21.223099 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:21.222615 5222 scope.go:122] "RemoveContainer" containerID="59855b1f4a9823241b6676e0d5643b0cdb142b69f4a3a8401af3729844307976" Oct 31 00:45:21.223099 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:21.223074 5222 scope.go:122] "RemoveContainer" containerID="24d26610892a5bcfb9d60e06de962b4b6e62ec28f5bf142040be1e8efeadea0a" Oct 31 00:45:21.223532 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:45:21.223492 5222 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-1e5e75-29890125-5xl6t_odh-ai-gateway-infra(05aae0c1-03f7-4a08-978e-adcbbd0d1a60)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1e5e75-29890125-5xl6t" podUID="05aae0c1-03f7-4a08-978e-adcbbd0d1a60" Oct 31 00:45:21.234062 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:21.234027 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-d58cd3-29890125-5z9v5"] Oct 31 00:45:21.266026 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:21.265991 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-d58cd3-29890125-5z9v5"] Oct 31 00:45:21.270262 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:21.270238 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-7d5f4de7/e2e-lifecycle-model-7d5f4de7-kserve-799dd698b9-p8scb" Oct 31 00:45:21.674867 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:21.674831 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b644185-b04a-4508-ae1a-61bb035c796e" path="/var/lib/kubelet/pods/0b644185-b04a-4508-ae1a-61bb035c796e/volumes" Oct 31 00:45:21.675321 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:21.675307 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17065afe-2927-4aa9-9dce-9c6c55315d1e" path="/var/lib/kubelet/pods/17065afe-2927-4aa9-9dce-9c6c55315d1e/volumes" Oct 31 00:45:21.675790 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:21.675776 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0edd0ed-32e0-4ba2-8ae0-958c6c530645" path="/var/lib/kubelet/pods/e0edd0ed-32e0-4ba2-8ae0-958c6c530645/volumes" Oct 31 00:45:21.676454 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:21.676436 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-e030e0-29890125-88clm"] Oct 31 00:45:21.772836 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:21.772803 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-7d5f4de7-29890125-wn9gz"] Oct 31 00:45:21.897529 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:21.897496 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-982ae3-29890125-mjwrq"] Oct 31 00:45:21.926252 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:21.926147 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-e030e0-29890125-88clm" Oct 31 00:45:22.029839 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:22.029795 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-7d5f4de7-29890125-wn9gz" Oct 31 00:45:22.071259 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:22.071217 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee2582c4-1fff-43ff-b101-98a384127187-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee2582c4-1fff-43ff-b101-98a384127187-maas-api-service-ca\") pod \"ee2582c4-1fff-43ff-b101-98a384127187\" (UID: \"ee2582c4-1fff-43ff-b101-98a384127187\") " Oct 31 00:45:22.071535 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:22.071509 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee2582c4-1fff-43ff-b101-98a384127187-kube-api-access-jh2dt\" (UniqueName: \"kubernetes.io/projected/ee2582c4-1fff-43ff-b101-98a384127187-kube-api-access-jh2dt\") pod \"ee2582c4-1fff-43ff-b101-98a384127187\" (UID: \"ee2582c4-1fff-43ff-b101-98a384127187\") " Oct 31 00:45:22.072288 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:22.072247 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee2582c4-1fff-43ff-b101-98a384127187-maas-api-service-ca" pod "ee2582c4-1fff-43ff-b101-98a384127187" (UID: "ee2582c4-1fff-43ff-b101-98a384127187"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:45:22.072480 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:22.072460 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee2582c4-1fff-43ff-b101-98a384127187-maas-api-service-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:22.075112 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:22.075088 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee2582c4-1fff-43ff-b101-98a384127187-kube-api-access-jh2dt" pod "ee2582c4-1fff-43ff-b101-98a384127187" (UID: "ee2582c4-1fff-43ff-b101-98a384127187"). InnerVolumeSpecName "kube-api-access-jh2dt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:45:22.119975 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:22.119946 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-982ae3-29890125-mjwrq" Oct 31 00:45:22.128304 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:22.128276 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da9d07-29890125-q7hqd"] Oct 31 00:45:22.173385 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:22.173340 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e8880e8-58ba-4e14-b50b-1d66c4765ebd-kube-api-access-psk6t\" (UniqueName: \"kubernetes.io/projected/9e8880e8-58ba-4e14-b50b-1d66c4765ebd-kube-api-access-psk6t\") pod \"9e8880e8-58ba-4e14-b50b-1d66c4765ebd\" (UID: \"9e8880e8-58ba-4e14-b50b-1d66c4765ebd\") " Oct 31 00:45:22.173493 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:22.173430 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9753f84f-09ef-4897-b46b-aec7dd93ee82-kube-api-access-tzmdc\" (UniqueName: \"kubernetes.io/projected/9753f84f-09ef-4897-b46b-aec7dd93ee82-kube-api-access-tzmdc\") pod \"9753f84f-09ef-4897-b46b-aec7dd93ee82\" (UID: \"9753f84f-09ef-4897-b46b-aec7dd93ee82\") " Oct 31 00:45:22.173493 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:22.173456 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e8880e8-58ba-4e14-b50b-1d66c4765ebd-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e8880e8-58ba-4e14-b50b-1d66c4765ebd-maas-api-service-ca\") pod \"9e8880e8-58ba-4e14-b50b-1d66c4765ebd\" (UID: \"9e8880e8-58ba-4e14-b50b-1d66c4765ebd\") " Oct 31 00:45:22.173576 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:22.173512 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9753f84f-09ef-4897-b46b-aec7dd93ee82-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9753f84f-09ef-4897-b46b-aec7dd93ee82-maas-api-service-ca\") pod \"9753f84f-09ef-4897-b46b-aec7dd93ee82\" (UID: \"9753f84f-09ef-4897-b46b-aec7dd93ee82\") " Oct 31 00:45:22.174008 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:22.173974 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jh2dt\" (UniqueName: \"kubernetes.io/projected/ee2582c4-1fff-43ff-b101-98a384127187-kube-api-access-jh2dt\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:22.174135 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:22.174041 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e8880e8-58ba-4e14-b50b-1d66c4765ebd-maas-api-service-ca" pod "9e8880e8-58ba-4e14-b50b-1d66c4765ebd" (UID: "9e8880e8-58ba-4e14-b50b-1d66c4765ebd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:45:22.174312 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:22.174288 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9753f84f-09ef-4897-b46b-aec7dd93ee82-maas-api-service-ca" pod "9753f84f-09ef-4897-b46b-aec7dd93ee82" (UID: "9753f84f-09ef-4897-b46b-aec7dd93ee82"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:45:22.175865 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:22.175843 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e8880e8-58ba-4e14-b50b-1d66c4765ebd-kube-api-access-psk6t" pod "9e8880e8-58ba-4e14-b50b-1d66c4765ebd" (UID: "9e8880e8-58ba-4e14-b50b-1d66c4765ebd"). InnerVolumeSpecName "kube-api-access-psk6t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:45:22.176784 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:22.176659 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9753f84f-09ef-4897-b46b-aec7dd93ee82-kube-api-access-tzmdc" pod "9753f84f-09ef-4897-b46b-aec7dd93ee82" (UID: "9753f84f-09ef-4897-b46b-aec7dd93ee82"). InnerVolumeSpecName "kube-api-access-tzmdc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:45:22.274460 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:22.274410 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-e030e0-29890125-88clm" event={"ID":"ee2582c4-1fff-43ff-b101-98a384127187","Type":"ContainerDied","Data":"8f7ac05afa20dfa9cd459c5d66acf32fcf3e16dcfb99c0290f2bd2edbc0e0232"} Oct 31 00:45:22.274937 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:22.274482 5222 scope.go:122] "RemoveContainer" containerID="76f67f62aa10db9f348e135f037dde5769b2608da3b3735fd6d680c06c3dd18a" Oct 31 00:45:22.274937 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:22.274623 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-e030e0-29890125-88clm" Oct 31 00:45:22.275951 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:22.275857 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psk6t\" (UniqueName: \"kubernetes.io/projected/9e8880e8-58ba-4e14-b50b-1d66c4765ebd-kube-api-access-psk6t\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:22.275951 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:22.275895 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzmdc\" (UniqueName: \"kubernetes.io/projected/9753f84f-09ef-4897-b46b-aec7dd93ee82-kube-api-access-tzmdc\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:22.275951 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:22.275913 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e8880e8-58ba-4e14-b50b-1d66c4765ebd-maas-api-service-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:22.275951 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:22.275929 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9753f84f-09ef-4897-b46b-aec7dd93ee82-maas-api-service-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:22.279329 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:22.279291 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-982ae3-29890125-mjwrq" event={"ID":"9753f84f-09ef-4897-b46b-aec7dd93ee82","Type":"ContainerDied","Data":"3326881aa00685634c94e8510eb614f65c7467c2601b5b764b989cda5ee95c27"} Oct 31 00:45:22.279469 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:22.279441 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-982ae3-29890125-mjwrq" Oct 31 00:45:22.294171 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:22.294147 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-7d5f4de7-29890125-wn9gz" Oct 31 00:45:22.295935 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:22.294610 5222 scope.go:122] "RemoveContainer" containerID="81bc9165a96cc70111de79f3ebcaf9260c7611dcd0a65a85e02a09e6547b7617" Oct 31 00:45:22.295935 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:22.294812 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-7d5f4de7-29890125-wn9gz" event={"ID":"9e8880e8-58ba-4e14-b50b-1d66c4765ebd","Type":"ContainerDied","Data":"5f0ac6b89ad7afee9505e129448678a820fb8c11f906e67043f33f93c79b6da9"} Oct 31 00:45:22.313022 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:22.313000 5222 scope.go:122] "RemoveContainer" containerID="e40268c6c61a996b421978e4d722a27dcc0440ec99893c1af540f1a5e098a4cc" Oct 31 00:45:22.397771 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:22.397738 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1e5e75-29890125-5xl6t"] Oct 31 00:45:22.478219 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:22.478191 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da9d07-29890125-q7hqd" Oct 31 00:45:22.533266 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:22.533233 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-7d5f4de7-29890125-wn9gz"] Oct 31 00:45:22.537176 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:22.537144 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-7d5f4de7-29890125-wn9gz"] Oct 31 00:45:22.596359 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:22.595295 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf7409ba-8c48-494c-92ac-a1686ce34ad2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf7409ba-8c48-494c-92ac-a1686ce34ad2-maas-api-service-ca\") pod \"bf7409ba-8c48-494c-92ac-a1686ce34ad2\" (UID: \"bf7409ba-8c48-494c-92ac-a1686ce34ad2\") " Oct 31 00:45:22.596359 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:22.595457 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf7409ba-8c48-494c-92ac-a1686ce34ad2-kube-api-access-4pc5w\" (UniqueName: \"kubernetes.io/projected/bf7409ba-8c48-494c-92ac-a1686ce34ad2-kube-api-access-4pc5w\") pod \"bf7409ba-8c48-494c-92ac-a1686ce34ad2\" (UID: \"bf7409ba-8c48-494c-92ac-a1686ce34ad2\") " Oct 31 00:45:22.596359 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:22.596230 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf7409ba-8c48-494c-92ac-a1686ce34ad2-maas-api-service-ca" pod "bf7409ba-8c48-494c-92ac-a1686ce34ad2" (UID: "bf7409ba-8c48-494c-92ac-a1686ce34ad2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:45:22.598505 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:22.598474 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf7409ba-8c48-494c-92ac-a1686ce34ad2-kube-api-access-4pc5w" pod "bf7409ba-8c48-494c-92ac-a1686ce34ad2" (UID: "bf7409ba-8c48-494c-92ac-a1686ce34ad2"). InnerVolumeSpecName "kube-api-access-4pc5w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:45:22.613992 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:22.613969 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1e5e75-29890125-5xl6t" Oct 31 00:45:22.696668 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:22.696568 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05aae0c1-03f7-4a08-978e-adcbbd0d1a60-kube-api-access-gsncp\" (UniqueName: \"kubernetes.io/projected/05aae0c1-03f7-4a08-978e-adcbbd0d1a60-kube-api-access-gsncp\") pod \"05aae0c1-03f7-4a08-978e-adcbbd0d1a60\" (UID: \"05aae0c1-03f7-4a08-978e-adcbbd0d1a60\") " Oct 31 00:45:22.696852 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:22.696822 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05aae0c1-03f7-4a08-978e-adcbbd0d1a60-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05aae0c1-03f7-4a08-978e-adcbbd0d1a60-maas-api-service-ca\") pod \"05aae0c1-03f7-4a08-978e-adcbbd0d1a60\" (UID: \"05aae0c1-03f7-4a08-978e-adcbbd0d1a60\") " Oct 31 00:45:22.697301 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:22.697269 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05aae0c1-03f7-4a08-978e-adcbbd0d1a60-maas-api-service-ca" pod "05aae0c1-03f7-4a08-978e-adcbbd0d1a60" (UID: "05aae0c1-03f7-4a08-978e-adcbbd0d1a60"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:45:22.697582 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:22.697564 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05aae0c1-03f7-4a08-978e-adcbbd0d1a60-maas-api-service-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:22.697659 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:22.697588 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf7409ba-8c48-494c-92ac-a1686ce34ad2-maas-api-service-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:22.697659 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:22.697603 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pc5w\" (UniqueName: \"kubernetes.io/projected/bf7409ba-8c48-494c-92ac-a1686ce34ad2-kube-api-access-4pc5w\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:22.699445 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:22.699421 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05aae0c1-03f7-4a08-978e-adcbbd0d1a60-kube-api-access-gsncp" pod "05aae0c1-03f7-4a08-978e-adcbbd0d1a60" (UID: "05aae0c1-03f7-4a08-978e-adcbbd0d1a60"). InnerVolumeSpecName "kube-api-access-gsncp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:45:22.724828 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:22.724795 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-982ae3-29890125-mjwrq"] Oct 31 00:45:22.728774 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:22.728749 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-982ae3-29890125-mjwrq"] Oct 31 00:45:22.799728 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:22.799505 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsncp\" (UniqueName: \"kubernetes.io/projected/05aae0c1-03f7-4a08-978e-adcbbd0d1a60-kube-api-access-gsncp\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:22.974727 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:22.974632 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-e030e0-29890125-88clm"] Oct 31 00:45:22.978310 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:22.978282 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-e030e0-29890125-88clm"] Oct 31 00:45:23.302140 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:23.302117 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da9d07-29890125-q7hqd" Oct 31 00:45:23.302585 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:23.302086 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da9d07-29890125-q7hqd" event={"ID":"bf7409ba-8c48-494c-92ac-a1686ce34ad2","Type":"ContainerDied","Data":"832b49bb758530d0f335271ef8be89c6588ebd2dcdc0347a59ef74882c4b1b9f"} Oct 31 00:45:23.302585 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:23.302527 5222 scope.go:122] "RemoveContainer" containerID="0fd1bb3d93d795087d7cef13cf82f78738b7c24ff2ee362683da382416445f79" Oct 31 00:45:23.306878 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:23.306842 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1e5e75-29890125-5xl6t" event={"ID":"05aae0c1-03f7-4a08-978e-adcbbd0d1a60","Type":"ContainerDied","Data":"9212566c28862df3e5845b8b6bd8d02baba4d1d52155382cd13f2061b03b8add"} Oct 31 00:45:23.307024 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:23.306851 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1e5e75-29890125-5xl6t" Oct 31 00:45:23.317348 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:23.317327 5222 scope.go:122] "RemoveContainer" containerID="24d26610892a5bcfb9d60e06de962b4b6e62ec28f5bf142040be1e8efeadea0a" Oct 31 00:45:23.504199 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:23.504151 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da9d07-29890125-q7hqd"] Oct 31 00:45:23.513417 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:23.511003 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da9d07-29890125-q7hqd"] Oct 31 00:45:23.607129 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:23.607036 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1e5e75-29890125-5xl6t"] Oct 31 00:45:23.612510 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:23.612484 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1e5e75-29890125-5xl6t"] Oct 31 00:45:23.661505 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:23.661472 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05aae0c1-03f7-4a08-978e-adcbbd0d1a60" path="/var/lib/kubelet/pods/05aae0c1-03f7-4a08-978e-adcbbd0d1a60/volumes" Oct 31 00:45:23.662238 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:23.662217 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9753f84f-09ef-4897-b46b-aec7dd93ee82" path="/var/lib/kubelet/pods/9753f84f-09ef-4897-b46b-aec7dd93ee82/volumes" Oct 31 00:45:23.662854 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:23.662835 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e8880e8-58ba-4e14-b50b-1d66c4765ebd" path="/var/lib/kubelet/pods/9e8880e8-58ba-4e14-b50b-1d66c4765ebd/volumes" Oct 31 00:45:23.663841 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:23.663823 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf7409ba-8c48-494c-92ac-a1686ce34ad2" path="/var/lib/kubelet/pods/bf7409ba-8c48-494c-92ac-a1686ce34ad2/volumes" Oct 31 00:45:23.664398 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:23.664357 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee2582c4-1fff-43ff-b101-98a384127187" path="/var/lib/kubelet/pods/ee2582c4-1fff-43ff-b101-98a384127187/volumes" Oct 31 00:45:27.344284 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:27.344248 5222 generic.go:356] "Generic (PLEG): container finished" podID="fcada5bd-0764-4b0c-883a-d5d1867ca38e" containerID="aba32c5fa7816f3b70cd111f5250a6773ea14a12a382519cf4288f57f4ca8115" exitCode=0 Oct 31 00:45:27.344765 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:27.344313 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-e030e0-openshift-default-5c976d9f4f-lt8jf" event={"ID":"fcada5bd-0764-4b0c-883a-d5d1867ca38e","Type":"ContainerDied","Data":"aba32c5fa7816f3b70cd111f5250a6773ea14a12a382519cf4288f57f4ca8115"} Oct 31 00:45:27.344816 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:27.344775 5222 scope.go:122] "RemoveContainer" containerID="aba32c5fa7816f3b70cd111f5250a6773ea14a12a382519cf4288f57f4ca8115" Oct 31 00:45:27.346572 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:27.346544 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:45:27.346641 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:27.346618 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:45:27.346682 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:27.346653 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:45:28.373785 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:28.373748 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-e030e0-openshift-default-5c976d9f4f-lt8jf" event={"ID":"fcada5bd-0764-4b0c-883a-d5d1867ca38e","Type":"ContainerStarted","Data":"aafe3fa517c262a8e1d2a3e86409496f65430f74f47ff02f5a54ff397117fe20"} Oct 31 00:45:28.458899 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:28.456778 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-e030e0-openshift-default-5c976d9f4f-lt8jf" Oct 31 00:45:28.463395 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:28.461511 5222 patch_prober.go:29] interesting pod/e2e-shared-a-w6-e030e0-openshift-default-5c976d9f4f-lt8jf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body= Oct 31 00:45:28.463395 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:28.461589 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-e030e0-openshift-default-5c976d9f4f-lt8jf" podUID="fcada5bd-0764-4b0c-883a-d5d1867ca38e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" Oct 31 00:45:29.460396 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:29.457834 5222 patch_prober.go:29] interesting pod/e2e-shared-a-w6-e030e0-openshift-default-5c976d9f4f-lt8jf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body= Oct 31 00:45:29.460396 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:29.457903 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-e030e0-openshift-default-5c976d9f4f-lt8jf" podUID="fcada5bd-0764-4b0c-883a-d5d1867ca38e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" Oct 31 00:45:30.062620 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:30.062551 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-e030e0-openshift-default-5c976d9f4f-lt8jf" Oct 31 00:45:30.463412 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:30.462857 5222 patch_prober.go:29] interesting pod/e2e-shared-a-w6-e030e0-openshift-default-5c976d9f4f-lt8jf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body= Oct 31 00:45:30.463412 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:30.462924 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-e030e0-openshift-default-5c976d9f4f-lt8jf" podUID="fcada5bd-0764-4b0c-883a-d5d1867ca38e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" Oct 31 00:45:31.460788 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:31.460750 5222 patch_prober.go:29] interesting pod/e2e-shared-a-w6-e030e0-openshift-default-5c976d9f4f-lt8jf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body= Oct 31 00:45:31.461081 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:31.461051 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-e030e0-openshift-default-5c976d9f4f-lt8jf" podUID="fcada5bd-0764-4b0c-883a-d5d1867ca38e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" Oct 31 00:45:32.458051 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:32.458010 5222 patch_prober.go:29] interesting pod/e2e-shared-a-w6-e030e0-openshift-default-5c976d9f4f-lt8jf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body= Oct 31 00:45:32.458787 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:32.458731 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-e030e0-openshift-default-5c976d9f4f-lt8jf" podUID="fcada5bd-0764-4b0c-883a-d5d1867ca38e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" Oct 31 00:45:33.458397 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:33.457818 5222 patch_prober.go:29] interesting pod/e2e-shared-a-w6-e030e0-openshift-default-5c976d9f4f-lt8jf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body= Oct 31 00:45:33.458397 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:33.457887 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-e030e0-openshift-default-5c976d9f4f-lt8jf" podUID="fcada5bd-0764-4b0c-883a-d5d1867ca38e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" Oct 31 00:45:34.457896 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:34.457857 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-e030e0-openshift-default-5c976d9f4f-lt8jf" Oct 31 00:45:34.458964 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:34.458940 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-e030e0-openshift-default-5c976d9f4f-lt8jf" Oct 31 00:45:35.891804 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:45:35.891767 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 00:45:52.052105 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:52.052065 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b1262dc-znvmb"] Oct 31 00:45:52.098262 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:52.098229 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b1262dc-znvmb"] Oct 31 00:45:53.657444 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:45:53.657408 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6a3822f-439f-4f89-a225-49c93c09e17f" path="/var/lib/kubelet/pods/b6a3822f-439f-4f89-a225-49c93c09e17f/volumes" Oct 31 00:46:01.804179 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:01.804140 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-7lvcw"] Oct 31 00:46:01.804719 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:01.804345 5222 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-7lvcw" podUID="65f6cbd7-9324-4459-8820-86b6bae5f8a2" containerName="istio-proxy" containerID="cri-o://967352fb7fb171892c398ba632590e5522e80327263ecae9b0005f0e21178005" gracePeriod=30 Oct 31 00:46:01.972298 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:01.972256 5222 patch_prober.go:29] interesting pod/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-7lvcw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:46:01.972513 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:01.972335 5222 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-7lvcw" podUID="65f6cbd7-9324-4459-8820-86b6bae5f8a2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:46:04.990121 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:04.990083 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e9c3c1-57685bbfff-ddgqw"] Oct 31 00:46:04.990823 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:04.990806 5222 cpu_manager.go:414] "containerMap: removing container" podUID="ee2582c4-1fff-43ff-b101-98a384127187" containerName="cleanup" Oct 31 00:46:04.990881 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:04.990825 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee2582c4-1fff-43ff-b101-98a384127187" containerName="cleanup" Oct 31 00:46:04.990881 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:04.990839 5222 cpu_manager.go:414] "containerMap: removing container" podUID="e0edd0ed-32e0-4ba2-8ae0-958c6c530645" containerName="cleanup" Oct 31 00:46:04.990881 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:04.990844 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0edd0ed-32e0-4ba2-8ae0-958c6c530645" containerName="cleanup" Oct 31 00:46:04.990881 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:04.990852 5222 cpu_manager.go:414] "containerMap: removing container" podUID="9753f84f-09ef-4897-b46b-aec7dd93ee82" containerName="cleanup" Oct 31 00:46:04.990881 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:04.990856 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9753f84f-09ef-4897-b46b-aec7dd93ee82" containerName="cleanup" Oct 31 00:46:04.990881 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:04.990866 5222 cpu_manager.go:414] "containerMap: removing container" podUID="e64541b5-6dc8-4677-aaf3-95422b177fce" containerName="cleanup" Oct 31 00:46:04.990881 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:04.990871 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e64541b5-6dc8-4677-aaf3-95422b177fce" containerName="cleanup" Oct 31 00:46:04.990881 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:04.990878 5222 cpu_manager.go:414] "containerMap: removing container" podUID="0b644185-b04a-4508-ae1a-61bb035c796e" containerName="cleanup" Oct 31 00:46:04.990881 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:04.990883 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b644185-b04a-4508-ae1a-61bb035c796e" containerName="cleanup" Oct 31 00:46:04.991151 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:04.990889 5222 cpu_manager.go:414] "containerMap: removing container" podUID="9e8880e8-58ba-4e14-b50b-1d66c4765ebd" containerName="cleanup" Oct 31 00:46:04.991151 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:04.990893 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e8880e8-58ba-4e14-b50b-1d66c4765ebd" containerName="cleanup" Oct 31 00:46:04.991151 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:04.990902 5222 cpu_manager.go:414] "containerMap: removing container" podUID="9e8880e8-58ba-4e14-b50b-1d66c4765ebd" containerName="cleanup" Oct 31 00:46:04.991151 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:04.990906 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e8880e8-58ba-4e14-b50b-1d66c4765ebd" containerName="cleanup" Oct 31 00:46:04.991151 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:04.990913 5222 cpu_manager.go:414] "containerMap: removing container" podUID="05aae0c1-03f7-4a08-978e-adcbbd0d1a60" containerName="cleanup" Oct 31 00:46:04.991151 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:04.990918 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05aae0c1-03f7-4a08-978e-adcbbd0d1a60" containerName="cleanup" Oct 31 00:46:04.991151 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:04.990928 5222 cpu_manager.go:414] "containerMap: removing container" podUID="05aae0c1-03f7-4a08-978e-adcbbd0d1a60" containerName="cleanup" Oct 31 00:46:04.991151 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:04.990933 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05aae0c1-03f7-4a08-978e-adcbbd0d1a60" containerName="cleanup" Oct 31 00:46:04.991151 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:04.990948 5222 cpu_manager.go:414] "containerMap: removing container" podUID="bf7409ba-8c48-494c-92ac-a1686ce34ad2" containerName="cleanup" Oct 31 00:46:04.991151 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:04.990953 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf7409ba-8c48-494c-92ac-a1686ce34ad2" containerName="cleanup" Oct 31 00:46:04.991151 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:04.990963 5222 cpu_manager.go:414] "containerMap: removing container" podUID="0b644185-b04a-4508-ae1a-61bb035c796e" containerName="cleanup" Oct 31 00:46:04.991151 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:04.990967 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b644185-b04a-4508-ae1a-61bb035c796e" containerName="cleanup" Oct 31 00:46:04.991151 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:04.990974 5222 cpu_manager.go:414] "containerMap: removing container" podUID="e64541b5-6dc8-4677-aaf3-95422b177fce" containerName="cleanup" Oct 31 00:46:04.991151 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:04.990978 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e64541b5-6dc8-4677-aaf3-95422b177fce" containerName="cleanup" Oct 31 00:46:04.991151 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:04.990986 5222 cpu_manager.go:414] "containerMap: removing container" podUID="0b644185-b04a-4508-ae1a-61bb035c796e" containerName="cleanup" Oct 31 00:46:04.991151 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:04.990990 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b644185-b04a-4508-ae1a-61bb035c796e" containerName="cleanup" Oct 31 00:46:04.991151 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:04.990997 5222 cpu_manager.go:414] "containerMap: removing container" podUID="9753f84f-09ef-4897-b46b-aec7dd93ee82" containerName="cleanup" Oct 31 00:46:04.991151 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:04.991001 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9753f84f-09ef-4897-b46b-aec7dd93ee82" containerName="cleanup" Oct 31 00:46:04.991151 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:04.991006 5222 cpu_manager.go:414] "containerMap: removing container" podUID="05aae0c1-03f7-4a08-978e-adcbbd0d1a60" containerName="cleanup" Oct 31 00:46:04.991151 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:04.991010 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05aae0c1-03f7-4a08-978e-adcbbd0d1a60" containerName="cleanup" Oct 31 00:46:04.991151 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:04.991018 5222 cpu_manager.go:414] "containerMap: removing container" podUID="ee2582c4-1fff-43ff-b101-98a384127187" containerName="cleanup" Oct 31 00:46:04.991151 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:04.991022 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee2582c4-1fff-43ff-b101-98a384127187" containerName="cleanup" Oct 31 00:46:04.991151 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:04.991028 5222 cpu_manager.go:414] "containerMap: removing container" podUID="bf7409ba-8c48-494c-92ac-a1686ce34ad2" containerName="cleanup" Oct 31 00:46:04.991151 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:04.991032 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf7409ba-8c48-494c-92ac-a1686ce34ad2" containerName="cleanup" Oct 31 00:46:04.991151 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:04.991045 5222 cpu_manager.go:414] "containerMap: removing container" podUID="17065afe-2927-4aa9-9dce-9c6c55315d1e" containerName="cleanup" Oct 31 00:46:04.991151 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:04.991049 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17065afe-2927-4aa9-9dce-9c6c55315d1e" containerName="cleanup" Oct 31 00:46:04.991151 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:04.991059 5222 cpu_manager.go:414] "containerMap: removing container" podUID="17065afe-2927-4aa9-9dce-9c6c55315d1e" containerName="cleanup" Oct 31 00:46:04.991151 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:04.991062 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17065afe-2927-4aa9-9dce-9c6c55315d1e" containerName="cleanup" Oct 31 00:46:04.991151 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:04.991069 5222 cpu_manager.go:414] "containerMap: removing container" podUID="e0edd0ed-32e0-4ba2-8ae0-958c6c530645" containerName="cleanup" Oct 31 00:46:04.991151 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:04.991073 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0edd0ed-32e0-4ba2-8ae0-958c6c530645" containerName="cleanup" Oct 31 00:46:04.991151 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:04.991078 5222 cpu_manager.go:414] "containerMap: removing container" podUID="17065afe-2927-4aa9-9dce-9c6c55315d1e" containerName="cleanup" Oct 31 00:46:04.991151 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:04.991082 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17065afe-2927-4aa9-9dce-9c6c55315d1e" containerName="cleanup" Oct 31 00:46:04.991151 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:04.991090 5222 cpu_manager.go:414] "containerMap: removing container" podUID="bf7409ba-8c48-494c-92ac-a1686ce34ad2" containerName="cleanup" Oct 31 00:46:04.991151 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:04.991094 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf7409ba-8c48-494c-92ac-a1686ce34ad2" containerName="cleanup" Oct 31 00:46:04.992268 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:04.991197 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf7409ba-8c48-494c-92ac-a1686ce34ad2" containerName="cleanup" Oct 31 00:46:04.992268 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:04.991206 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b644185-b04a-4508-ae1a-61bb035c796e" containerName="cleanup" Oct 31 00:46:04.992268 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:04.991212 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="e64541b5-6dc8-4677-aaf3-95422b177fce" containerName="cleanup" Oct 31 00:46:04.992268 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:04.991216 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="e64541b5-6dc8-4677-aaf3-95422b177fce" containerName="cleanup" Oct 31 00:46:04.992268 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:04.991222 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0edd0ed-32e0-4ba2-8ae0-958c6c530645" containerName="cleanup" Oct 31 00:46:04.992268 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:04.991227 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="9753f84f-09ef-4897-b46b-aec7dd93ee82" containerName="cleanup" Oct 31 00:46:04.992268 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:04.991232 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee2582c4-1fff-43ff-b101-98a384127187" containerName="cleanup" Oct 31 00:46:04.992268 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:04.991237 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b644185-b04a-4508-ae1a-61bb035c796e" containerName="cleanup" Oct 31 00:46:04.992268 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:04.991241 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="9753f84f-09ef-4897-b46b-aec7dd93ee82" containerName="cleanup" Oct 31 00:46:04.992268 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:04.991246 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="05aae0c1-03f7-4a08-978e-adcbbd0d1a60" containerName="cleanup" Oct 31 00:46:04.992268 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:04.991251 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee2582c4-1fff-43ff-b101-98a384127187" containerName="cleanup" Oct 31 00:46:04.992268 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:04.991257 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e8880e8-58ba-4e14-b50b-1d66c4765ebd" containerName="cleanup" Oct 31 00:46:04.992268 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:04.991263 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="17065afe-2927-4aa9-9dce-9c6c55315d1e" containerName="cleanup" Oct 31 00:46:04.992268 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:04.991270 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0edd0ed-32e0-4ba2-8ae0-958c6c530645" containerName="cleanup" Oct 31 00:46:04.992268 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:04.991302 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="17065afe-2927-4aa9-9dce-9c6c55315d1e" containerName="cleanup" Oct 31 00:46:04.992268 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:04.991308 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf7409ba-8c48-494c-92ac-a1686ce34ad2" containerName="cleanup" Oct 31 00:46:04.992268 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:04.991316 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0edd0ed-32e0-4ba2-8ae0-958c6c530645" containerName="cleanup" Oct 31 00:46:04.992268 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:04.991323 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e8880e8-58ba-4e14-b50b-1d66c4765ebd" containerName="cleanup" Oct 31 00:46:04.992268 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:04.991329 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="05aae0c1-03f7-4a08-978e-adcbbd0d1a60" containerName="cleanup" Oct 31 00:46:04.995275 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:04.995253 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e9c3c1-57685bbfff-ddgqw" Oct 31 00:46:05.049660 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:05.049624 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e9c3c1-57685bbfff-ddgqw"] Oct 31 00:46:05.097688 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:05.097654 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-e9c3c1\"" Oct 31 00:46:05.098206 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:05.098180 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-e9c3c1-dockercfg-nlst6\"" Oct 31 00:46:05.202459 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:05.192945 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fadfc07d-4c35-4936-a0ae-aded9677b62b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-e9c3c1-57685bbfff-ddgqw\" (UID: \"fadfc07d-4c35-4936-a0ae-aded9677b62b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e9c3c1-57685bbfff-ddgqw" Oct 31 00:46:05.202954 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:05.202919 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ltvw\" (UniqueName: \"kubernetes.io/projected/fadfc07d-4c35-4936-a0ae-aded9677b62b-kube-api-access-5ltvw\") pod \"payload-pre-processing-e2e-shared-b-w0-e9c3c1-57685bbfff-ddgqw\" (UID: \"fadfc07d-4c35-4936-a0ae-aded9677b62b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e9c3c1-57685bbfff-ddgqw" Oct 31 00:46:05.316023 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:05.315880 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ltvw\" (UniqueName: \"kubernetes.io/projected/fadfc07d-4c35-4936-a0ae-aded9677b62b-kube-api-access-5ltvw\") pod \"payload-pre-processing-e2e-shared-b-w0-e9c3c1-57685bbfff-ddgqw\" (UID: \"fadfc07d-4c35-4936-a0ae-aded9677b62b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e9c3c1-57685bbfff-ddgqw" Oct 31 00:46:05.317542 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:05.317492 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fadfc07d-4c35-4936-a0ae-aded9677b62b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-e9c3c1-57685bbfff-ddgqw\" (UID: \"fadfc07d-4c35-4936-a0ae-aded9677b62b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e9c3c1-57685bbfff-ddgqw" Oct 31 00:46:05.318727 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:05.318687 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fadfc07d-4c35-4936-a0ae-aded9677b62b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-e9c3c1-57685bbfff-ddgqw\" (UID: \"fadfc07d-4c35-4936-a0ae-aded9677b62b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e9c3c1-57685bbfff-ddgqw" Oct 31 00:46:05.379284 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:05.379237 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ltvw\" (UniqueName: \"kubernetes.io/projected/fadfc07d-4c35-4936-a0ae-aded9677b62b-kube-api-access-5ltvw\") pod \"payload-pre-processing-e2e-shared-b-w0-e9c3c1-57685bbfff-ddgqw\" (UID: \"fadfc07d-4c35-4936-a0ae-aded9677b62b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e9c3c1-57685bbfff-ddgqw" Oct 31 00:46:05.611204 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:05.611109 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e9c3c1-57685bbfff-ddgqw" Oct 31 00:46:05.898064 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:46:05.897957 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 00:46:06.073394 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:06.044426 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e9c3c1-57685bbfff-ddgqw"] Oct 31 00:46:06.883455 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:06.881791 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e9c3c1-57685bbfff-ddgqw" event={"ID":"fadfc07d-4c35-4936-a0ae-aded9677b62b","Type":"ContainerStarted","Data":"e30312bfbd59764574bf4f8c5c225bc97370fbc41af9f6e73209c8cb8919dd40"} Oct 31 00:46:06.883455 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:06.881908 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e9c3c1-57685bbfff-ddgqw" event={"ID":"fadfc07d-4c35-4936-a0ae-aded9677b62b","Type":"ContainerStarted","Data":"8fe13fc858628313fd91e823b19cbf0226334ff8be69524aaf74b648446d396d"} Oct 31 00:46:06.883455 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:06.881937 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e9c3c1-57685bbfff-ddgqw" Oct 31 00:46:07.011440 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:07.011358 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e9c3c1-57685bbfff-ddgqw" podStartSLOduration=3.011340214 podStartE2EDuration="3.011340214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:46:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:46:07.009886572 +0000 UTC m=+1859.481056322" watchObservedRunningTime="2026-10-31 00:46:07.011340214 +0000 UTC m=+1859.482509964" Oct 31 00:46:07.187890 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:07.164920 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-7lvcw" Oct 31 00:46:07.187890 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:07.176917 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65f6cbd7-9324-4459-8820-86b6bae5f8a2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65f6cbd7-9324-4459-8820-86b6bae5f8a2-istio-data\") pod \"65f6cbd7-9324-4459-8820-86b6bae5f8a2\" (UID: \"65f6cbd7-9324-4459-8820-86b6bae5f8a2\") " Oct 31 00:46:07.187890 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:07.177038 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65f6cbd7-9324-4459-8820-86b6bae5f8a2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65f6cbd7-9324-4459-8820-86b6bae5f8a2-istio-envoy\") pod \"65f6cbd7-9324-4459-8820-86b6bae5f8a2\" (UID: \"65f6cbd7-9324-4459-8820-86b6bae5f8a2\") " Oct 31 00:46:07.187890 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:07.177135 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65f6cbd7-9324-4459-8820-86b6bae5f8a2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65f6cbd7-9324-4459-8820-86b6bae5f8a2-workload-socket\") pod \"65f6cbd7-9324-4459-8820-86b6bae5f8a2\" (UID: \"65f6cbd7-9324-4459-8820-86b6bae5f8a2\") " Oct 31 00:46:07.187890 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:07.177202 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65f6cbd7-9324-4459-8820-86b6bae5f8a2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65f6cbd7-9324-4459-8820-86b6bae5f8a2-credential-socket\") pod \"65f6cbd7-9324-4459-8820-86b6bae5f8a2\" (UID: \"65f6cbd7-9324-4459-8820-86b6bae5f8a2\") " Oct 31 00:46:07.187890 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:07.177233 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65f6cbd7-9324-4459-8820-86b6bae5f8a2-kube-api-access-ntx2k\" (UniqueName: \"kubernetes.io/projected/65f6cbd7-9324-4459-8820-86b6bae5f8a2-kube-api-access-ntx2k\") pod \"65f6cbd7-9324-4459-8820-86b6bae5f8a2\" (UID: \"65f6cbd7-9324-4459-8820-86b6bae5f8a2\") " Oct 31 00:46:07.187890 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:07.177318 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/65f6cbd7-9324-4459-8820-86b6bae5f8a2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65f6cbd7-9324-4459-8820-86b6bae5f8a2-istio-podinfo\") pod \"65f6cbd7-9324-4459-8820-86b6bae5f8a2\" (UID: \"65f6cbd7-9324-4459-8820-86b6bae5f8a2\") " Oct 31 00:46:07.187890 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:07.177360 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65f6cbd7-9324-4459-8820-86b6bae5f8a2-istio-token\" (UniqueName: \"kubernetes.io/projected/65f6cbd7-9324-4459-8820-86b6bae5f8a2-istio-token\") pod \"65f6cbd7-9324-4459-8820-86b6bae5f8a2\" (UID: \"65f6cbd7-9324-4459-8820-86b6bae5f8a2\") " Oct 31 00:46:07.187890 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:07.177471 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65f6cbd7-9324-4459-8820-86b6bae5f8a2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65f6cbd7-9324-4459-8820-86b6bae5f8a2-istiod-ca-cert\") pod \"65f6cbd7-9324-4459-8820-86b6bae5f8a2\" (UID: \"65f6cbd7-9324-4459-8820-86b6bae5f8a2\") " Oct 31 00:46:07.187890 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:07.177587 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65f6cbd7-9324-4459-8820-86b6bae5f8a2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65f6cbd7-9324-4459-8820-86b6bae5f8a2-workload-certs\") pod \"65f6cbd7-9324-4459-8820-86b6bae5f8a2\" (UID: \"65f6cbd7-9324-4459-8820-86b6bae5f8a2\") " Oct 31 00:46:07.187890 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:07.187545 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65f6cbd7-9324-4459-8820-86b6bae5f8a2-workload-certs" pod "65f6cbd7-9324-4459-8820-86b6bae5f8a2" (UID: "65f6cbd7-9324-4459-8820-86b6bae5f8a2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:46:07.187890 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:07.187780 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65f6cbd7-9324-4459-8820-86b6bae5f8a2-credential-socket" pod "65f6cbd7-9324-4459-8820-86b6bae5f8a2" (UID: "65f6cbd7-9324-4459-8820-86b6bae5f8a2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:46:07.189235 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:07.187990 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65f6cbd7-9324-4459-8820-86b6bae5f8a2-istio-data" pod "65f6cbd7-9324-4459-8820-86b6bae5f8a2" (UID: "65f6cbd7-9324-4459-8820-86b6bae5f8a2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:46:07.190670 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:07.190635 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65f6cbd7-9324-4459-8820-86b6bae5f8a2-workload-socket" pod "65f6cbd7-9324-4459-8820-86b6bae5f8a2" (UID: "65f6cbd7-9324-4459-8820-86b6bae5f8a2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:46:07.190828 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:07.190809 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65f6cbd7-9324-4459-8820-86b6bae5f8a2-istiod-ca-cert" pod "65f6cbd7-9324-4459-8820-86b6bae5f8a2" (UID: "65f6cbd7-9324-4459-8820-86b6bae5f8a2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:46:07.192803 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:07.192055 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65f6cbd7-9324-4459-8820-86b6bae5f8a2-istio-token" pod "65f6cbd7-9324-4459-8820-86b6bae5f8a2" (UID: "65f6cbd7-9324-4459-8820-86b6bae5f8a2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:46:07.195272 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:07.193561 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/65f6cbd7-9324-4459-8820-86b6bae5f8a2-istio-podinfo" pod "65f6cbd7-9324-4459-8820-86b6bae5f8a2" (UID: "65f6cbd7-9324-4459-8820-86b6bae5f8a2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:46:07.195272 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:07.193846 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65f6cbd7-9324-4459-8820-86b6bae5f8a2-kube-api-access-ntx2k" pod "65f6cbd7-9324-4459-8820-86b6bae5f8a2" (UID: "65f6cbd7-9324-4459-8820-86b6bae5f8a2"). InnerVolumeSpecName "kube-api-access-ntx2k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:46:07.198533 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:07.198504 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65f6cbd7-9324-4459-8820-86b6bae5f8a2-istio-envoy" pod "65f6cbd7-9324-4459-8820-86b6bae5f8a2" (UID: "65f6cbd7-9324-4459-8820-86b6bae5f8a2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:46:07.279402 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:07.279350 5222 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65f6cbd7-9324-4459-8820-86b6bae5f8a2-credential-socket\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:46:07.279402 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:07.279401 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntx2k\" (UniqueName: \"kubernetes.io/projected/65f6cbd7-9324-4459-8820-86b6bae5f8a2-kube-api-access-ntx2k\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:46:07.279402 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:07.279412 5222 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65f6cbd7-9324-4459-8820-86b6bae5f8a2-istio-podinfo\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:46:07.279402 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:07.279422 5222 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65f6cbd7-9324-4459-8820-86b6bae5f8a2-istio-token\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:46:07.279742 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:07.279431 5222 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65f6cbd7-9324-4459-8820-86b6bae5f8a2-istiod-ca-cert\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:46:07.279742 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:07.279440 5222 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65f6cbd7-9324-4459-8820-86b6bae5f8a2-workload-certs\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:46:07.279742 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:07.279450 5222 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65f6cbd7-9324-4459-8820-86b6bae5f8a2-istio-data\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:46:07.279742 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:07.279459 5222 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65f6cbd7-9324-4459-8820-86b6bae5f8a2-istio-envoy\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:46:07.279742 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:07.279467 5222 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65f6cbd7-9324-4459-8820-86b6bae5f8a2-workload-socket\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:46:07.892241 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:07.892206 5222 generic.go:356] "Generic (PLEG): container finished" podID="65f6cbd7-9324-4459-8820-86b6bae5f8a2" containerID="967352fb7fb171892c398ba632590e5522e80327263ecae9b0005f0e21178005" exitCode=0 Oct 31 00:46:07.892452 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:07.892276 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-7lvcw" Oct 31 00:46:07.892452 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:07.892299 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-7lvcw" event={"ID":"65f6cbd7-9324-4459-8820-86b6bae5f8a2","Type":"ContainerDied","Data":"967352fb7fb171892c398ba632590e5522e80327263ecae9b0005f0e21178005"} Oct 31 00:46:07.892452 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:07.892360 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-7lvcw" event={"ID":"65f6cbd7-9324-4459-8820-86b6bae5f8a2","Type":"ContainerDied","Data":"afb299c366973aeb9b962d77518413f7bc20c870ab932a24ec373581301cdcd0"} Oct 31 00:46:07.892452 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:07.892408 5222 scope.go:122] "RemoveContainer" containerID="967352fb7fb171892c398ba632590e5522e80327263ecae9b0005f0e21178005" Oct 31 00:46:07.909748 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:07.909713 5222 scope.go:122] "RemoveContainer" containerID="967352fb7fb171892c398ba632590e5522e80327263ecae9b0005f0e21178005" Oct 31 00:46:07.911288 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:46:07.911253 5222 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"967352fb7fb171892c398ba632590e5522e80327263ecae9b0005f0e21178005\": container with ID starting with 967352fb7fb171892c398ba632590e5522e80327263ecae9b0005f0e21178005 not found: ID does not exist" containerID="967352fb7fb171892c398ba632590e5522e80327263ecae9b0005f0e21178005" Oct 31 00:46:07.911483 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:07.911299 5222 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"967352fb7fb171892c398ba632590e5522e80327263ecae9b0005f0e21178005"} err="failed to get container status \"967352fb7fb171892c398ba632590e5522e80327263ecae9b0005f0e21178005\": rpc error: code = NotFound desc = could not find container \"967352fb7fb171892c398ba632590e5522e80327263ecae9b0005f0e21178005\": container with ID starting with 967352fb7fb171892c398ba632590e5522e80327263ecae9b0005f0e21178005 not found: ID does not exist" Oct 31 00:46:08.065830 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:08.065790 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-7lvcw"] Oct 31 00:46:08.117030 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:08.116994 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-7lvcw"] Oct 31 00:46:09.103759 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:09.103717 5222 scope.go:122] "RemoveContainer" containerID="15257effc61ebaf3d1fcabcb48bfa63dce43e6d96d2e9019579ade74e583b4e6" Oct 31 00:46:09.115821 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:09.115796 5222 scope.go:122] "RemoveContainer" containerID="aa89b850223d8f994b3df7b38393faf69ca84e7ddc442d3be0476310687963b4" Oct 31 00:46:09.126072 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:09.126049 5222 scope.go:122] "RemoveContainer" containerID="ae9d30ee54c56260c54d9c0645579e5892dfa5a01eb9562a9ce2ef4cd3a66422" Oct 31 00:46:09.135964 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:09.135941 5222 scope.go:122] "RemoveContainer" containerID="61a57c631923b74d92012939a8abafd73a0b78f18152017b0ecc53b5e4ea03b3" Oct 31 00:46:09.657813 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:09.657769 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65f6cbd7-9324-4459-8820-86b6bae5f8a2" path="/var/lib/kubelet/pods/65f6cbd7-9324-4459-8820-86b6bae5f8a2/volumes" Oct 31 00:46:09.923230 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:09.923149 5222 generic.go:356] "Generic (PLEG): container finished" podID="d5eb6fe4-1be0-468b-91e0-d398870307b6" containerID="7c7c18c30eb9362a22bd2e11df1913e7e2d5bacdb670e046c93ef47dc33eac3f" exitCode=0 Oct 31 00:46:09.923403 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:09.923223 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-0da668-openshift-default-5467f58b7-tzrb7" event={"ID":"d5eb6fe4-1be0-468b-91e0-d398870307b6","Type":"ContainerDied","Data":"7c7c18c30eb9362a22bd2e11df1913e7e2d5bacdb670e046c93ef47dc33eac3f"} Oct 31 00:46:09.923744 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:09.923731 5222 scope.go:122] "RemoveContainer" containerID="7c7c18c30eb9362a22bd2e11df1913e7e2d5bacdb670e046c93ef47dc33eac3f" Oct 31 00:46:09.925587 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:09.925553 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:46:09.925699 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:09.925629 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:46:09.925699 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:09.925660 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:46:10.936846 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:10.936813 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-0da668-openshift-default-5467f58b7-tzrb7" event={"ID":"d5eb6fe4-1be0-468b-91e0-d398870307b6","Type":"ContainerStarted","Data":"8e6e7087ecc3616c96560b1b7b372159fb4eb1f877a9ab9e45a03c54a7cb4b73"} Oct 31 00:46:11.260173 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:11.258783 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-0da668-openshift-default-5467f58b7-tzrb7" Oct 31 00:46:11.262572 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:11.262534 5222 patch_prober.go:29] interesting pod/e2e-shared-b-w4-0da668-openshift-default-5467f58b7-tzrb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Oct 31 00:46:11.262768 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:11.262594 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-0da668-openshift-default-5467f58b7-tzrb7" podUID="d5eb6fe4-1be0-468b-91e0-d398870307b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Oct 31 00:46:12.262912 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:12.262875 5222 patch_prober.go:29] interesting pod/e2e-shared-b-w4-0da668-openshift-default-5467f58b7-tzrb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Oct 31 00:46:12.263565 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:12.263526 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-0da668-openshift-default-5467f58b7-tzrb7" podUID="d5eb6fe4-1be0-468b-91e0-d398870307b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Oct 31 00:46:12.265665 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:12.265637 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-2594a9-openshift-default-8694ddf5f7-v4jsh"] Oct 31 00:46:12.266784 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:12.266762 5222 cpu_manager.go:414] "containerMap: removing container" podUID="65f6cbd7-9324-4459-8820-86b6bae5f8a2" containerName="istio-proxy" Oct 31 00:46:12.266830 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:12.266790 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65f6cbd7-9324-4459-8820-86b6bae5f8a2" containerName="istio-proxy" Oct 31 00:46:12.266830 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:12.266811 5222 cpu_manager.go:414] "containerMap: removing container" podUID="e0edd0ed-32e0-4ba2-8ae0-958c6c530645" containerName="cleanup" Oct 31 00:46:12.266830 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:12.266821 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0edd0ed-32e0-4ba2-8ae0-958c6c530645" containerName="cleanup" Oct 31 00:46:12.266931 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:12.266844 5222 cpu_manager.go:414] "containerMap: removing container" podUID="9e8880e8-58ba-4e14-b50b-1d66c4765ebd" containerName="cleanup" Oct 31 00:46:12.266931 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:12.266852 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e8880e8-58ba-4e14-b50b-1d66c4765ebd" containerName="cleanup" Oct 31 00:46:12.266931 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:12.266897 5222 cpu_manager.go:414] "containerMap: removing container" podUID="e64541b5-6dc8-4677-aaf3-95422b177fce" containerName="cleanup" Oct 31 00:46:12.266931 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:12.266904 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e64541b5-6dc8-4677-aaf3-95422b177fce" containerName="cleanup" Oct 31 00:46:12.266931 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:12.266920 5222 cpu_manager.go:414] "containerMap: removing container" podUID="9753f84f-09ef-4897-b46b-aec7dd93ee82" containerName="cleanup" Oct 31 00:46:12.266931 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:12.266927 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9753f84f-09ef-4897-b46b-aec7dd93ee82" containerName="cleanup" Oct 31 00:46:12.267103 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:12.266936 5222 cpu_manager.go:414] "containerMap: removing container" podUID="ee2582c4-1fff-43ff-b101-98a384127187" containerName="cleanup" Oct 31 00:46:12.267103 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:12.266944 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee2582c4-1fff-43ff-b101-98a384127187" containerName="cleanup" Oct 31 00:46:12.267162 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:12.267144 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="05aae0c1-03f7-4a08-978e-adcbbd0d1a60" containerName="cleanup" Oct 31 00:46:12.267517 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:12.267500 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="e64541b5-6dc8-4677-aaf3-95422b177fce" containerName="cleanup" Oct 31 00:46:12.267575 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:12.267531 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b644185-b04a-4508-ae1a-61bb035c796e" containerName="cleanup" Oct 31 00:46:12.267575 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:12.267544 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="9753f84f-09ef-4897-b46b-aec7dd93ee82" containerName="cleanup" Oct 31 00:46:12.267575 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:12.267555 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee2582c4-1fff-43ff-b101-98a384127187" containerName="cleanup" Oct 31 00:46:12.267677 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:12.267589 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="17065afe-2927-4aa9-9dce-9c6c55315d1e" containerName="cleanup" Oct 31 00:46:12.267677 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:12.267597 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf7409ba-8c48-494c-92ac-a1686ce34ad2" containerName="cleanup" Oct 31 00:46:12.267677 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:12.267608 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="65f6cbd7-9324-4459-8820-86b6bae5f8a2" containerName="istio-proxy" Oct 31 00:46:12.267677 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:12.267622 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e8880e8-58ba-4e14-b50b-1d66c4765ebd" containerName="cleanup" Oct 31 00:46:12.271829 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:12.271809 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-2594a9-openshift-default-8694ddf5f7-v4jsh" Oct 31 00:46:12.336976 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:12.336939 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w6-2594a9-openshift-default-dockercfg-4xq8w\"" Oct 31 00:46:12.377907 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:12.377878 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-2594a9-openshift-default-8694ddf5f7-v4jsh"] Oct 31 00:46:12.419944 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:12.412711 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c57fa36-80bb-403d-97a3-b18601c386aa-istiod-ca-cert\") pod \"e2e-shared-b-w6-2594a9-openshift-default-8694ddf5f7-v4jsh\" (UID: \"4c57fa36-80bb-403d-97a3-b18601c386aa\") " pod="openshift-ingress/e2e-shared-b-w6-2594a9-openshift-default-8694ddf5f7-v4jsh" Oct 31 00:46:12.419944 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:12.412780 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87hgh\" (UniqueName: \"kubernetes.io/projected/4c57fa36-80bb-403d-97a3-b18601c386aa-kube-api-access-87hgh\") pod \"e2e-shared-b-w6-2594a9-openshift-default-8694ddf5f7-v4jsh\" (UID: \"4c57fa36-80bb-403d-97a3-b18601c386aa\") " pod="openshift-ingress/e2e-shared-b-w6-2594a9-openshift-default-8694ddf5f7-v4jsh" Oct 31 00:46:12.419944 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:12.412849 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c57fa36-80bb-403d-97a3-b18601c386aa-credential-socket\") pod \"e2e-shared-b-w6-2594a9-openshift-default-8694ddf5f7-v4jsh\" (UID: \"4c57fa36-80bb-403d-97a3-b18601c386aa\") " pod="openshift-ingress/e2e-shared-b-w6-2594a9-openshift-default-8694ddf5f7-v4jsh" Oct 31 00:46:12.419944 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:12.412886 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c57fa36-80bb-403d-97a3-b18601c386aa-istio-token\") pod \"e2e-shared-b-w6-2594a9-openshift-default-8694ddf5f7-v4jsh\" (UID: \"4c57fa36-80bb-403d-97a3-b18601c386aa\") " pod="openshift-ingress/e2e-shared-b-w6-2594a9-openshift-default-8694ddf5f7-v4jsh" Oct 31 00:46:12.419944 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:12.412916 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c57fa36-80bb-403d-97a3-b18601c386aa-istio-data\") pod \"e2e-shared-b-w6-2594a9-openshift-default-8694ddf5f7-v4jsh\" (UID: \"4c57fa36-80bb-403d-97a3-b18601c386aa\") " pod="openshift-ingress/e2e-shared-b-w6-2594a9-openshift-default-8694ddf5f7-v4jsh" Oct 31 00:46:12.419944 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:12.412997 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c57fa36-80bb-403d-97a3-b18601c386aa-istio-podinfo\") pod \"e2e-shared-b-w6-2594a9-openshift-default-8694ddf5f7-v4jsh\" (UID: \"4c57fa36-80bb-403d-97a3-b18601c386aa\") " pod="openshift-ingress/e2e-shared-b-w6-2594a9-openshift-default-8694ddf5f7-v4jsh" Oct 31 00:46:12.419944 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:12.413047 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c57fa36-80bb-403d-97a3-b18601c386aa-istio-envoy\") pod \"e2e-shared-b-w6-2594a9-openshift-default-8694ddf5f7-v4jsh\" (UID: \"4c57fa36-80bb-403d-97a3-b18601c386aa\") " pod="openshift-ingress/e2e-shared-b-w6-2594a9-openshift-default-8694ddf5f7-v4jsh" Oct 31 00:46:12.419944 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:12.413208 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c57fa36-80bb-403d-97a3-b18601c386aa-workload-socket\") pod \"e2e-shared-b-w6-2594a9-openshift-default-8694ddf5f7-v4jsh\" (UID: \"4c57fa36-80bb-403d-97a3-b18601c386aa\") " pod="openshift-ingress/e2e-shared-b-w6-2594a9-openshift-default-8694ddf5f7-v4jsh" Oct 31 00:46:12.419944 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:12.413258 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c57fa36-80bb-403d-97a3-b18601c386aa-workload-certs\") pod \"e2e-shared-b-w6-2594a9-openshift-default-8694ddf5f7-v4jsh\" (UID: \"4c57fa36-80bb-403d-97a3-b18601c386aa\") " pod="openshift-ingress/e2e-shared-b-w6-2594a9-openshift-default-8694ddf5f7-v4jsh" Oct 31 00:46:12.524627 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:12.524550 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c57fa36-80bb-403d-97a3-b18601c386aa-istiod-ca-cert\") pod \"e2e-shared-b-w6-2594a9-openshift-default-8694ddf5f7-v4jsh\" (UID: \"4c57fa36-80bb-403d-97a3-b18601c386aa\") " pod="openshift-ingress/e2e-shared-b-w6-2594a9-openshift-default-8694ddf5f7-v4jsh" Oct 31 00:46:12.524627 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:12.524608 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87hgh\" (UniqueName: \"kubernetes.io/projected/4c57fa36-80bb-403d-97a3-b18601c386aa-kube-api-access-87hgh\") pod \"e2e-shared-b-w6-2594a9-openshift-default-8694ddf5f7-v4jsh\" (UID: \"4c57fa36-80bb-403d-97a3-b18601c386aa\") " pod="openshift-ingress/e2e-shared-b-w6-2594a9-openshift-default-8694ddf5f7-v4jsh" Oct 31 00:46:12.524967 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:12.524663 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c57fa36-80bb-403d-97a3-b18601c386aa-credential-socket\") pod \"e2e-shared-b-w6-2594a9-openshift-default-8694ddf5f7-v4jsh\" (UID: \"4c57fa36-80bb-403d-97a3-b18601c386aa\") " pod="openshift-ingress/e2e-shared-b-w6-2594a9-openshift-default-8694ddf5f7-v4jsh" Oct 31 00:46:12.524967 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:12.524700 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c57fa36-80bb-403d-97a3-b18601c386aa-istio-token\") pod \"e2e-shared-b-w6-2594a9-openshift-default-8694ddf5f7-v4jsh\" (UID: \"4c57fa36-80bb-403d-97a3-b18601c386aa\") " pod="openshift-ingress/e2e-shared-b-w6-2594a9-openshift-default-8694ddf5f7-v4jsh" Oct 31 00:46:12.524967 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:12.524728 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c57fa36-80bb-403d-97a3-b18601c386aa-istio-data\") pod \"e2e-shared-b-w6-2594a9-openshift-default-8694ddf5f7-v4jsh\" (UID: \"4c57fa36-80bb-403d-97a3-b18601c386aa\") " pod="openshift-ingress/e2e-shared-b-w6-2594a9-openshift-default-8694ddf5f7-v4jsh" Oct 31 00:46:12.524967 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:12.524811 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c57fa36-80bb-403d-97a3-b18601c386aa-istio-podinfo\") pod \"e2e-shared-b-w6-2594a9-openshift-default-8694ddf5f7-v4jsh\" (UID: \"4c57fa36-80bb-403d-97a3-b18601c386aa\") " pod="openshift-ingress/e2e-shared-b-w6-2594a9-openshift-default-8694ddf5f7-v4jsh" Oct 31 00:46:12.524967 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:12.524862 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c57fa36-80bb-403d-97a3-b18601c386aa-istio-envoy\") pod \"e2e-shared-b-w6-2594a9-openshift-default-8694ddf5f7-v4jsh\" (UID: \"4c57fa36-80bb-403d-97a3-b18601c386aa\") " pod="openshift-ingress/e2e-shared-b-w6-2594a9-openshift-default-8694ddf5f7-v4jsh" Oct 31 00:46:12.525263 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:12.524998 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c57fa36-80bb-403d-97a3-b18601c386aa-workload-socket\") pod \"e2e-shared-b-w6-2594a9-openshift-default-8694ddf5f7-v4jsh\" (UID: \"4c57fa36-80bb-403d-97a3-b18601c386aa\") " pod="openshift-ingress/e2e-shared-b-w6-2594a9-openshift-default-8694ddf5f7-v4jsh" Oct 31 00:46:12.525263 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:12.525048 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c57fa36-80bb-403d-97a3-b18601c386aa-workload-certs\") pod \"e2e-shared-b-w6-2594a9-openshift-default-8694ddf5f7-v4jsh\" (UID: \"4c57fa36-80bb-403d-97a3-b18601c386aa\") " pod="openshift-ingress/e2e-shared-b-w6-2594a9-openshift-default-8694ddf5f7-v4jsh" Oct 31 00:46:12.525397 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:12.525348 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c57fa36-80bb-403d-97a3-b18601c386aa-istio-data\") pod \"e2e-shared-b-w6-2594a9-openshift-default-8694ddf5f7-v4jsh\" (UID: \"4c57fa36-80bb-403d-97a3-b18601c386aa\") " pod="openshift-ingress/e2e-shared-b-w6-2594a9-openshift-default-8694ddf5f7-v4jsh" Oct 31 00:46:12.525628 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:12.525523 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c57fa36-80bb-403d-97a3-b18601c386aa-istiod-ca-cert\") pod \"e2e-shared-b-w6-2594a9-openshift-default-8694ddf5f7-v4jsh\" (UID: \"4c57fa36-80bb-403d-97a3-b18601c386aa\") " pod="openshift-ingress/e2e-shared-b-w6-2594a9-openshift-default-8694ddf5f7-v4jsh" Oct 31 00:46:12.525628 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:12.525554 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c57fa36-80bb-403d-97a3-b18601c386aa-workload-certs\") pod \"e2e-shared-b-w6-2594a9-openshift-default-8694ddf5f7-v4jsh\" (UID: \"4c57fa36-80bb-403d-97a3-b18601c386aa\") " pod="openshift-ingress/e2e-shared-b-w6-2594a9-openshift-default-8694ddf5f7-v4jsh" Oct 31 00:46:12.525836 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:12.525811 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c57fa36-80bb-403d-97a3-b18601c386aa-workload-socket\") pod \"e2e-shared-b-w6-2594a9-openshift-default-8694ddf5f7-v4jsh\" (UID: \"4c57fa36-80bb-403d-97a3-b18601c386aa\") " pod="openshift-ingress/e2e-shared-b-w6-2594a9-openshift-default-8694ddf5f7-v4jsh" Oct 31 00:46:12.526061 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:12.526044 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c57fa36-80bb-403d-97a3-b18601c386aa-credential-socket\") pod \"e2e-shared-b-w6-2594a9-openshift-default-8694ddf5f7-v4jsh\" (UID: \"4c57fa36-80bb-403d-97a3-b18601c386aa\") " pod="openshift-ingress/e2e-shared-b-w6-2594a9-openshift-default-8694ddf5f7-v4jsh" Oct 31 00:46:12.528920 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:12.528821 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c57fa36-80bb-403d-97a3-b18601c386aa-istio-podinfo\") pod \"e2e-shared-b-w6-2594a9-openshift-default-8694ddf5f7-v4jsh\" (UID: \"4c57fa36-80bb-403d-97a3-b18601c386aa\") " pod="openshift-ingress/e2e-shared-b-w6-2594a9-openshift-default-8694ddf5f7-v4jsh" Oct 31 00:46:12.529362 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:12.529341 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c57fa36-80bb-403d-97a3-b18601c386aa-istio-envoy\") pod \"e2e-shared-b-w6-2594a9-openshift-default-8694ddf5f7-v4jsh\" (UID: \"4c57fa36-80bb-403d-97a3-b18601c386aa\") " pod="openshift-ingress/e2e-shared-b-w6-2594a9-openshift-default-8694ddf5f7-v4jsh" Oct 31 00:46:12.587336 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:12.587298 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87hgh\" (UniqueName: \"kubernetes.io/projected/4c57fa36-80bb-403d-97a3-b18601c386aa-kube-api-access-87hgh\") pod \"e2e-shared-b-w6-2594a9-openshift-default-8694ddf5f7-v4jsh\" (UID: \"4c57fa36-80bb-403d-97a3-b18601c386aa\") " pod="openshift-ingress/e2e-shared-b-w6-2594a9-openshift-default-8694ddf5f7-v4jsh" Oct 31 00:46:12.589023 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:12.588515 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c57fa36-80bb-403d-97a3-b18601c386aa-istio-token\") pod \"e2e-shared-b-w6-2594a9-openshift-default-8694ddf5f7-v4jsh\" (UID: \"4c57fa36-80bb-403d-97a3-b18601c386aa\") " pod="openshift-ingress/e2e-shared-b-w6-2594a9-openshift-default-8694ddf5f7-v4jsh" Oct 31 00:46:12.888872 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:12.888770 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-2594a9-openshift-default-8694ddf5f7-v4jsh" Oct 31 00:46:13.262618 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:13.255836 5222 patch_prober.go:29] interesting pod/e2e-shared-b-w4-0da668-openshift-default-5467f58b7-tzrb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Oct 31 00:46:13.262618 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:13.255896 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-0da668-openshift-default-5467f58b7-tzrb7" podUID="d5eb6fe4-1be0-468b-91e0-d398870307b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Oct 31 00:46:13.399012 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:46:13.398982 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c57fa36_80bb_403d_97a3_b18601c386aa.slice/crio-35532464daf3da8c07d46b0578bd0188319988020a4100e92a79900468105d94 WatchSource:0}: Error finding container 35532464daf3da8c07d46b0578bd0188319988020a4100e92a79900468105d94: Status 404 returned error can't find the container with id 35532464daf3da8c07d46b0578bd0188319988020a4100e92a79900468105d94 Oct 31 00:46:13.402051 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:13.402013 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:46:13.402162 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:13.402104 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:46:13.402162 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:13.402152 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:46:13.427569 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:13.427542 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-2594a9-openshift-default-8694ddf5f7-v4jsh"] Oct 31 00:46:14.052534 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:14.050493 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-0da668-openshift-default-5467f58b7-tzrb7" Oct 31 00:46:14.068629 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:14.067748 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-2594a9-openshift-default-8694ddf5f7-v4jsh" event={"ID":"4c57fa36-80bb-403d-97a3-b18601c386aa","Type":"ContainerStarted","Data":"0d360cc605ccde706b1a6139edee34d8c211f6e0ed82e81d3ee0caa7de814e0c"} Oct 31 00:46:14.068629 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:14.067818 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-2594a9-openshift-default-8694ddf5f7-v4jsh" event={"ID":"4c57fa36-80bb-403d-97a3-b18601c386aa","Type":"ContainerStarted","Data":"35532464daf3da8c07d46b0578bd0188319988020a4100e92a79900468105d94"} Oct 31 00:46:14.127409 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:14.123714 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c0f56c-6bbdcff4c-dswhj"] Oct 31 00:46:14.136147 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:14.132068 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c0f56c-6bbdcff4c-dswhj" Oct 31 00:46:14.205407 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:14.196192 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7hbk\" (UniqueName: \"kubernetes.io/projected/445117f4-90c9-4084-9847-30d6e03e1d2e-kube-api-access-x7hbk\") pod \"payload-pre-processing-e2e-api-a-c0f56c-6bbdcff4c-dswhj\" (UID: \"445117f4-90c9-4084-9847-30d6e03e1d2e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c0f56c-6bbdcff4c-dswhj" Oct 31 00:46:14.205407 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:14.196515 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/445117f4-90c9-4084-9847-30d6e03e1d2e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-c0f56c-6bbdcff4c-dswhj\" (UID: \"445117f4-90c9-4084-9847-30d6e03e1d2e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c0f56c-6bbdcff4c-dswhj" Oct 31 00:46:14.205407 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:14.203017 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-c0f56c-dockercfg-vw6b5\"" Oct 31 00:46:14.227402 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:14.221940 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-c0f56c\"" Oct 31 00:46:14.286685 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:14.239923 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c0f56c-6bbdcff4c-dswhj"] Oct 31 00:46:14.286685 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:14.262347 5222 patch_prober.go:29] interesting pod/e2e-shared-b-w4-0da668-openshift-default-5467f58b7-tzrb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Oct 31 00:46:14.286685 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:14.262422 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-0da668-openshift-default-5467f58b7-tzrb7" podUID="d5eb6fe4-1be0-468b-91e0-d398870307b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Oct 31 00:46:14.356452 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:14.306221 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/445117f4-90c9-4084-9847-30d6e03e1d2e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-c0f56c-6bbdcff4c-dswhj\" (UID: \"445117f4-90c9-4084-9847-30d6e03e1d2e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c0f56c-6bbdcff4c-dswhj" Oct 31 00:46:14.356452 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:14.336198 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/445117f4-90c9-4084-9847-30d6e03e1d2e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-c0f56c-6bbdcff4c-dswhj\" (UID: \"445117f4-90c9-4084-9847-30d6e03e1d2e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c0f56c-6bbdcff4c-dswhj" Oct 31 00:46:14.356452 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:14.336637 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7hbk\" (UniqueName: \"kubernetes.io/projected/445117f4-90c9-4084-9847-30d6e03e1d2e-kube-api-access-x7hbk\") pod \"payload-pre-processing-e2e-api-a-c0f56c-6bbdcff4c-dswhj\" (UID: \"445117f4-90c9-4084-9847-30d6e03e1d2e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c0f56c-6bbdcff4c-dswhj" Oct 31 00:46:14.429394 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:14.298342 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-2594a9-openshift-default-8694ddf5f7-v4jsh" podStartSLOduration=2.298316747 podStartE2EDuration="2.298316747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:46:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:46:14.288984535 +0000 UTC m=+1866.760154287" watchObservedRunningTime="2026-10-31 00:46:14.298316747 +0000 UTC m=+1866.769486496" Oct 31 00:46:14.440250 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:14.440169 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7hbk\" (UniqueName: \"kubernetes.io/projected/445117f4-90c9-4084-9847-30d6e03e1d2e-kube-api-access-x7hbk\") pod \"payload-pre-processing-e2e-api-a-c0f56c-6bbdcff4c-dswhj\" (UID: \"445117f4-90c9-4084-9847-30d6e03e1d2e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c0f56c-6bbdcff4c-dswhj" Oct 31 00:46:14.627787 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:14.597770 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c0f56c-6bbdcff4c-dswhj" Oct 31 00:46:14.647398 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:14.633905 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c0f56c-679fccf4f-wqpxd"] Oct 31 00:46:14.647398 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:14.640597 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c0f56c-679fccf4f-wqpxd" Oct 31 00:46:14.647398 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:14.645686 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c0f56c-679fccf4f-wqpxd"] Oct 31 00:46:14.769400 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:14.768850 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24572644-6def-4e05-bc4b-3ad70a7ca70d-plugins-config-volume\") pod \"payload-processing-e2e-api-a-c0f56c-679fccf4f-wqpxd\" (UID: \"24572644-6def-4e05-bc4b-3ad70a7ca70d\") " pod="openshift-ingress/payload-processing-e2e-api-a-c0f56c-679fccf4f-wqpxd" Oct 31 00:46:14.769400 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:14.769141 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frx5d\" (UniqueName: \"kubernetes.io/projected/24572644-6def-4e05-bc4b-3ad70a7ca70d-kube-api-access-frx5d\") pod \"payload-processing-e2e-api-a-c0f56c-679fccf4f-wqpxd\" (UID: \"24572644-6def-4e05-bc4b-3ad70a7ca70d\") " pod="openshift-ingress/payload-processing-e2e-api-a-c0f56c-679fccf4f-wqpxd" Oct 31 00:46:14.871362 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:14.871318 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frx5d\" (UniqueName: \"kubernetes.io/projected/24572644-6def-4e05-bc4b-3ad70a7ca70d-kube-api-access-frx5d\") pod \"payload-processing-e2e-api-a-c0f56c-679fccf4f-wqpxd\" (UID: \"24572644-6def-4e05-bc4b-3ad70a7ca70d\") " pod="openshift-ingress/payload-processing-e2e-api-a-c0f56c-679fccf4f-wqpxd" Oct 31 00:46:14.871718 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:14.871691 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24572644-6def-4e05-bc4b-3ad70a7ca70d-plugins-config-volume\") pod \"payload-processing-e2e-api-a-c0f56c-679fccf4f-wqpxd\" (UID: \"24572644-6def-4e05-bc4b-3ad70a7ca70d\") " pod="openshift-ingress/payload-processing-e2e-api-a-c0f56c-679fccf4f-wqpxd" Oct 31 00:46:14.872494 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:14.872475 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24572644-6def-4e05-bc4b-3ad70a7ca70d-plugins-config-volume\") pod \"payload-processing-e2e-api-a-c0f56c-679fccf4f-wqpxd\" (UID: \"24572644-6def-4e05-bc4b-3ad70a7ca70d\") " pod="openshift-ingress/payload-processing-e2e-api-a-c0f56c-679fccf4f-wqpxd" Oct 31 00:46:14.889176 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:14.889081 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-2594a9-openshift-default-8694ddf5f7-v4jsh" Oct 31 00:46:14.896727 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:14.896701 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-2594a9-openshift-default-8694ddf5f7-v4jsh" Oct 31 00:46:14.934299 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:14.934270 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frx5d\" (UniqueName: \"kubernetes.io/projected/24572644-6def-4e05-bc4b-3ad70a7ca70d-kube-api-access-frx5d\") pod \"payload-processing-e2e-api-a-c0f56c-679fccf4f-wqpxd\" (UID: \"24572644-6def-4e05-bc4b-3ad70a7ca70d\") " pod="openshift-ingress/payload-processing-e2e-api-a-c0f56c-679fccf4f-wqpxd" Oct 31 00:46:15.074705 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:15.074679 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-2594a9-openshift-default-8694ddf5f7-v4jsh" Oct 31 00:46:15.075735 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:15.075710 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-2594a9-openshift-default-8694ddf5f7-v4jsh" Oct 31 00:46:15.084111 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:15.084083 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c0f56c-679fccf4f-wqpxd" Oct 31 00:46:15.136066 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:15.136036 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c0f56c-6bbdcff4c-dswhj"] Oct 31 00:46:15.277308 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:15.276700 5222 patch_prober.go:29] interesting pod/e2e-shared-b-w4-0da668-openshift-default-5467f58b7-tzrb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Oct 31 00:46:15.277308 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:15.276757 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-0da668-openshift-default-5467f58b7-tzrb7" podUID="d5eb6fe4-1be0-468b-91e0-d398870307b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Oct 31 00:46:15.491779 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:15.491716 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c0f56c-679fccf4f-wqpxd"] Oct 31 00:46:16.083488 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:16.083448 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c0f56c-679fccf4f-wqpxd" event={"ID":"24572644-6def-4e05-bc4b-3ad70a7ca70d","Type":"ContainerStarted","Data":"69293cfe73f8cab07b9b7df9fe15c8611230e494942f7ffc599c6d9d17e91002"} Oct 31 00:46:16.083679 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:16.083505 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c0f56c-679fccf4f-wqpxd" event={"ID":"24572644-6def-4e05-bc4b-3ad70a7ca70d","Type":"ContainerStarted","Data":"f07ec7bbf5ee3bb687dd289df4a7647929c8abccfc07cdb4bd719336a3311962"} Oct 31 00:46:16.084508 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:16.084489 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-c0f56c-679fccf4f-wqpxd" Oct 31 00:46:16.086271 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:16.086245 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c0f56c-6bbdcff4c-dswhj" event={"ID":"445117f4-90c9-4084-9847-30d6e03e1d2e","Type":"ContainerStarted","Data":"c6d945ad8eb46cbe9a1244b07388903932fa13f5ce2839db3cd58918d1402c0a"} Oct 31 00:46:16.086438 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:16.086281 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c0f56c-6bbdcff4c-dswhj" event={"ID":"445117f4-90c9-4084-9847-30d6e03e1d2e","Type":"ContainerStarted","Data":"dcc71deb89eb7fa36bae7195b51f29adaa815d628942a76a5030273178dc220d"} Oct 31 00:46:16.086438 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:16.086317 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c0f56c-6bbdcff4c-dswhj" Oct 31 00:46:16.203323 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:16.203268 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-c0f56c-679fccf4f-wqpxd" podStartSLOduration=2.203255205 podStartE2EDuration="2.203255205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:46:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:46:16.201832434 +0000 UTC m=+1868.673002182" watchObservedRunningTime="2026-10-31 00:46:16.203255205 +0000 UTC m=+1868.674424954" Oct 31 00:46:16.263520 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:16.263481 5222 patch_prober.go:29] interesting pod/e2e-shared-b-w4-0da668-openshift-default-5467f58b7-tzrb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Oct 31 00:46:16.263731 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:16.263538 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-0da668-openshift-default-5467f58b7-tzrb7" podUID="d5eb6fe4-1be0-468b-91e0-d398870307b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Oct 31 00:46:17.260741 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:17.260711 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-0da668-openshift-default-5467f58b7-tzrb7" Oct 31 00:46:17.261957 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:17.261939 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-0da668-openshift-default-5467f58b7-tzrb7" Oct 31 00:46:17.447701 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:17.447645 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c0f56c-6bbdcff4c-dswhj" podStartSLOduration=3.447613759 podStartE2EDuration="3.447613759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:46:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:46:16.333223384 +0000 UTC m=+1868.804393133" watchObservedRunningTime="2026-10-31 00:46:17.447613759 +0000 UTC m=+1869.918783545" Oct 31 00:46:17.967442 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:17.966580 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e9c3c1-57685bbfff-ddgqw" Oct 31 00:46:27.098041 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:27.098012 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c0f56c-6bbdcff4c-dswhj" Oct 31 00:46:28.108412 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:28.108362 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-c0f56c-679fccf4f-wqpxd" Oct 31 00:46:35.795795 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:35.795757 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-79tk8"] Oct 31 00:46:36.052356 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:46:36.052261 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 00:46:46.644263 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:46.644231 5222 generic.go:356] "Generic (PLEG): container finished" podID="18761079-b183-45ca-9b61-cd9717b0195a" containerID="17f34d286c445ba41db967e1cc0a46cb4287bfcc862122233af07db17a2e4fc3" exitCode=0 Oct 31 00:46:46.644849 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:46.644313 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f891e6-openshift-default-565fdf54d7-56xb6" event={"ID":"18761079-b183-45ca-9b61-cd9717b0195a","Type":"ContainerDied","Data":"17f34d286c445ba41db967e1cc0a46cb4287bfcc862122233af07db17a2e4fc3"} Oct 31 00:46:46.644921 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:46.644904 5222 scope.go:122] "RemoveContainer" containerID="17f34d286c445ba41db967e1cc0a46cb4287bfcc862122233af07db17a2e4fc3" Oct 31 00:46:46.646568 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:46.646540 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:46:46.646656 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:46.646607 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:46:46.646656 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:46.646641 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:46:46.755329 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:46.755297 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7372-15e712"] Oct 31 00:46:46.758784 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:46.758762 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7372-15e712" Oct 31 00:46:46.785161 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:46.785130 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7372-15e712"] Oct 31 00:46:46.812588 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:46.812552 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-pdlhw\"" Oct 31 00:46:46.950721 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:46.950631 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtkj2\" (UniqueName: \"kubernetes.io/projected/38a0dd66-9fa7-4a2f-977e-3ab8c691d53a-kube-api-access-gtkj2\") pod \"test-curl-7372-15e712\" (UID: \"38a0dd66-9fa7-4a2f-977e-3ab8c691d53a\") " pod="opendatahub/test-curl-7372-15e712" Oct 31 00:46:47.052205 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:47.052161 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtkj2\" (UniqueName: \"kubernetes.io/projected/38a0dd66-9fa7-4a2f-977e-3ab8c691d53a-kube-api-access-gtkj2\") pod \"test-curl-7372-15e712\" (UID: \"38a0dd66-9fa7-4a2f-977e-3ab8c691d53a\") " pod="opendatahub/test-curl-7372-15e712" Oct 31 00:46:47.114667 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:47.114626 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtkj2\" (UniqueName: \"kubernetes.io/projected/38a0dd66-9fa7-4a2f-977e-3ab8c691d53a-kube-api-access-gtkj2\") pod \"test-curl-7372-15e712\" (UID: \"38a0dd66-9fa7-4a2f-977e-3ab8c691d53a\") " pod="opendatahub/test-curl-7372-15e712" Oct 31 00:46:47.380601 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:47.380565 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7372-15e712" Oct 31 00:46:47.688331 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:47.688235 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f891e6-openshift-default-565fdf54d7-56xb6" event={"ID":"18761079-b183-45ca-9b61-cd9717b0195a","Type":"ContainerStarted","Data":"55c5be21442828cfe86324df5a0b3caed961495f1c97af0b2ba6f559e359e926"} Oct 31 00:46:47.769048 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:46:47.769016 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod38a0dd66_9fa7_4a2f_977e_3ab8c691d53a.slice/crio-9de4387e4ffbd778732f2ddd5a6bd580a461442558813844b9c9a8a661a7bb9a WatchSource:0}: Error finding container 9de4387e4ffbd778732f2ddd5a6bd580a461442558813844b9c9a8a661a7bb9a: Status 404 returned error can't find the container with id 9de4387e4ffbd778732f2ddd5a6bd580a461442558813844b9c9a8a661a7bb9a Oct 31 00:46:47.771741 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:47.771721 5222 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 00:46:47.772030 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:47.772010 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7372-15e712"] Oct 31 00:46:48.386336 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:48.386291 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-f891e6-openshift-default-565fdf54d7-56xb6" Oct 31 00:46:48.387863 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:48.387832 5222 patch_prober.go:29] interesting pod/e2e-api-a-f891e6-openshift-default-565fdf54d7-56xb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body= Oct 31 00:46:48.387984 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:48.387889 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f891e6-openshift-default-565fdf54d7-56xb6" podUID="18761079-b183-45ca-9b61-cd9717b0195a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" Oct 31 00:46:48.705491 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:48.705363 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7372-15e712" event={"ID":"38a0dd66-9fa7-4a2f-977e-3ab8c691d53a","Type":"ContainerStarted","Data":"9de4387e4ffbd778732f2ddd5a6bd580a461442558813844b9c9a8a661a7bb9a"} Oct 31 00:46:49.386268 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:49.386237 5222 patch_prober.go:29] interesting pod/e2e-api-a-f891e6-openshift-default-565fdf54d7-56xb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body= Oct 31 00:46:49.386486 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:49.386304 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f891e6-openshift-default-565fdf54d7-56xb6" podUID="18761079-b183-45ca-9b61-cd9717b0195a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" Oct 31 00:46:49.714484 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:49.714453 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7372-15e712" event={"ID":"38a0dd66-9fa7-4a2f-977e-3ab8c691d53a","Type":"ContainerStarted","Data":"206fc25585d006760816d1778cf63569a82c41f117e1b4aa4cb7e8338f027570"} Oct 31 00:46:49.831616 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:49.831563 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7372-15e712" podStartSLOduration=2.426022985 podStartE2EDuration="3.831548331s" totalImagesPullingTime="1.405525346s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:46:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:46:49.83028425 +0000 UTC m=+1902.301454000" watchObservedRunningTime="2026-10-31 00:46:49.831548331 +0000 UTC m=+1902.302718079" Oct 31 00:46:50.386339 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:50.386307 5222 patch_prober.go:29] interesting pod/e2e-api-a-f891e6-openshift-default-565fdf54d7-56xb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body= Oct 31 00:46:50.386562 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:50.386401 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f891e6-openshift-default-565fdf54d7-56xb6" podUID="18761079-b183-45ca-9b61-cd9717b0195a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" Oct 31 00:46:51.369128 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:51.369094 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7372-15e712"] Oct 31 00:46:51.376318 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:51.376283 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7372-15e712"] Oct 31 00:46:51.386430 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:51.386400 5222 patch_prober.go:29] interesting pod/e2e-api-a-f891e6-openshift-default-565fdf54d7-56xb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body= Oct 31 00:46:51.386591 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:51.386457 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f891e6-openshift-default-565fdf54d7-56xb6" podUID="18761079-b183-45ca-9b61-cd9717b0195a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" Oct 31 00:46:51.730678 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:51.730579 5222 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7372-15e712" podUID="38a0dd66-9fa7-4a2f-977e-3ab8c691d53a" containerName="test-curl-7372-15e712" containerID="cri-o://206fc25585d006760816d1778cf63569a82c41f117e1b4aa4cb7e8338f027570" gracePeriod=2 Oct 31 00:46:52.103649 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:52.103613 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f891e6-openshift-default-565fdf54d7-56xb6" Oct 31 00:46:52.387088 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:52.386993 5222 patch_prober.go:29] interesting pod/e2e-api-a-f891e6-openshift-default-565fdf54d7-56xb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body= Oct 31 00:46:52.387088 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:52.387060 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f891e6-openshift-default-565fdf54d7-56xb6" podUID="18761079-b183-45ca-9b61-cd9717b0195a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" Oct 31 00:46:52.507294 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:52.507259 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7d5f4de7-npdwh"] Oct 31 00:46:52.508041 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:52.508021 5222 cpu_manager.go:414] "containerMap: removing container" podUID="38a0dd66-9fa7-4a2f-977e-3ab8c691d53a" containerName="test-curl-7372-15e712" Oct 31 00:46:52.508106 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:52.508043 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38a0dd66-9fa7-4a2f-977e-3ab8c691d53a" containerName="test-curl-7372-15e712" Oct 31 00:46:52.508167 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:52.508158 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="38a0dd66-9fa7-4a2f-977e-3ab8c691d53a" containerName="test-curl-7372-15e712" Oct 31 00:46:52.510714 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:52.510697 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7d5f4de7-npdwh" Oct 31 00:46:52.567724 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:52.567682 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10937a00-0cff-4b28-9a52-6cef09efe247-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-7d5f4de7-npdwh\" (UID: \"10937a00-0cff-4b28-9a52-6cef09efe247\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7d5f4de7-npdwh" Oct 31 00:46:52.567972 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:52.567945 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcdmb\" (UniqueName: \"kubernetes.io/projected/10937a00-0cff-4b28-9a52-6cef09efe247-kube-api-access-bcdmb\") pod \"maas-api-revoke-keys-e2e-mt-7d5f4de7-npdwh\" (UID: \"10937a00-0cff-4b28-9a52-6cef09efe247\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7d5f4de7-npdwh" Oct 31 00:46:52.568112 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:52.567989 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10937a00-0cff-4b28-9a52-6cef09efe247-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-7d5f4de7-npdwh\" (UID: \"10937a00-0cff-4b28-9a52-6cef09efe247\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7d5f4de7-npdwh" Oct 31 00:46:52.568401 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:52.568360 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:46:52.568518 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:52.568492 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:46:52.570280 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:52.570256 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7d5f4de7-npdwh"] Oct 31 00:46:52.616523 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:52.616497 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfz9z\"" Oct 31 00:46:52.616523 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:52.616500 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:46:52.669980 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:52.669889 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcdmb\" (UniqueName: \"kubernetes.io/projected/10937a00-0cff-4b28-9a52-6cef09efe247-kube-api-access-bcdmb\") pod \"maas-api-revoke-keys-e2e-mt-7d5f4de7-npdwh\" (UID: \"10937a00-0cff-4b28-9a52-6cef09efe247\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7d5f4de7-npdwh" Oct 31 00:46:52.669980 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:52.669931 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10937a00-0cff-4b28-9a52-6cef09efe247-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-7d5f4de7-npdwh\" (UID: \"10937a00-0cff-4b28-9a52-6cef09efe247\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7d5f4de7-npdwh" Oct 31 00:46:52.670242 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:52.670030 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10937a00-0cff-4b28-9a52-6cef09efe247-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-7d5f4de7-npdwh\" (UID: \"10937a00-0cff-4b28-9a52-6cef09efe247\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7d5f4de7-npdwh" Oct 31 00:46:52.670767 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:52.670741 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10937a00-0cff-4b28-9a52-6cef09efe247-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-7d5f4de7-npdwh\" (UID: \"10937a00-0cff-4b28-9a52-6cef09efe247\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7d5f4de7-npdwh" Oct 31 00:46:52.672708 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:52.672687 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10937a00-0cff-4b28-9a52-6cef09efe247-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-7d5f4de7-npdwh\" (UID: \"10937a00-0cff-4b28-9a52-6cef09efe247\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7d5f4de7-npdwh" Oct 31 00:46:52.733418 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:52.733383 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcdmb\" (UniqueName: \"kubernetes.io/projected/10937a00-0cff-4b28-9a52-6cef09efe247-kube-api-access-bcdmb\") pod \"maas-api-revoke-keys-e2e-mt-7d5f4de7-npdwh\" (UID: \"10937a00-0cff-4b28-9a52-6cef09efe247\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7d5f4de7-npdwh" Oct 31 00:46:52.824210 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:52.824173 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7d5f4de7-npdwh" Oct 31 00:46:53.241866 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:46:53.241834 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10937a00_0cff_4b28_9a52_6cef09efe247.slice/crio-55ae58f7ee823aa0cbf629c981ba2bd0d7a5d7431b6823c9ecdc821828df253d WatchSource:0}: Error finding container 55ae58f7ee823aa0cbf629c981ba2bd0d7a5d7431b6823c9ecdc821828df253d: Status 404 returned error can't find the container with id 55ae58f7ee823aa0cbf629c981ba2bd0d7a5d7431b6823c9ecdc821828df253d Oct 31 00:46:53.245725 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:53.245696 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7d5f4de7-npdwh"] Oct 31 00:46:53.391816 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:53.391784 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-f891e6-openshift-default-565fdf54d7-56xb6" Oct 31 00:46:53.393162 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:53.393139 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-f891e6-openshift-default-565fdf54d7-56xb6" Oct 31 00:46:53.759715 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:53.759679 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7d5f4de7-npdwh" event={"ID":"10937a00-0cff-4b28-9a52-6cef09efe247","Type":"ContainerStarted","Data":"1b53ba0c0585ba3b15fddb818908d77734c1620fbdf074a161c4b104a9916bc9"} Oct 31 00:46:53.759903 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:53.759748 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7d5f4de7-npdwh" event={"ID":"10937a00-0cff-4b28-9a52-6cef09efe247","Type":"ContainerStarted","Data":"55ae58f7ee823aa0cbf629c981ba2bd0d7a5d7431b6823c9ecdc821828df253d"} Oct 31 00:46:53.880156 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:53.880099 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7d5f4de7-npdwh" podStartSLOduration=1.880082461 podStartE2EDuration="1.880082461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:46:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:46:53.876777521 +0000 UTC m=+1906.347947284" watchObservedRunningTime="2026-10-31 00:46:53.880082461 +0000 UTC m=+1906.351252210" Oct 31 00:46:53.995556 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:53.995531 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7372-15e712" Oct 31 00:46:54.101596 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:54.101566 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38a0dd66-9fa7-4a2f-977e-3ab8c691d53a-kube-api-access-gtkj2\" (UniqueName: \"kubernetes.io/projected/38a0dd66-9fa7-4a2f-977e-3ab8c691d53a-kube-api-access-gtkj2\") pod \"38a0dd66-9fa7-4a2f-977e-3ab8c691d53a\" (UID: \"38a0dd66-9fa7-4a2f-977e-3ab8c691d53a\") " Oct 31 00:46:54.104217 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:54.104190 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38a0dd66-9fa7-4a2f-977e-3ab8c691d53a-kube-api-access-gtkj2" pod "38a0dd66-9fa7-4a2f-977e-3ab8c691d53a" (UID: "38a0dd66-9fa7-4a2f-977e-3ab8c691d53a"). InnerVolumeSpecName "kube-api-access-gtkj2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:46:54.203281 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:54.203236 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtkj2\" (UniqueName: \"kubernetes.io/projected/38a0dd66-9fa7-4a2f-977e-3ab8c691d53a-kube-api-access-gtkj2\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:46:54.769854 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:54.769817 5222 generic.go:356] "Generic (PLEG): container finished" podID="10937a00-0cff-4b28-9a52-6cef09efe247" containerID="1b53ba0c0585ba3b15fddb818908d77734c1620fbdf074a161c4b104a9916bc9" exitCode=0 Oct 31 00:46:54.770330 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:54.769907 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7d5f4de7-npdwh" event={"ID":"10937a00-0cff-4b28-9a52-6cef09efe247","Type":"ContainerDied","Data":"1b53ba0c0585ba3b15fddb818908d77734c1620fbdf074a161c4b104a9916bc9"} Oct 31 00:46:54.771630 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:54.771604 5222 generic.go:356] "Generic (PLEG): container finished" podID="38a0dd66-9fa7-4a2f-977e-3ab8c691d53a" containerID="206fc25585d006760816d1778cf63569a82c41f117e1b4aa4cb7e8338f027570" exitCode=137 Oct 31 00:46:54.771764 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:54.771647 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7372-15e712" Oct 31 00:46:54.771764 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:54.771673 5222 scope.go:122] "RemoveContainer" containerID="206fc25585d006760816d1778cf63569a82c41f117e1b4aa4cb7e8338f027570" Oct 31 00:46:54.783806 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:54.783785 5222 scope.go:122] "RemoveContainer" containerID="206fc25585d006760816d1778cf63569a82c41f117e1b4aa4cb7e8338f027570" Oct 31 00:46:54.784160 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:46:54.784138 5222 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"206fc25585d006760816d1778cf63569a82c41f117e1b4aa4cb7e8338f027570\": container with ID starting with 206fc25585d006760816d1778cf63569a82c41f117e1b4aa4cb7e8338f027570 not found: ID does not exist" containerID="206fc25585d006760816d1778cf63569a82c41f117e1b4aa4cb7e8338f027570" Oct 31 00:46:54.784241 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:54.784168 5222 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"206fc25585d006760816d1778cf63569a82c41f117e1b4aa4cb7e8338f027570"} err="failed to get container status \"206fc25585d006760816d1778cf63569a82c41f117e1b4aa4cb7e8338f027570\": rpc error: code = NotFound desc = could not find container \"206fc25585d006760816d1778cf63569a82c41f117e1b4aa4cb7e8338f027570\": container with ID starting with 206fc25585d006760816d1778cf63569a82c41f117e1b4aa4cb7e8338f027570 not found: ID does not exist" Oct 31 00:46:54.941344 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:46:54.941250 5222 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7372-15e712\" is forbidden: User \"system:node:ip-10-0-137-56.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-56.us-west-2.compute.internal' and this object" podUID="38a0dd66-9fa7-4a2f-977e-3ab8c691d53a" pod="opendatahub/test-curl-7372-15e712" Oct 31 00:46:55.659214 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:55.659183 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38a0dd66-9fa7-4a2f-977e-3ab8c691d53a" path="/var/lib/kubelet/pods/38a0dd66-9fa7-4a2f-977e-3ab8c691d53a/volumes" Oct 31 00:46:55.947060 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:55.947035 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7d5f4de7-npdwh" Oct 31 00:46:56.028432 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:56.028347 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10937a00-0cff-4b28-9a52-6cef09efe247-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10937a00-0cff-4b28-9a52-6cef09efe247-maas-api-xks-ca\") pod \"10937a00-0cff-4b28-9a52-6cef09efe247\" (UID: \"10937a00-0cff-4b28-9a52-6cef09efe247\") " Oct 31 00:46:56.028585 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:56.028522 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10937a00-0cff-4b28-9a52-6cef09efe247-kube-api-access-bcdmb\" (UniqueName: \"kubernetes.io/projected/10937a00-0cff-4b28-9a52-6cef09efe247-kube-api-access-bcdmb\") pod \"10937a00-0cff-4b28-9a52-6cef09efe247\" (UID: \"10937a00-0cff-4b28-9a52-6cef09efe247\") " Oct 31 00:46:56.028585 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:56.028543 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10937a00-0cff-4b28-9a52-6cef09efe247-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10937a00-0cff-4b28-9a52-6cef09efe247-maas-api-openshift-ca\") pod \"10937a00-0cff-4b28-9a52-6cef09efe247\" (UID: \"10937a00-0cff-4b28-9a52-6cef09efe247\") " Oct 31 00:46:56.029011 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:56.028975 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10937a00-0cff-4b28-9a52-6cef09efe247-maas-api-openshift-ca" pod "10937a00-0cff-4b28-9a52-6cef09efe247" (UID: "10937a00-0cff-4b28-9a52-6cef09efe247"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:46:56.030667 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:56.030641 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10937a00-0cff-4b28-9a52-6cef09efe247-maas-api-xks-ca" pod "10937a00-0cff-4b28-9a52-6cef09efe247" (UID: "10937a00-0cff-4b28-9a52-6cef09efe247"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:46:56.030871 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:56.030840 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10937a00-0cff-4b28-9a52-6cef09efe247-kube-api-access-bcdmb" pod "10937a00-0cff-4b28-9a52-6cef09efe247" (UID: "10937a00-0cff-4b28-9a52-6cef09efe247"). InnerVolumeSpecName "kube-api-access-bcdmb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:46:56.130463 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:56.130425 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10937a00-0cff-4b28-9a52-6cef09efe247-maas-api-xks-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:46:56.130463 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:56.130458 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcdmb\" (UniqueName: \"kubernetes.io/projected/10937a00-0cff-4b28-9a52-6cef09efe247-kube-api-access-bcdmb\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:46:56.130463 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:56.130469 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10937a00-0cff-4b28-9a52-6cef09efe247-maas-api-openshift-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:46:56.841787 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:56.841143 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7d5f4de7-npdwh" event={"ID":"10937a00-0cff-4b28-9a52-6cef09efe247","Type":"ContainerDied","Data":"55ae58f7ee823aa0cbf629c981ba2bd0d7a5d7431b6823c9ecdc821828df253d"} Oct 31 00:46:56.841787 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:56.841206 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="55ae58f7ee823aa0cbf629c981ba2bd0d7a5d7431b6823c9ecdc821828df253d" Oct 31 00:46:56.841787 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:46:56.841276 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7d5f4de7-npdwh" Oct 31 00:47:05.893359 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:47:05.893319 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 00:47:09.242771 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:09.242737 5222 scope.go:122] "RemoveContainer" containerID="9f6dd793ff8d0d0f8c0af3501a89585eb64eedaadc9abf3f077c7765644cdb41" Oct 31 00:47:11.817088 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:11.817050 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-r57tc"] Oct 31 00:47:11.817634 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:11.817267 5222 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-r57tc" podUID="33d0dda2-34fd-409e-9165-5861f2373db9" containerName="istio-proxy" containerID="cri-o://daba08f2af2f279c5bc195222337caad8b9790336f58e37cdfd843d3b391ca69" gracePeriod=30 Oct 31 00:47:11.868449 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:11.868410 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mjxmq"] Oct 31 00:47:11.868731 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:11.868679 5222 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mjxmq" podUID="0b9592e3-41e5-4ada-9e42-ea705c3eb1f3" containerName="istio-proxy" containerID="cri-o://a1e2c5bf2768aaf6e70d1488262ba6fc79c1d52e426ab722ba9126e949dcd73d" gracePeriod=30 Oct 31 00:47:11.874421 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:11.872500 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lkfxl"] Oct 31 00:47:11.874421 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:11.872733 5222 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lkfxl" podUID="21900006-51c4-44ab-848d-68f885effa5f" containerName="istio-proxy" containerID="cri-o://b462c5e3696412164e49cd679a7dec56a51ec00d8f2db0ac8e72f6a41edb9bb5" gracePeriod=30 Oct 31 00:47:15.921346 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:15.921309 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-79tk8"] Oct 31 00:47:17.000986 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:17.000946 5222 generic.go:356] "Generic (PLEG): container finished" podID="21900006-51c4-44ab-848d-68f885effa5f" containerID="b462c5e3696412164e49cd679a7dec56a51ec00d8f2db0ac8e72f6a41edb9bb5" exitCode=0 Oct 31 00:47:17.001470 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:17.001054 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lkfxl" event={"ID":"21900006-51c4-44ab-848d-68f885effa5f","Type":"ContainerDied","Data":"b462c5e3696412164e49cd679a7dec56a51ec00d8f2db0ac8e72f6a41edb9bb5"} Oct 31 00:47:17.003449 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:17.003413 5222 generic.go:356] "Generic (PLEG): container finished" podID="33d0dda2-34fd-409e-9165-5861f2373db9" containerID="daba08f2af2f279c5bc195222337caad8b9790336f58e37cdfd843d3b391ca69" exitCode=0 Oct 31 00:47:17.003604 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:17.003565 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-r57tc" event={"ID":"33d0dda2-34fd-409e-9165-5861f2373db9","Type":"ContainerDied","Data":"daba08f2af2f279c5bc195222337caad8b9790336f58e37cdfd843d3b391ca69"} Oct 31 00:47:17.005864 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:17.005688 5222 generic.go:356] "Generic (PLEG): container finished" podID="0b9592e3-41e5-4ada-9e42-ea705c3eb1f3" containerID="a1e2c5bf2768aaf6e70d1488262ba6fc79c1d52e426ab722ba9126e949dcd73d" exitCode=0 Oct 31 00:47:17.005864 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:17.005802 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mjxmq" event={"ID":"0b9592e3-41e5-4ada-9e42-ea705c3eb1f3","Type":"ContainerDied","Data":"a1e2c5bf2768aaf6e70d1488262ba6fc79c1d52e426ab722ba9126e949dcd73d"} Oct 31 00:47:17.219412 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:17.219292 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-r57tc" Oct 31 00:47:17.298080 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:17.298054 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lkfxl" Oct 31 00:47:17.301963 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:17.301939 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mjxmq" Oct 31 00:47:17.304176 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:17.304154 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33d0dda2-34fd-409e-9165-5861f2373db9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33d0dda2-34fd-409e-9165-5861f2373db9-istio-data\") pod \"33d0dda2-34fd-409e-9165-5861f2373db9\" (UID: \"33d0dda2-34fd-409e-9165-5861f2373db9\") " Oct 31 00:47:17.304301 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:17.304221 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33d0dda2-34fd-409e-9165-5861f2373db9-istio-token\" (UniqueName: \"kubernetes.io/projected/33d0dda2-34fd-409e-9165-5861f2373db9-istio-token\") pod \"33d0dda2-34fd-409e-9165-5861f2373db9\" (UID: \"33d0dda2-34fd-409e-9165-5861f2373db9\") " Oct 31 00:47:17.304301 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:17.304254 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33d0dda2-34fd-409e-9165-5861f2373db9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33d0dda2-34fd-409e-9165-5861f2373db9-workload-socket\") pod \"33d0dda2-34fd-409e-9165-5861f2373db9\" (UID: \"33d0dda2-34fd-409e-9165-5861f2373db9\") " Oct 31 00:47:17.304443 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:17.304317 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33d0dda2-34fd-409e-9165-5861f2373db9-kube-api-access-vbvqj\" (UniqueName: \"kubernetes.io/projected/33d0dda2-34fd-409e-9165-5861f2373db9-kube-api-access-vbvqj\") pod \"33d0dda2-34fd-409e-9165-5861f2373db9\" (UID: \"33d0dda2-34fd-409e-9165-5861f2373db9\") " Oct 31 00:47:17.304443 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:17.304395 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33d0dda2-34fd-409e-9165-5861f2373db9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33d0dda2-34fd-409e-9165-5861f2373db9-credential-socket\") pod \"33d0dda2-34fd-409e-9165-5861f2373db9\" (UID: \"33d0dda2-34fd-409e-9165-5861f2373db9\") " Oct 31 00:47:17.304559 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:17.304479 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33d0dda2-34fd-409e-9165-5861f2373db9-istio-data" pod "33d0dda2-34fd-409e-9165-5861f2373db9" (UID: "33d0dda2-34fd-409e-9165-5861f2373db9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:47:17.304559 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:17.304497 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/33d0dda2-34fd-409e-9165-5861f2373db9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33d0dda2-34fd-409e-9165-5861f2373db9-istio-podinfo\") pod \"33d0dda2-34fd-409e-9165-5861f2373db9\" (UID: \"33d0dda2-34fd-409e-9165-5861f2373db9\") " Oct 31 00:47:17.304559 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:17.304529 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33d0dda2-34fd-409e-9165-5861f2373db9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33d0dda2-34fd-409e-9165-5861f2373db9-istiod-ca-cert\") pod \"33d0dda2-34fd-409e-9165-5861f2373db9\" (UID: \"33d0dda2-34fd-409e-9165-5861f2373db9\") " Oct 31 00:47:17.304711 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:17.304578 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33d0dda2-34fd-409e-9165-5861f2373db9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33d0dda2-34fd-409e-9165-5861f2373db9-istio-envoy\") pod \"33d0dda2-34fd-409e-9165-5861f2373db9\" (UID: \"33d0dda2-34fd-409e-9165-5861f2373db9\") " Oct 31 00:47:17.304711 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:17.304615 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33d0dda2-34fd-409e-9165-5861f2373db9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33d0dda2-34fd-409e-9165-5861f2373db9-workload-certs\") pod \"33d0dda2-34fd-409e-9165-5861f2373db9\" (UID: \"33d0dda2-34fd-409e-9165-5861f2373db9\") " Oct 31 00:47:17.304711 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:17.304616 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33d0dda2-34fd-409e-9165-5861f2373db9-workload-socket" pod "33d0dda2-34fd-409e-9165-5861f2373db9" (UID: "33d0dda2-34fd-409e-9165-5861f2373db9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:47:17.304978 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:17.304948 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33d0dda2-34fd-409e-9165-5861f2373db9-workload-certs" pod "33d0dda2-34fd-409e-9165-5861f2373db9" (UID: "33d0dda2-34fd-409e-9165-5861f2373db9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:47:17.305127 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:17.305104 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33d0dda2-34fd-409e-9165-5861f2373db9-istiod-ca-cert" pod "33d0dda2-34fd-409e-9165-5861f2373db9" (UID: "33d0dda2-34fd-409e-9165-5861f2373db9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:47:17.307417 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:17.305391 5222 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33d0dda2-34fd-409e-9165-5861f2373db9-istio-data\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:17.307417 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:17.305411 5222 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33d0dda2-34fd-409e-9165-5861f2373db9-workload-socket\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:17.307417 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:17.305425 5222 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33d0dda2-34fd-409e-9165-5861f2373db9-istiod-ca-cert\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:17.307417 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:17.305437 5222 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33d0dda2-34fd-409e-9165-5861f2373db9-workload-certs\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:17.307417 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:17.305457 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33d0dda2-34fd-409e-9165-5861f2373db9-credential-socket" pod "33d0dda2-34fd-409e-9165-5861f2373db9" (UID: "33d0dda2-34fd-409e-9165-5861f2373db9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:47:17.307417 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:17.306909 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33d0dda2-34fd-409e-9165-5861f2373db9-istio-envoy" pod "33d0dda2-34fd-409e-9165-5861f2373db9" (UID: "33d0dda2-34fd-409e-9165-5861f2373db9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:47:17.307417 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:17.307133 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/33d0dda2-34fd-409e-9165-5861f2373db9-istio-podinfo" pod "33d0dda2-34fd-409e-9165-5861f2373db9" (UID: "33d0dda2-34fd-409e-9165-5861f2373db9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:47:17.307417 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:17.307278 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33d0dda2-34fd-409e-9165-5861f2373db9-istio-token" pod "33d0dda2-34fd-409e-9165-5861f2373db9" (UID: "33d0dda2-34fd-409e-9165-5861f2373db9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:47:17.308356 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:17.308333 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33d0dda2-34fd-409e-9165-5861f2373db9-kube-api-access-vbvqj" pod "33d0dda2-34fd-409e-9165-5861f2373db9" (UID: "33d0dda2-34fd-409e-9165-5861f2373db9"). InnerVolumeSpecName "kube-api-access-vbvqj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:47:17.406166 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:17.406074 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b9592e3-41e5-4ada-9e42-ea705c3eb1f3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b9592e3-41e5-4ada-9e42-ea705c3eb1f3-workload-socket\") pod \"0b9592e3-41e5-4ada-9e42-ea705c3eb1f3\" (UID: \"0b9592e3-41e5-4ada-9e42-ea705c3eb1f3\") " Oct 31 00:47:17.406166 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:17.406119 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b9592e3-41e5-4ada-9e42-ea705c3eb1f3-istio-token\" (UniqueName: \"kubernetes.io/projected/0b9592e3-41e5-4ada-9e42-ea705c3eb1f3-istio-token\") pod \"0b9592e3-41e5-4ada-9e42-ea705c3eb1f3\" (UID: \"0b9592e3-41e5-4ada-9e42-ea705c3eb1f3\") " Oct 31 00:47:17.406166 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:17.406138 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21900006-51c4-44ab-848d-68f885effa5f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21900006-51c4-44ab-848d-68f885effa5f-credential-socket\") pod \"21900006-51c4-44ab-848d-68f885effa5f\" (UID: \"21900006-51c4-44ab-848d-68f885effa5f\") " Oct 31 00:47:17.406525 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:17.406175 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21900006-51c4-44ab-848d-68f885effa5f-istio-token\" (UniqueName: \"kubernetes.io/projected/21900006-51c4-44ab-848d-68f885effa5f-istio-token\") pod \"21900006-51c4-44ab-848d-68f885effa5f\" (UID: \"21900006-51c4-44ab-848d-68f885effa5f\") " Oct 31 00:47:17.406525 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:17.406218 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b9592e3-41e5-4ada-9e42-ea705c3eb1f3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b9592e3-41e5-4ada-9e42-ea705c3eb1f3-istiod-ca-cert\") pod \"0b9592e3-41e5-4ada-9e42-ea705c3eb1f3\" (UID: \"0b9592e3-41e5-4ada-9e42-ea705c3eb1f3\") " Oct 31 00:47:17.406525 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:17.406243 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b9592e3-41e5-4ada-9e42-ea705c3eb1f3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b9592e3-41e5-4ada-9e42-ea705c3eb1f3-workload-certs\") pod \"0b9592e3-41e5-4ada-9e42-ea705c3eb1f3\" (UID: \"0b9592e3-41e5-4ada-9e42-ea705c3eb1f3\") " Oct 31 00:47:17.406525 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:17.406281 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21900006-51c4-44ab-848d-68f885effa5f-kube-api-access-m9mnp\" (UniqueName: \"kubernetes.io/projected/21900006-51c4-44ab-848d-68f885effa5f-kube-api-access-m9mnp\") pod \"21900006-51c4-44ab-848d-68f885effa5f\" (UID: \"21900006-51c4-44ab-848d-68f885effa5f\") " Oct 31 00:47:17.406525 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:17.406319 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21900006-51c4-44ab-848d-68f885effa5f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21900006-51c4-44ab-848d-68f885effa5f-istio-envoy\") pod \"21900006-51c4-44ab-848d-68f885effa5f\" (UID: \"21900006-51c4-44ab-848d-68f885effa5f\") " Oct 31 00:47:17.406525 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:17.406343 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b9592e3-41e5-4ada-9e42-ea705c3eb1f3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b9592e3-41e5-4ada-9e42-ea705c3eb1f3-istio-data\") pod \"0b9592e3-41e5-4ada-9e42-ea705c3eb1f3\" (UID: \"0b9592e3-41e5-4ada-9e42-ea705c3eb1f3\") " Oct 31 00:47:17.406525 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:17.406424 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21900006-51c4-44ab-848d-68f885effa5f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21900006-51c4-44ab-848d-68f885effa5f-workload-socket\") pod \"21900006-51c4-44ab-848d-68f885effa5f\" (UID: \"21900006-51c4-44ab-848d-68f885effa5f\") " Oct 31 00:47:17.406525 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:17.406466 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21900006-51c4-44ab-848d-68f885effa5f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21900006-51c4-44ab-848d-68f885effa5f-istio-data\") pod \"21900006-51c4-44ab-848d-68f885effa5f\" (UID: \"21900006-51c4-44ab-848d-68f885effa5f\") " Oct 31 00:47:17.406525 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:17.406487 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b9592e3-41e5-4ada-9e42-ea705c3eb1f3-workload-socket" pod "0b9592e3-41e5-4ada-9e42-ea705c3eb1f3" (UID: "0b9592e3-41e5-4ada-9e42-ea705c3eb1f3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:47:17.406525 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:17.406499 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21900006-51c4-44ab-848d-68f885effa5f-credential-socket" pod "21900006-51c4-44ab-848d-68f885effa5f" (UID: "21900006-51c4-44ab-848d-68f885effa5f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:47:17.407042 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:17.406533 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b9592e3-41e5-4ada-9e42-ea705c3eb1f3-kube-api-access-qbtc5\" (UniqueName: \"kubernetes.io/projected/0b9592e3-41e5-4ada-9e42-ea705c3eb1f3-kube-api-access-qbtc5\") pod \"0b9592e3-41e5-4ada-9e42-ea705c3eb1f3\" (UID: \"0b9592e3-41e5-4ada-9e42-ea705c3eb1f3\") " Oct 31 00:47:17.407042 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:17.406573 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21900006-51c4-44ab-848d-68f885effa5f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21900006-51c4-44ab-848d-68f885effa5f-workload-certs\") pod \"21900006-51c4-44ab-848d-68f885effa5f\" (UID: \"21900006-51c4-44ab-848d-68f885effa5f\") " Oct 31 00:47:17.407042 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:17.406604 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b9592e3-41e5-4ada-9e42-ea705c3eb1f3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b9592e3-41e5-4ada-9e42-ea705c3eb1f3-istio-envoy\") pod \"0b9592e3-41e5-4ada-9e42-ea705c3eb1f3\" (UID: \"0b9592e3-41e5-4ada-9e42-ea705c3eb1f3\") " Oct 31 00:47:17.407042 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:17.406648 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0b9592e3-41e5-4ada-9e42-ea705c3eb1f3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b9592e3-41e5-4ada-9e42-ea705c3eb1f3-istio-podinfo\") pod \"0b9592e3-41e5-4ada-9e42-ea705c3eb1f3\" (UID: \"0b9592e3-41e5-4ada-9e42-ea705c3eb1f3\") " Oct 31 00:47:17.407042 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:17.406679 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21900006-51c4-44ab-848d-68f885effa5f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21900006-51c4-44ab-848d-68f885effa5f-istiod-ca-cert\") pod \"21900006-51c4-44ab-848d-68f885effa5f\" (UID: \"21900006-51c4-44ab-848d-68f885effa5f\") " Oct 31 00:47:17.407042 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:17.406735 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b9592e3-41e5-4ada-9e42-ea705c3eb1f3-istio-data" pod "0b9592e3-41e5-4ada-9e42-ea705c3eb1f3" (UID: "0b9592e3-41e5-4ada-9e42-ea705c3eb1f3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:47:17.407042 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:17.406753 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b9592e3-41e5-4ada-9e42-ea705c3eb1f3-workload-certs" pod "0b9592e3-41e5-4ada-9e42-ea705c3eb1f3" (UID: "0b9592e3-41e5-4ada-9e42-ea705c3eb1f3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:47:17.407042 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:17.406847 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/21900006-51c4-44ab-848d-68f885effa5f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21900006-51c4-44ab-848d-68f885effa5f-istio-podinfo\") pod \"21900006-51c4-44ab-848d-68f885effa5f\" (UID: \"21900006-51c4-44ab-848d-68f885effa5f\") " Oct 31 00:47:17.407042 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:17.406981 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b9592e3-41e5-4ada-9e42-ea705c3eb1f3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b9592e3-41e5-4ada-9e42-ea705c3eb1f3-credential-socket\") pod \"0b9592e3-41e5-4ada-9e42-ea705c3eb1f3\" (UID: \"0b9592e3-41e5-4ada-9e42-ea705c3eb1f3\") " Oct 31 00:47:17.407491 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:17.407211 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21900006-51c4-44ab-848d-68f885effa5f-istiod-ca-cert" pod "21900006-51c4-44ab-848d-68f885effa5f" (UID: "21900006-51c4-44ab-848d-68f885effa5f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:47:17.407491 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:17.407434 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b9592e3-41e5-4ada-9e42-ea705c3eb1f3-istiod-ca-cert" pod "0b9592e3-41e5-4ada-9e42-ea705c3eb1f3" (UID: "0b9592e3-41e5-4ada-9e42-ea705c3eb1f3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:47:17.407606 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:17.407484 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21900006-51c4-44ab-848d-68f885effa5f-workload-certs" pod "21900006-51c4-44ab-848d-68f885effa5f" (UID: "21900006-51c4-44ab-848d-68f885effa5f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:47:17.407894 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:17.407860 5222 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b9592e3-41e5-4ada-9e42-ea705c3eb1f3-workload-socket\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:17.407894 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:17.407888 5222 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21900006-51c4-44ab-848d-68f885effa5f-credential-socket\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:17.408212 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:17.407904 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbvqj\" (UniqueName: \"kubernetes.io/projected/33d0dda2-34fd-409e-9165-5861f2373db9-kube-api-access-vbvqj\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:17.408212 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:17.407917 5222 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33d0dda2-34fd-409e-9165-5861f2373db9-credential-socket\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:17.408212 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:17.407930 5222 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b9592e3-41e5-4ada-9e42-ea705c3eb1f3-istiod-ca-cert\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:17.408212 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:17.407947 5222 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b9592e3-41e5-4ada-9e42-ea705c3eb1f3-workload-certs\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:17.408212 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:17.407961 5222 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b9592e3-41e5-4ada-9e42-ea705c3eb1f3-istio-data\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:17.408212 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:17.407976 5222 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33d0dda2-34fd-409e-9165-5861f2373db9-istio-podinfo\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:17.408212 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:17.407989 5222 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33d0dda2-34fd-409e-9165-5861f2373db9-istio-envoy\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:17.408212 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:17.408004 5222 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21900006-51c4-44ab-848d-68f885effa5f-workload-certs\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:17.408212 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:17.408021 5222 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21900006-51c4-44ab-848d-68f885effa5f-istiod-ca-cert\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:17.408212 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:17.408132 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21900006-51c4-44ab-848d-68f885effa5f-workload-socket" pod "21900006-51c4-44ab-848d-68f885effa5f" (UID: "21900006-51c4-44ab-848d-68f885effa5f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:47:17.408212 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:17.408034 5222 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33d0dda2-34fd-409e-9165-5861f2373db9-istio-token\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:17.408880 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:17.408275 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21900006-51c4-44ab-848d-68f885effa5f-istio-data" pod "21900006-51c4-44ab-848d-68f885effa5f" (UID: "21900006-51c4-44ab-848d-68f885effa5f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:47:17.408880 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:17.408458 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b9592e3-41e5-4ada-9e42-ea705c3eb1f3-credential-socket" pod "0b9592e3-41e5-4ada-9e42-ea705c3eb1f3" (UID: "0b9592e3-41e5-4ada-9e42-ea705c3eb1f3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:47:17.409254 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:17.409216 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21900006-51c4-44ab-848d-68f885effa5f-istio-envoy" pod "21900006-51c4-44ab-848d-68f885effa5f" (UID: "21900006-51c4-44ab-848d-68f885effa5f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:47:17.409396 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:17.409268 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21900006-51c4-44ab-848d-68f885effa5f-kube-api-access-m9mnp" pod "21900006-51c4-44ab-848d-68f885effa5f" (UID: "21900006-51c4-44ab-848d-68f885effa5f"). InnerVolumeSpecName "kube-api-access-m9mnp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:47:17.409396 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:17.409355 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21900006-51c4-44ab-848d-68f885effa5f-istio-token" pod "21900006-51c4-44ab-848d-68f885effa5f" (UID: "21900006-51c4-44ab-848d-68f885effa5f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:47:17.409767 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:17.409743 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b9592e3-41e5-4ada-9e42-ea705c3eb1f3-istio-token" pod "0b9592e3-41e5-4ada-9e42-ea705c3eb1f3" (UID: "0b9592e3-41e5-4ada-9e42-ea705c3eb1f3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:47:17.410110 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:17.410095 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/21900006-51c4-44ab-848d-68f885effa5f-istio-podinfo" pod "21900006-51c4-44ab-848d-68f885effa5f" (UID: "21900006-51c4-44ab-848d-68f885effa5f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:47:17.410256 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:17.410235 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b9592e3-41e5-4ada-9e42-ea705c3eb1f3-istio-envoy" pod "0b9592e3-41e5-4ada-9e42-ea705c3eb1f3" (UID: "0b9592e3-41e5-4ada-9e42-ea705c3eb1f3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:47:17.410834 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:17.410819 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0b9592e3-41e5-4ada-9e42-ea705c3eb1f3-istio-podinfo" pod "0b9592e3-41e5-4ada-9e42-ea705c3eb1f3" (UID: "0b9592e3-41e5-4ada-9e42-ea705c3eb1f3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:47:17.410919 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:17.410901 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b9592e3-41e5-4ada-9e42-ea705c3eb1f3-kube-api-access-qbtc5" pod "0b9592e3-41e5-4ada-9e42-ea705c3eb1f3" (UID: "0b9592e3-41e5-4ada-9e42-ea705c3eb1f3"). InnerVolumeSpecName "kube-api-access-qbtc5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:47:17.509575 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:17.509531 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbtc5\" (UniqueName: \"kubernetes.io/projected/0b9592e3-41e5-4ada-9e42-ea705c3eb1f3-kube-api-access-qbtc5\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:17.509575 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:17.509564 5222 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b9592e3-41e5-4ada-9e42-ea705c3eb1f3-istio-envoy\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:17.509575 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:17.509576 5222 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b9592e3-41e5-4ada-9e42-ea705c3eb1f3-istio-podinfo\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:17.509575 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:17.509585 5222 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21900006-51c4-44ab-848d-68f885effa5f-istio-podinfo\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:17.509957 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:17.509594 5222 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b9592e3-41e5-4ada-9e42-ea705c3eb1f3-credential-socket\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:17.509957 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:17.509602 5222 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b9592e3-41e5-4ada-9e42-ea705c3eb1f3-istio-token\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:17.509957 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:17.509611 5222 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21900006-51c4-44ab-848d-68f885effa5f-istio-token\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:17.509957 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:17.509619 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9mnp\" (UniqueName: \"kubernetes.io/projected/21900006-51c4-44ab-848d-68f885effa5f-kube-api-access-m9mnp\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:17.509957 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:17.509628 5222 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21900006-51c4-44ab-848d-68f885effa5f-istio-envoy\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:17.509957 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:17.509636 5222 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21900006-51c4-44ab-848d-68f885effa5f-workload-socket\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:17.509957 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:17.509644 5222 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21900006-51c4-44ab-848d-68f885effa5f-istio-data\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:18.014706 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:18.014612 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lkfxl" event={"ID":"21900006-51c4-44ab-848d-68f885effa5f","Type":"ContainerDied","Data":"7bb498b4c1301febf43dcef03cca8d286bdbd13629d96c9765be5d5e9a87fecc"} Oct 31 00:47:18.014706 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:18.014633 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lkfxl" Oct 31 00:47:18.014706 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:18.014694 5222 scope.go:122] "RemoveContainer" containerID="b462c5e3696412164e49cd679a7dec56a51ec00d8f2db0ac8e72f6a41edb9bb5" Oct 31 00:47:18.016359 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:18.016335 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-r57tc" Oct 31 00:47:18.016512 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:18.016336 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-r57tc" event={"ID":"33d0dda2-34fd-409e-9165-5861f2373db9","Type":"ContainerDied","Data":"2de1c8c1d775aa9edb0cba202523dfd8957aafd806c342e499ffe222d108f748"} Oct 31 00:47:18.018098 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:18.018075 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mjxmq" Oct 31 00:47:18.018206 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:18.018133 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mjxmq" event={"ID":"0b9592e3-41e5-4ada-9e42-ea705c3eb1f3","Type":"ContainerDied","Data":"9af5e3d2c7c8f2cacab9eba6dada9ae783638bb7da85a56184293ec0d6c30ecb"} Oct 31 00:47:18.027229 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:18.027208 5222 scope.go:122] "RemoveContainer" containerID="daba08f2af2f279c5bc195222337caad8b9790336f58e37cdfd843d3b391ca69" Oct 31 00:47:18.062509 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:18.062446 5222 scope.go:122] "RemoveContainer" containerID="a1e2c5bf2768aaf6e70d1488262ba6fc79c1d52e426ab722ba9126e949dcd73d" Oct 31 00:47:18.188301 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:18.188266 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lkfxl"] Oct 31 00:47:18.269396 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:18.268491 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lkfxl"] Oct 31 00:47:18.370393 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:18.370321 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-r57tc"] Oct 31 00:47:18.373898 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:18.373869 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-r57tc"] Oct 31 00:47:18.477903 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:18.477868 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mjxmq"] Oct 31 00:47:18.481541 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:18.481517 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mjxmq"] Oct 31 00:47:19.672395 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:19.672346 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b9592e3-41e5-4ada-9e42-ea705c3eb1f3" path="/var/lib/kubelet/pods/0b9592e3-41e5-4ada-9e42-ea705c3eb1f3/volumes" Oct 31 00:47:19.673107 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:19.673082 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21900006-51c4-44ab-848d-68f885effa5f" path="/var/lib/kubelet/pods/21900006-51c4-44ab-848d-68f885effa5f/volumes" Oct 31 00:47:19.673617 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:19.673597 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33d0dda2-34fd-409e-9165-5861f2373db9" path="/var/lib/kubelet/pods/33d0dda2-34fd-409e-9165-5861f2373db9/volumes" Oct 31 00:47:27.498666 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:27.498621 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-50a19d-openshift-default-68c6d7d74c-24fwv"] Oct 31 00:47:27.499886 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:27.499838 5222 cpu_manager.go:414] "containerMap: removing container" podUID="0b9592e3-41e5-4ada-9e42-ea705c3eb1f3" containerName="istio-proxy" Oct 31 00:47:27.500084 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:27.500064 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b9592e3-41e5-4ada-9e42-ea705c3eb1f3" containerName="istio-proxy" Oct 31 00:47:27.500255 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:27.500243 5222 cpu_manager.go:414] "containerMap: removing container" podUID="33d0dda2-34fd-409e-9165-5861f2373db9" containerName="istio-proxy" Oct 31 00:47:27.500341 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:27.500329 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33d0dda2-34fd-409e-9165-5861f2373db9" containerName="istio-proxy" Oct 31 00:47:27.500465 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:27.500453 5222 cpu_manager.go:414] "containerMap: removing container" podUID="10937a00-0cff-4b28-9a52-6cef09efe247" containerName="revoke-keys" Oct 31 00:47:27.500554 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:27.500542 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10937a00-0cff-4b28-9a52-6cef09efe247" containerName="revoke-keys" Oct 31 00:47:27.500679 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:27.500664 5222 cpu_manager.go:414] "containerMap: removing container" podUID="21900006-51c4-44ab-848d-68f885effa5f" containerName="istio-proxy" Oct 31 00:47:27.500851 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:27.500836 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21900006-51c4-44ab-848d-68f885effa5f" containerName="istio-proxy" Oct 31 00:47:27.501162 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:27.501139 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b9592e3-41e5-4ada-9e42-ea705c3eb1f3" containerName="istio-proxy" Oct 31 00:47:27.501288 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:27.501276 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="33d0dda2-34fd-409e-9165-5861f2373db9" containerName="istio-proxy" Oct 31 00:47:27.501405 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:27.501393 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="10937a00-0cff-4b28-9a52-6cef09efe247" containerName="revoke-keys" Oct 31 00:47:27.501506 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:27.501496 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="21900006-51c4-44ab-848d-68f885effa5f" containerName="istio-proxy" Oct 31 00:47:27.512092 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:27.509474 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-50a19d-openshift-default-68c6d7d74c-24fwv" Oct 31 00:47:27.569617 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:27.568757 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-50a19d-openshift-default-dockercfg-4l8q9\"" Oct 31 00:47:27.610509 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:27.610474 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f2eb643-034b-4b9a-bd2f-6ddb034008b8-istio-data\") pod \"e2e-api-b-50a19d-openshift-default-68c6d7d74c-24fwv\" (UID: \"1f2eb643-034b-4b9a-bd2f-6ddb034008b8\") " pod="openshift-ingress/e2e-api-b-50a19d-openshift-default-68c6d7d74c-24fwv" Oct 31 00:47:27.610697 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:27.610523 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vx7tp\" (UniqueName: \"kubernetes.io/projected/1f2eb643-034b-4b9a-bd2f-6ddb034008b8-kube-api-access-vx7tp\") pod \"e2e-api-b-50a19d-openshift-default-68c6d7d74c-24fwv\" (UID: \"1f2eb643-034b-4b9a-bd2f-6ddb034008b8\") " pod="openshift-ingress/e2e-api-b-50a19d-openshift-default-68c6d7d74c-24fwv" Oct 31 00:47:27.610697 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:27.610615 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f2eb643-034b-4b9a-bd2f-6ddb034008b8-workload-socket\") pod \"e2e-api-b-50a19d-openshift-default-68c6d7d74c-24fwv\" (UID: \"1f2eb643-034b-4b9a-bd2f-6ddb034008b8\") " pod="openshift-ingress/e2e-api-b-50a19d-openshift-default-68c6d7d74c-24fwv" Oct 31 00:47:27.610837 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:27.610731 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f2eb643-034b-4b9a-bd2f-6ddb034008b8-istio-podinfo\") pod \"e2e-api-b-50a19d-openshift-default-68c6d7d74c-24fwv\" (UID: \"1f2eb643-034b-4b9a-bd2f-6ddb034008b8\") " pod="openshift-ingress/e2e-api-b-50a19d-openshift-default-68c6d7d74c-24fwv" Oct 31 00:47:27.610837 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:27.610807 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f2eb643-034b-4b9a-bd2f-6ddb034008b8-workload-certs\") pod \"e2e-api-b-50a19d-openshift-default-68c6d7d74c-24fwv\" (UID: \"1f2eb643-034b-4b9a-bd2f-6ddb034008b8\") " pod="openshift-ingress/e2e-api-b-50a19d-openshift-default-68c6d7d74c-24fwv" Oct 31 00:47:27.610954 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:27.610888 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f2eb643-034b-4b9a-bd2f-6ddb034008b8-credential-socket\") pod \"e2e-api-b-50a19d-openshift-default-68c6d7d74c-24fwv\" (UID: \"1f2eb643-034b-4b9a-bd2f-6ddb034008b8\") " pod="openshift-ingress/e2e-api-b-50a19d-openshift-default-68c6d7d74c-24fwv" Oct 31 00:47:27.611013 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:27.610973 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f2eb643-034b-4b9a-bd2f-6ddb034008b8-istio-token\") pod \"e2e-api-b-50a19d-openshift-default-68c6d7d74c-24fwv\" (UID: \"1f2eb643-034b-4b9a-bd2f-6ddb034008b8\") " pod="openshift-ingress/e2e-api-b-50a19d-openshift-default-68c6d7d74c-24fwv" Oct 31 00:47:27.611075 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:27.611019 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f2eb643-034b-4b9a-bd2f-6ddb034008b8-istiod-ca-cert\") pod \"e2e-api-b-50a19d-openshift-default-68c6d7d74c-24fwv\" (UID: \"1f2eb643-034b-4b9a-bd2f-6ddb034008b8\") " pod="openshift-ingress/e2e-api-b-50a19d-openshift-default-68c6d7d74c-24fwv" Oct 31 00:47:27.611132 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:27.611100 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f2eb643-034b-4b9a-bd2f-6ddb034008b8-istio-envoy\") pod \"e2e-api-b-50a19d-openshift-default-68c6d7d74c-24fwv\" (UID: \"1f2eb643-034b-4b9a-bd2f-6ddb034008b8\") " pod="openshift-ingress/e2e-api-b-50a19d-openshift-default-68c6d7d74c-24fwv" Oct 31 00:47:27.623771 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:27.623741 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-50a19d-openshift-default-68c6d7d74c-24fwv"] Oct 31 00:47:27.715725 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:27.712499 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f2eb643-034b-4b9a-bd2f-6ddb034008b8-istio-podinfo\") pod \"e2e-api-b-50a19d-openshift-default-68c6d7d74c-24fwv\" (UID: \"1f2eb643-034b-4b9a-bd2f-6ddb034008b8\") " pod="openshift-ingress/e2e-api-b-50a19d-openshift-default-68c6d7d74c-24fwv" Oct 31 00:47:27.715725 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:27.712616 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f2eb643-034b-4b9a-bd2f-6ddb034008b8-workload-certs\") pod \"e2e-api-b-50a19d-openshift-default-68c6d7d74c-24fwv\" (UID: \"1f2eb643-034b-4b9a-bd2f-6ddb034008b8\") " pod="openshift-ingress/e2e-api-b-50a19d-openshift-default-68c6d7d74c-24fwv" Oct 31 00:47:27.715725 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:27.712665 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f2eb643-034b-4b9a-bd2f-6ddb034008b8-credential-socket\") pod \"e2e-api-b-50a19d-openshift-default-68c6d7d74c-24fwv\" (UID: \"1f2eb643-034b-4b9a-bd2f-6ddb034008b8\") " pod="openshift-ingress/e2e-api-b-50a19d-openshift-default-68c6d7d74c-24fwv" Oct 31 00:47:27.715725 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:27.712734 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f2eb643-034b-4b9a-bd2f-6ddb034008b8-istio-token\") pod \"e2e-api-b-50a19d-openshift-default-68c6d7d74c-24fwv\" (UID: \"1f2eb643-034b-4b9a-bd2f-6ddb034008b8\") " pod="openshift-ingress/e2e-api-b-50a19d-openshift-default-68c6d7d74c-24fwv" Oct 31 00:47:27.715725 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:27.712776 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f2eb643-034b-4b9a-bd2f-6ddb034008b8-istiod-ca-cert\") pod \"e2e-api-b-50a19d-openshift-default-68c6d7d74c-24fwv\" (UID: \"1f2eb643-034b-4b9a-bd2f-6ddb034008b8\") " pod="openshift-ingress/e2e-api-b-50a19d-openshift-default-68c6d7d74c-24fwv" Oct 31 00:47:27.715725 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:27.712822 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f2eb643-034b-4b9a-bd2f-6ddb034008b8-istio-envoy\") pod \"e2e-api-b-50a19d-openshift-default-68c6d7d74c-24fwv\" (UID: \"1f2eb643-034b-4b9a-bd2f-6ddb034008b8\") " pod="openshift-ingress/e2e-api-b-50a19d-openshift-default-68c6d7d74c-24fwv" Oct 31 00:47:27.715725 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:27.712940 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f2eb643-034b-4b9a-bd2f-6ddb034008b8-istio-data\") pod \"e2e-api-b-50a19d-openshift-default-68c6d7d74c-24fwv\" (UID: \"1f2eb643-034b-4b9a-bd2f-6ddb034008b8\") " pod="openshift-ingress/e2e-api-b-50a19d-openshift-default-68c6d7d74c-24fwv" Oct 31 00:47:27.715725 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:27.712985 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vx7tp\" (UniqueName: \"kubernetes.io/projected/1f2eb643-034b-4b9a-bd2f-6ddb034008b8-kube-api-access-vx7tp\") pod \"e2e-api-b-50a19d-openshift-default-68c6d7d74c-24fwv\" (UID: \"1f2eb643-034b-4b9a-bd2f-6ddb034008b8\") " pod="openshift-ingress/e2e-api-b-50a19d-openshift-default-68c6d7d74c-24fwv" Oct 31 00:47:27.715725 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:27.713035 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f2eb643-034b-4b9a-bd2f-6ddb034008b8-workload-socket\") pod \"e2e-api-b-50a19d-openshift-default-68c6d7d74c-24fwv\" (UID: \"1f2eb643-034b-4b9a-bd2f-6ddb034008b8\") " pod="openshift-ingress/e2e-api-b-50a19d-openshift-default-68c6d7d74c-24fwv" Oct 31 00:47:27.715725 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:27.713565 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f2eb643-034b-4b9a-bd2f-6ddb034008b8-workload-socket\") pod \"e2e-api-b-50a19d-openshift-default-68c6d7d74c-24fwv\" (UID: \"1f2eb643-034b-4b9a-bd2f-6ddb034008b8\") " pod="openshift-ingress/e2e-api-b-50a19d-openshift-default-68c6d7d74c-24fwv" Oct 31 00:47:27.731960 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:27.729868 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f2eb643-034b-4b9a-bd2f-6ddb034008b8-istio-data\") pod \"e2e-api-b-50a19d-openshift-default-68c6d7d74c-24fwv\" (UID: \"1f2eb643-034b-4b9a-bd2f-6ddb034008b8\") " pod="openshift-ingress/e2e-api-b-50a19d-openshift-default-68c6d7d74c-24fwv" Oct 31 00:47:27.731960 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:27.730904 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f2eb643-034b-4b9a-bd2f-6ddb034008b8-istio-podinfo\") pod \"e2e-api-b-50a19d-openshift-default-68c6d7d74c-24fwv\" (UID: \"1f2eb643-034b-4b9a-bd2f-6ddb034008b8\") " pod="openshift-ingress/e2e-api-b-50a19d-openshift-default-68c6d7d74c-24fwv" Oct 31 00:47:27.731960 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:27.731488 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f2eb643-034b-4b9a-bd2f-6ddb034008b8-credential-socket\") pod \"e2e-api-b-50a19d-openshift-default-68c6d7d74c-24fwv\" (UID: \"1f2eb643-034b-4b9a-bd2f-6ddb034008b8\") " pod="openshift-ingress/e2e-api-b-50a19d-openshift-default-68c6d7d74c-24fwv" Oct 31 00:47:27.731960 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:27.731737 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f2eb643-034b-4b9a-bd2f-6ddb034008b8-workload-certs\") pod \"e2e-api-b-50a19d-openshift-default-68c6d7d74c-24fwv\" (UID: \"1f2eb643-034b-4b9a-bd2f-6ddb034008b8\") " pod="openshift-ingress/e2e-api-b-50a19d-openshift-default-68c6d7d74c-24fwv" Oct 31 00:47:27.732390 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:27.732332 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f2eb643-034b-4b9a-bd2f-6ddb034008b8-istio-envoy\") pod \"e2e-api-b-50a19d-openshift-default-68c6d7d74c-24fwv\" (UID: \"1f2eb643-034b-4b9a-bd2f-6ddb034008b8\") " pod="openshift-ingress/e2e-api-b-50a19d-openshift-default-68c6d7d74c-24fwv" Oct 31 00:47:27.732759 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:27.732734 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f2eb643-034b-4b9a-bd2f-6ddb034008b8-istiod-ca-cert\") pod \"e2e-api-b-50a19d-openshift-default-68c6d7d74c-24fwv\" (UID: \"1f2eb643-034b-4b9a-bd2f-6ddb034008b8\") " pod="openshift-ingress/e2e-api-b-50a19d-openshift-default-68c6d7d74c-24fwv" Oct 31 00:47:27.794777 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:27.794746 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vx7tp\" (UniqueName: \"kubernetes.io/projected/1f2eb643-034b-4b9a-bd2f-6ddb034008b8-kube-api-access-vx7tp\") pod \"e2e-api-b-50a19d-openshift-default-68c6d7d74c-24fwv\" (UID: \"1f2eb643-034b-4b9a-bd2f-6ddb034008b8\") " pod="openshift-ingress/e2e-api-b-50a19d-openshift-default-68c6d7d74c-24fwv" Oct 31 00:47:27.803433 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:27.795546 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f2eb643-034b-4b9a-bd2f-6ddb034008b8-istio-token\") pod \"e2e-api-b-50a19d-openshift-default-68c6d7d74c-24fwv\" (UID: \"1f2eb643-034b-4b9a-bd2f-6ddb034008b8\") " pod="openshift-ingress/e2e-api-b-50a19d-openshift-default-68c6d7d74c-24fwv" Oct 31 00:47:27.833737 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:27.833697 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-50a19d-openshift-default-68c6d7d74c-24fwv" Oct 31 00:47:28.120431 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:28.120285 5222 generic.go:356] "Generic (PLEG): container finished" podID="d0694975-c551-4a8a-b296-eaee47077b0c" containerID="ce3d7e380246894abac01542c0fdcd9b01ae85322ca675c9d5d7ac62e7e8f8ce" exitCode=0 Oct 31 00:47:28.120431 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:28.120360 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-dcdbd0-openshift-default-86cd956cc9-p8h4z" event={"ID":"d0694975-c551-4a8a-b296-eaee47077b0c","Type":"ContainerDied","Data":"ce3d7e380246894abac01542c0fdcd9b01ae85322ca675c9d5d7ac62e7e8f8ce"} Oct 31 00:47:28.120940 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:28.120908 5222 scope.go:122] "RemoveContainer" containerID="ce3d7e380246894abac01542c0fdcd9b01ae85322ca675c9d5d7ac62e7e8f8ce" Oct 31 00:47:28.122933 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:28.122902 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:47:28.123036 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:28.122996 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:47:28.123036 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:28.123029 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:47:28.410194 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:47:28.410159 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f2eb643_034b_4b9a_bd2f_6ddb034008b8.slice/crio-08ac91432afffe59486d44a26612b12c335cf6faf89eef4c1b069877f259c3f8 WatchSource:0}: Error finding container 08ac91432afffe59486d44a26612b12c335cf6faf89eef4c1b069877f259c3f8: Status 404 returned error can't find the container with id 08ac91432afffe59486d44a26612b12c335cf6faf89eef4c1b069877f259c3f8 Oct 31 00:47:28.412872 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:28.412848 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-50a19d-openshift-default-68c6d7d74c-24fwv"] Oct 31 00:47:28.413309 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:28.413279 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:47:28.413352 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:28.413344 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:47:28.413433 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:28.413394 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:47:28.788119 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:28.788082 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-c5pmx"] Oct 31 00:47:28.804250 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:28.804215 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-c5pmx" Oct 31 00:47:28.837445 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:28.834916 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-c5pmx"] Oct 31 00:47:28.895270 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:28.894098 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35e93965-7511-4e3b-ae8e-af51574f79f1-istio-token\") pod \"e2e-api-a-5b4db8-openshift-default-5cc6998bbc-c5pmx\" (UID: \"35e93965-7511-4e3b-ae8e-af51574f79f1\") " pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-c5pmx" Oct 31 00:47:28.895270 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:28.894318 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mz5tj\" (UniqueName: \"kubernetes.io/projected/35e93965-7511-4e3b-ae8e-af51574f79f1-kube-api-access-mz5tj\") pod \"e2e-api-a-5b4db8-openshift-default-5cc6998bbc-c5pmx\" (UID: \"35e93965-7511-4e3b-ae8e-af51574f79f1\") " pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-c5pmx" Oct 31 00:47:28.895270 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:28.894423 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35e93965-7511-4e3b-ae8e-af51574f79f1-istio-envoy\") pod \"e2e-api-a-5b4db8-openshift-default-5cc6998bbc-c5pmx\" (UID: \"35e93965-7511-4e3b-ae8e-af51574f79f1\") " pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-c5pmx" Oct 31 00:47:28.895270 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:28.894512 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35e93965-7511-4e3b-ae8e-af51574f79f1-istio-data\") pod \"e2e-api-a-5b4db8-openshift-default-5cc6998bbc-c5pmx\" (UID: \"35e93965-7511-4e3b-ae8e-af51574f79f1\") " pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-c5pmx" Oct 31 00:47:28.895270 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:28.894596 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35e93965-7511-4e3b-ae8e-af51574f79f1-workload-socket\") pod \"e2e-api-a-5b4db8-openshift-default-5cc6998bbc-c5pmx\" (UID: \"35e93965-7511-4e3b-ae8e-af51574f79f1\") " pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-c5pmx" Oct 31 00:47:28.895270 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:28.894755 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35e93965-7511-4e3b-ae8e-af51574f79f1-istio-podinfo\") pod \"e2e-api-a-5b4db8-openshift-default-5cc6998bbc-c5pmx\" (UID: \"35e93965-7511-4e3b-ae8e-af51574f79f1\") " pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-c5pmx" Oct 31 00:47:28.895270 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:28.894857 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35e93965-7511-4e3b-ae8e-af51574f79f1-workload-certs\") pod \"e2e-api-a-5b4db8-openshift-default-5cc6998bbc-c5pmx\" (UID: \"35e93965-7511-4e3b-ae8e-af51574f79f1\") " pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-c5pmx" Oct 31 00:47:28.895270 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:28.894905 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35e93965-7511-4e3b-ae8e-af51574f79f1-istiod-ca-cert\") pod \"e2e-api-a-5b4db8-openshift-default-5cc6998bbc-c5pmx\" (UID: \"35e93965-7511-4e3b-ae8e-af51574f79f1\") " pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-c5pmx" Oct 31 00:47:28.895270 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:28.894922 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35e93965-7511-4e3b-ae8e-af51574f79f1-credential-socket\") pod \"e2e-api-a-5b4db8-openshift-default-5cc6998bbc-c5pmx\" (UID: \"35e93965-7511-4e3b-ae8e-af51574f79f1\") " pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-c5pmx" Oct 31 00:47:29.008459 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:29.007525 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mz5tj\" (UniqueName: \"kubernetes.io/projected/35e93965-7511-4e3b-ae8e-af51574f79f1-kube-api-access-mz5tj\") pod \"e2e-api-a-5b4db8-openshift-default-5cc6998bbc-c5pmx\" (UID: \"35e93965-7511-4e3b-ae8e-af51574f79f1\") " pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-c5pmx" Oct 31 00:47:29.008459 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:29.007588 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35e93965-7511-4e3b-ae8e-af51574f79f1-istio-envoy\") pod \"e2e-api-a-5b4db8-openshift-default-5cc6998bbc-c5pmx\" (UID: \"35e93965-7511-4e3b-ae8e-af51574f79f1\") " pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-c5pmx" Oct 31 00:47:29.008459 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:29.007653 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35e93965-7511-4e3b-ae8e-af51574f79f1-istio-data\") pod \"e2e-api-a-5b4db8-openshift-default-5cc6998bbc-c5pmx\" (UID: \"35e93965-7511-4e3b-ae8e-af51574f79f1\") " pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-c5pmx" Oct 31 00:47:29.008459 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:29.007680 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35e93965-7511-4e3b-ae8e-af51574f79f1-workload-socket\") pod \"e2e-api-a-5b4db8-openshift-default-5cc6998bbc-c5pmx\" (UID: \"35e93965-7511-4e3b-ae8e-af51574f79f1\") " pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-c5pmx" Oct 31 00:47:29.008459 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:29.007839 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35e93965-7511-4e3b-ae8e-af51574f79f1-istio-podinfo\") pod \"e2e-api-a-5b4db8-openshift-default-5cc6998bbc-c5pmx\" (UID: \"35e93965-7511-4e3b-ae8e-af51574f79f1\") " pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-c5pmx" Oct 31 00:47:29.008459 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:29.007996 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35e93965-7511-4e3b-ae8e-af51574f79f1-workload-certs\") pod \"e2e-api-a-5b4db8-openshift-default-5cc6998bbc-c5pmx\" (UID: \"35e93965-7511-4e3b-ae8e-af51574f79f1\") " pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-c5pmx" Oct 31 00:47:29.008459 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:29.008025 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35e93965-7511-4e3b-ae8e-af51574f79f1-istiod-ca-cert\") pod \"e2e-api-a-5b4db8-openshift-default-5cc6998bbc-c5pmx\" (UID: \"35e93965-7511-4e3b-ae8e-af51574f79f1\") " pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-c5pmx" Oct 31 00:47:29.008459 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:29.008048 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35e93965-7511-4e3b-ae8e-af51574f79f1-credential-socket\") pod \"e2e-api-a-5b4db8-openshift-default-5cc6998bbc-c5pmx\" (UID: \"35e93965-7511-4e3b-ae8e-af51574f79f1\") " pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-c5pmx" Oct 31 00:47:29.008459 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:29.008126 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35e93965-7511-4e3b-ae8e-af51574f79f1-istio-token\") pod \"e2e-api-a-5b4db8-openshift-default-5cc6998bbc-c5pmx\" (UID: \"35e93965-7511-4e3b-ae8e-af51574f79f1\") " pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-c5pmx" Oct 31 00:47:29.009058 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:29.008696 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35e93965-7511-4e3b-ae8e-af51574f79f1-workload-socket\") pod \"e2e-api-a-5b4db8-openshift-default-5cc6998bbc-c5pmx\" (UID: \"35e93965-7511-4e3b-ae8e-af51574f79f1\") " pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-c5pmx" Oct 31 00:47:29.009489 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:29.009414 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35e93965-7511-4e3b-ae8e-af51574f79f1-istio-data\") pod \"e2e-api-a-5b4db8-openshift-default-5cc6998bbc-c5pmx\" (UID: \"35e93965-7511-4e3b-ae8e-af51574f79f1\") " pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-c5pmx" Oct 31 00:47:29.009711 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:29.009689 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35e93965-7511-4e3b-ae8e-af51574f79f1-istiod-ca-cert\") pod \"e2e-api-a-5b4db8-openshift-default-5cc6998bbc-c5pmx\" (UID: \"35e93965-7511-4e3b-ae8e-af51574f79f1\") " pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-c5pmx" Oct 31 00:47:29.009800 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:29.009736 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35e93965-7511-4e3b-ae8e-af51574f79f1-workload-certs\") pod \"e2e-api-a-5b4db8-openshift-default-5cc6998bbc-c5pmx\" (UID: \"35e93965-7511-4e3b-ae8e-af51574f79f1\") " pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-c5pmx" Oct 31 00:47:29.010997 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:29.009994 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35e93965-7511-4e3b-ae8e-af51574f79f1-credential-socket\") pod \"e2e-api-a-5b4db8-openshift-default-5cc6998bbc-c5pmx\" (UID: \"35e93965-7511-4e3b-ae8e-af51574f79f1\") " pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-c5pmx" Oct 31 00:47:29.013596 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:29.013545 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35e93965-7511-4e3b-ae8e-af51574f79f1-istio-podinfo\") pod \"e2e-api-a-5b4db8-openshift-default-5cc6998bbc-c5pmx\" (UID: \"35e93965-7511-4e3b-ae8e-af51574f79f1\") " pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-c5pmx" Oct 31 00:47:29.015891 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:29.015865 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35e93965-7511-4e3b-ae8e-af51574f79f1-istio-envoy\") pod \"e2e-api-a-5b4db8-openshift-default-5cc6998bbc-c5pmx\" (UID: \"35e93965-7511-4e3b-ae8e-af51574f79f1\") " pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-c5pmx" Oct 31 00:47:29.071462 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:29.071429 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mz5tj\" (UniqueName: \"kubernetes.io/projected/35e93965-7511-4e3b-ae8e-af51574f79f1-kube-api-access-mz5tj\") pod \"e2e-api-a-5b4db8-openshift-default-5cc6998bbc-c5pmx\" (UID: \"35e93965-7511-4e3b-ae8e-af51574f79f1\") " pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-c5pmx" Oct 31 00:47:29.071623 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:29.071434 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35e93965-7511-4e3b-ae8e-af51574f79f1-istio-token\") pod \"e2e-api-a-5b4db8-openshift-default-5cc6998bbc-c5pmx\" (UID: \"35e93965-7511-4e3b-ae8e-af51574f79f1\") " pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-c5pmx" Oct 31 00:47:29.145396 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:29.145252 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-c5pmx" Oct 31 00:47:29.207526 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:29.207486 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-dcdbd0-openshift-default-86cd956cc9-p8h4z" event={"ID":"d0694975-c551-4a8a-b296-eaee47077b0c","Type":"ContainerStarted","Data":"e5dd597aea1c6554fa4de02548aad9b94a18448800059c5d26af7c4a84b57352"} Oct 31 00:47:29.213398 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:29.211803 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-50a19d-openshift-default-68c6d7d74c-24fwv" event={"ID":"1f2eb643-034b-4b9a-bd2f-6ddb034008b8","Type":"ContainerStarted","Data":"31c580e668900e489b539a14c43cd7cf2fc958b6bd1ddb1e866d31c62799d239"} Oct 31 00:47:29.213398 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:29.211861 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-50a19d-openshift-default-68c6d7d74c-24fwv" event={"ID":"1f2eb643-034b-4b9a-bd2f-6ddb034008b8","Type":"ContainerStarted","Data":"08ac91432afffe59486d44a26612b12c335cf6faf89eef4c1b069877f259c3f8"} Oct 31 00:47:29.497265 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:29.497206 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-50a19d-openshift-default-68c6d7d74c-24fwv" podStartSLOduration=2.49718947 podStartE2EDuration="2.49718947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:47:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:47:29.490277742 +0000 UTC m=+1941.961447508" watchObservedRunningTime="2026-10-31 00:47:29.49718947 +0000 UTC m=+1941.968359215" Oct 31 00:47:29.703082 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:47:29.703043 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35e93965_7511_4e3b_ae8e_af51574f79f1.slice/crio-55e50c731ed697309ccc5dd2d8baeaa60e4d4f0de83e0c3af2d77985f1c452af WatchSource:0}: Error finding container 55e50c731ed697309ccc5dd2d8baeaa60e4d4f0de83e0c3af2d77985f1c452af: Status 404 returned error can't find the container with id 55e50c731ed697309ccc5dd2d8baeaa60e4d4f0de83e0c3af2d77985f1c452af Oct 31 00:47:29.712945 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:29.712736 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-c5pmx"] Oct 31 00:47:29.712945 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:29.712837 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:47:29.712945 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:29.712936 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:47:29.713177 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:29.712978 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:47:29.834077 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:29.834037 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-50a19d-openshift-default-68c6d7d74c-24fwv" Oct 31 00:47:29.845720 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:29.845670 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-50a19d-openshift-default-68c6d7d74c-24fwv" Oct 31 00:47:30.000651 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:30.000557 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-dcdbd0-openshift-default-86cd956cc9-p8h4z" Oct 31 00:47:30.001730 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:30.001694 5222 patch_prober.go:29] interesting pod/e2e-shared-b-w2-dcdbd0-openshift-default-86cd956cc9-p8h4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Oct 31 00:47:30.001870 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:30.001747 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-dcdbd0-openshift-default-86cd956cc9-p8h4z" podUID="d0694975-c551-4a8a-b296-eaee47077b0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Oct 31 00:47:30.222603 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:30.222566 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-c5pmx" event={"ID":"35e93965-7511-4e3b-ae8e-af51574f79f1","Type":"ContainerStarted","Data":"8b9b985c6a48a3040e67e9a999fa9f63a05981fbf411c49a8ad0d0cfd25d2e2b"} Oct 31 00:47:30.222807 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:30.222627 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-c5pmx" event={"ID":"35e93965-7511-4e3b-ae8e-af51574f79f1","Type":"ContainerStarted","Data":"55e50c731ed697309ccc5dd2d8baeaa60e4d4f0de83e0c3af2d77985f1c452af"} Oct 31 00:47:30.223266 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:30.223244 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-50a19d-openshift-default-68c6d7d74c-24fwv" Oct 31 00:47:30.224160 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:30.224142 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-50a19d-openshift-default-68c6d7d74c-24fwv" Oct 31 00:47:30.372153 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:30.372077 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-c5pmx" podStartSLOduration=2.372049752 podStartE2EDuration="2.372049752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:47:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:47:30.356745671 +0000 UTC m=+1942.827915421" watchObservedRunningTime="2026-10-31 00:47:30.372049752 +0000 UTC m=+1942.843219502" Oct 31 00:47:30.721591 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:30.721509 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-dcdbd0-openshift-default-86cd956cc9-p8h4z" Oct 31 00:47:30.999903 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:30.999813 5222 patch_prober.go:29] interesting pod/e2e-shared-b-w2-dcdbd0-openshift-default-86cd956cc9-p8h4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Oct 31 00:47:30.999903 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:30.999867 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-dcdbd0-openshift-default-86cd956cc9-p8h4z" podUID="d0694975-c551-4a8a-b296-eaee47077b0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Oct 31 00:47:31.157418 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:31.150485 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-c5pmx" Oct 31 00:47:31.157418 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:31.151021 5222 patch_prober.go:29] interesting pod/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-c5pmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body= Oct 31 00:47:31.157418 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:31.151067 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-c5pmx" podUID="35e93965-7511-4e3b-ae8e-af51574f79f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" Oct 31 00:47:31.832601 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:31.832549 5222 patch_prober.go:29] interesting pod/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-tf2dv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:47:31.832816 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:31.832615 5222 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-tf2dv" podUID="b191ce67-4983-446c-bd11-394b1f008034" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:32.002178 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:32.001235 5222 patch_prober.go:29] interesting pod/e2e-shared-b-w2-dcdbd0-openshift-default-86cd956cc9-p8h4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Oct 31 00:47:32.002178 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:32.001303 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-dcdbd0-openshift-default-86cd956cc9-p8h4z" podUID="d0694975-c551-4a8a-b296-eaee47077b0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Oct 31 00:47:32.151455 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:32.150833 5222 patch_prober.go:29] interesting pod/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-c5pmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body= Oct 31 00:47:32.151455 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:32.150905 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-c5pmx" podUID="35e93965-7511-4e3b-ae8e-af51574f79f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" Oct 31 00:47:33.000030 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:32.999989 5222 patch_prober.go:29] interesting pod/e2e-shared-b-w2-dcdbd0-openshift-default-86cd956cc9-p8h4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Oct 31 00:47:33.000200 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:33.000046 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-dcdbd0-openshift-default-86cd956cc9-p8h4z" podUID="d0694975-c551-4a8a-b296-eaee47077b0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Oct 31 00:47:33.158976 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:33.158937 5222 patch_prober.go:29] interesting pod/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-c5pmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body= Oct 31 00:47:33.168118 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:33.167659 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-c5pmx" podUID="35e93965-7511-4e3b-ae8e-af51574f79f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" Oct 31 00:47:34.000070 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:34.000033 5222 patch_prober.go:29] interesting pod/e2e-shared-b-w2-dcdbd0-openshift-default-86cd956cc9-p8h4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Oct 31 00:47:34.000250 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:34.000085 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-dcdbd0-openshift-default-86cd956cc9-p8h4z" podUID="d0694975-c551-4a8a-b296-eaee47077b0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Oct 31 00:47:34.163398 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:34.162528 5222 patch_prober.go:29] interesting pod/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-c5pmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body= Oct 31 00:47:34.163398 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:34.162612 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-c5pmx" podUID="35e93965-7511-4e3b-ae8e-af51574f79f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" Oct 31 00:47:35.006296 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:35.006261 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-dcdbd0-openshift-default-86cd956cc9-p8h4z" Oct 31 00:47:35.007461 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:35.007438 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-dcdbd0-openshift-default-86cd956cc9-p8h4z" Oct 31 00:47:35.157857 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:35.157821 5222 patch_prober.go:29] interesting pod/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-c5pmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body= Oct 31 00:47:35.158156 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:35.158127 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-c5pmx" podUID="35e93965-7511-4e3b-ae8e-af51574f79f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" Oct 31 00:47:35.893317 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:47:35.893234 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 00:47:36.146723 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:36.146688 5222 patch_prober.go:29] interesting pod/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-c5pmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body= Oct 31 00:47:36.146939 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:36.146756 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-c5pmx" podUID="35e93965-7511-4e3b-ae8e-af51574f79f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" Oct 31 00:47:36.315967 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:36.315932 5222 generic.go:356] "Generic (PLEG): container finished" podID="b191ce67-4983-446c-bd11-394b1f008034" containerID="be0e03299ea7d929c61339c68d60b33a9e7dc6dea816d5a994593d9ae2a5a510" exitCode=0 Oct 31 00:47:36.316156 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:36.316001 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-tf2dv" event={"ID":"b191ce67-4983-446c-bd11-394b1f008034","Type":"ContainerDied","Data":"be0e03299ea7d929c61339c68d60b33a9e7dc6dea816d5a994593d9ae2a5a510"} Oct 31 00:47:36.316519 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:36.316504 5222 scope.go:122] "RemoveContainer" containerID="be0e03299ea7d929c61339c68d60b33a9e7dc6dea816d5a994593d9ae2a5a510" Oct 31 00:47:36.318300 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:36.318271 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:47:36.318411 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:36.318341 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:47:36.318411 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:36.318405 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:47:37.151380 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:37.151345 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-c5pmx" Oct 31 00:47:37.151901 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:37.151791 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-c5pmx" Oct 31 00:47:37.152493 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:37.152472 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-c5pmx" Oct 31 00:47:37.328672 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:37.328637 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-tf2dv" event={"ID":"b191ce67-4983-446c-bd11-394b1f008034","Type":"ContainerStarted","Data":"6d55730e52ec751adbaa2c9af21dad117a8ec237252148846f89c78a720a0be9"} Oct 31 00:47:37.759854 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:37.759736 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-50a19d-9874b478c-mqhwf"] Oct 31 00:47:37.818244 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:37.798770 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-50a19d-9874b478c-mqhwf" Oct 31 00:47:37.818244 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:37.817915 5222 patch_prober.go:29] interesting pod/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-tf2dv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body= Oct 31 00:47:37.818244 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:37.817970 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-tf2dv" podUID="b191ce67-4983-446c-bd11-394b1f008034" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" Oct 31 00:47:37.854775 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:37.853894 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-50a19d\"" Oct 31 00:47:37.854775 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:37.854731 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-50a19d-dockercfg-62hvg\"" Oct 31 00:47:38.029788 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:38.022392 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-50a19d-9874b478c-mqhwf"] Oct 31 00:47:38.029788 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:38.022501 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-tf2dv" Oct 31 00:47:38.029788 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:38.022536 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-50a19d-867c4c57b5-xknkf"] Oct 31 00:47:38.048817 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:38.048785 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-50a19d-867c4c57b5-xknkf" Oct 31 00:47:38.075064 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:38.075033 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-50a19d-867c4c57b5-xknkf"] Oct 31 00:47:38.079687 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:38.079659 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04cf35a1-79f2-4fbd-8a84-7df54fbc0029-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-50a19d-9874b478c-mqhwf\" (UID: \"04cf35a1-79f2-4fbd-8a84-7df54fbc0029\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-50a19d-9874b478c-mqhwf" Oct 31 00:47:38.079798 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:38.079716 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29b749ef-a951-4833-bb86-8cb2317f6f15-plugins-config-volume\") pod \"payload-processing-e2e-api-b-50a19d-867c4c57b5-xknkf\" (UID: \"29b749ef-a951-4833-bb86-8cb2317f6f15\") " pod="openshift-ingress/payload-processing-e2e-api-b-50a19d-867c4c57b5-xknkf" Oct 31 00:47:38.079838 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:38.079806 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqjgh\" (UniqueName: \"kubernetes.io/projected/04cf35a1-79f2-4fbd-8a84-7df54fbc0029-kube-api-access-zqjgh\") pod \"payload-pre-processing-e2e-api-b-50a19d-9874b478c-mqhwf\" (UID: \"04cf35a1-79f2-4fbd-8a84-7df54fbc0029\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-50a19d-9874b478c-mqhwf" Oct 31 00:47:38.079838 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:38.079830 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npwz9\" (UniqueName: \"kubernetes.io/projected/29b749ef-a951-4833-bb86-8cb2317f6f15-kube-api-access-npwz9\") pod \"payload-processing-e2e-api-b-50a19d-867c4c57b5-xknkf\" (UID: \"29b749ef-a951-4833-bb86-8cb2317f6f15\") " pod="openshift-ingress/payload-processing-e2e-api-b-50a19d-867c4c57b5-xknkf" Oct 31 00:47:38.181428 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:38.181360 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04cf35a1-79f2-4fbd-8a84-7df54fbc0029-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-50a19d-9874b478c-mqhwf\" (UID: \"04cf35a1-79f2-4fbd-8a84-7df54fbc0029\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-50a19d-9874b478c-mqhwf" Oct 31 00:47:38.181863 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:38.181463 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29b749ef-a951-4833-bb86-8cb2317f6f15-plugins-config-volume\") pod \"payload-processing-e2e-api-b-50a19d-867c4c57b5-xknkf\" (UID: \"29b749ef-a951-4833-bb86-8cb2317f6f15\") " pod="openshift-ingress/payload-processing-e2e-api-b-50a19d-867c4c57b5-xknkf" Oct 31 00:47:38.181863 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:38.181528 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqjgh\" (UniqueName: \"kubernetes.io/projected/04cf35a1-79f2-4fbd-8a84-7df54fbc0029-kube-api-access-zqjgh\") pod \"payload-pre-processing-e2e-api-b-50a19d-9874b478c-mqhwf\" (UID: \"04cf35a1-79f2-4fbd-8a84-7df54fbc0029\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-50a19d-9874b478c-mqhwf" Oct 31 00:47:38.181863 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:38.181563 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npwz9\" (UniqueName: \"kubernetes.io/projected/29b749ef-a951-4833-bb86-8cb2317f6f15-kube-api-access-npwz9\") pod \"payload-processing-e2e-api-b-50a19d-867c4c57b5-xknkf\" (UID: \"29b749ef-a951-4833-bb86-8cb2317f6f15\") " pod="openshift-ingress/payload-processing-e2e-api-b-50a19d-867c4c57b5-xknkf" Oct 31 00:47:38.182114 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:38.182092 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04cf35a1-79f2-4fbd-8a84-7df54fbc0029-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-50a19d-9874b478c-mqhwf\" (UID: \"04cf35a1-79f2-4fbd-8a84-7df54fbc0029\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-50a19d-9874b478c-mqhwf" Oct 31 00:47:38.182236 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:38.182215 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29b749ef-a951-4833-bb86-8cb2317f6f15-plugins-config-volume\") pod \"payload-processing-e2e-api-b-50a19d-867c4c57b5-xknkf\" (UID: \"29b749ef-a951-4833-bb86-8cb2317f6f15\") " pod="openshift-ingress/payload-processing-e2e-api-b-50a19d-867c4c57b5-xknkf" Oct 31 00:47:38.264554 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:38.264515 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npwz9\" (UniqueName: \"kubernetes.io/projected/29b749ef-a951-4833-bb86-8cb2317f6f15-kube-api-access-npwz9\") pod \"payload-processing-e2e-api-b-50a19d-867c4c57b5-xknkf\" (UID: \"29b749ef-a951-4833-bb86-8cb2317f6f15\") " pod="openshift-ingress/payload-processing-e2e-api-b-50a19d-867c4c57b5-xknkf" Oct 31 00:47:38.265588 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:38.265560 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqjgh\" (UniqueName: \"kubernetes.io/projected/04cf35a1-79f2-4fbd-8a84-7df54fbc0029-kube-api-access-zqjgh\") pod \"payload-pre-processing-e2e-api-b-50a19d-9874b478c-mqhwf\" (UID: \"04cf35a1-79f2-4fbd-8a84-7df54fbc0029\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-50a19d-9874b478c-mqhwf" Oct 31 00:47:38.299267 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:38.299173 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-50a19d-9874b478c-mqhwf" Oct 31 00:47:38.388007 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:38.387977 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-50a19d-867c4c57b5-xknkf" Oct 31 00:47:38.709262 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:47:38.709227 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04cf35a1_79f2_4fbd_8a84_7df54fbc0029.slice/crio-b216aba7b0be2e2583f95e49a8c492916dd90dc9336488a469533a56be5adac3 WatchSource:0}: Error finding container b216aba7b0be2e2583f95e49a8c492916dd90dc9336488a469533a56be5adac3: Status 404 returned error can't find the container with id b216aba7b0be2e2583f95e49a8c492916dd90dc9336488a469533a56be5adac3 Oct 31 00:47:38.711660 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:38.711635 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-50a19d-9874b478c-mqhwf"] Oct 31 00:47:38.809534 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:47:38.809503 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29b749ef_a951_4833_bb86_8cb2317f6f15.slice/crio-085d9b2496825831c0b4e7e71a36e3d8b3d2643c18ecf68a930cfa23c2925833 WatchSource:0}: Error finding container 085d9b2496825831c0b4e7e71a36e3d8b3d2643c18ecf68a930cfa23c2925833: Status 404 returned error can't find the container with id 085d9b2496825831c0b4e7e71a36e3d8b3d2643c18ecf68a930cfa23c2925833 Oct 31 00:47:38.812510 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:38.812479 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-50a19d-867c4c57b5-xknkf"] Oct 31 00:47:38.817262 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:38.817237 5222 patch_prober.go:29] interesting pod/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-tf2dv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body= Oct 31 00:47:38.817356 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:38.817281 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-tf2dv" podUID="b191ce67-4983-446c-bd11-394b1f008034" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" Oct 31 00:47:39.375077 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:39.375002 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-50a19d-9874b478c-mqhwf" event={"ID":"04cf35a1-79f2-4fbd-8a84-7df54fbc0029","Type":"ContainerStarted","Data":"4f55dbcc7d66fa4ad84ed926aa1c1877809d114ae0aec61bff70059b1dc8b805"} Oct 31 00:47:39.375647 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:39.375114 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-50a19d-9874b478c-mqhwf" Oct 31 00:47:39.375716 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:39.375138 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-50a19d-9874b478c-mqhwf" event={"ID":"04cf35a1-79f2-4fbd-8a84-7df54fbc0029","Type":"ContainerStarted","Data":"b216aba7b0be2e2583f95e49a8c492916dd90dc9336488a469533a56be5adac3"} Oct 31 00:47:39.377797 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:39.377769 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-50a19d-867c4c57b5-xknkf" event={"ID":"29b749ef-a951-4833-bb86-8cb2317f6f15","Type":"ContainerStarted","Data":"7820dec6e9b18ce0033ab5257f398af1893e869bbed7b802c297796059ce8e80"} Oct 31 00:47:39.377939 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:39.377806 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-50a19d-867c4c57b5-xknkf" event={"ID":"29b749ef-a951-4833-bb86-8cb2317f6f15","Type":"ContainerStarted","Data":"085d9b2496825831c0b4e7e71a36e3d8b3d2643c18ecf68a930cfa23c2925833"} Oct 31 00:47:39.377939 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:39.377844 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-50a19d-867c4c57b5-xknkf" Oct 31 00:47:39.495133 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:39.495077 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-50a19d-9874b478c-mqhwf" podStartSLOduration=2.495063365 podStartE2EDuration="2.495063365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:47:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:47:39.492605545 +0000 UTC m=+1951.963775299" watchObservedRunningTime="2026-10-31 00:47:39.495063365 +0000 UTC m=+1951.966233114" Oct 31 00:47:39.642483 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:39.641853 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-50a19d-867c4c57b5-xknkf" podStartSLOduration=2.6418299 podStartE2EDuration="2.6418299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:47:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:47:39.630667383 +0000 UTC m=+1952.101837134" watchObservedRunningTime="2026-10-31 00:47:39.6418299 +0000 UTC m=+1952.112999651" Oct 31 00:47:39.816931 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:39.816636 5222 patch_prober.go:29] interesting pod/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-tf2dv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body= Oct 31 00:47:39.816931 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:39.816721 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-tf2dv" podUID="b191ce67-4983-446c-bd11-394b1f008034" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" Oct 31 00:47:40.816963 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:40.816926 5222 patch_prober.go:29] interesting pod/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-tf2dv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body= Oct 31 00:47:40.817341 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:40.816980 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-tf2dv" podUID="b191ce67-4983-446c-bd11-394b1f008034" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" Oct 31 00:47:41.817289 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:41.817244 5222 patch_prober.go:29] interesting pod/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-tf2dv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body= Oct 31 00:47:41.817758 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:41.817319 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-tf2dv" podUID="b191ce67-4983-446c-bd11-394b1f008034" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" Oct 31 00:47:42.816744 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:42.816706 5222 patch_prober.go:29] interesting pod/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-tf2dv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body= Oct 31 00:47:42.816930 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:42.816774 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-tf2dv" podUID="b191ce67-4983-446c-bd11-394b1f008034" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" Oct 31 00:47:43.822353 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:43.822318 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-tf2dv" Oct 31 00:47:43.822768 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:43.822680 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-tf2dv" Oct 31 00:47:43.823437 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:43.823419 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-tf2dv" Oct 31 00:47:50.387960 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:50.387933 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-50a19d-9874b478c-mqhwf" Oct 31 00:47:50.388474 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:50.388003 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-50a19d-867c4c57b5-xknkf" Oct 31 00:47:53.307400 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:53.307352 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f891e6-2d2pq"] Oct 31 00:47:53.310964 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:53.310945 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f891e6-2d2pq" Oct 31 00:47:53.354671 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:53.354636 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knlcw\" (UniqueName: \"kubernetes.io/projected/f238e161-3983-4294-9da6-e99c34417a48-kube-api-access-knlcw\") pod \"maas-api-revoke-keys-e2e-api-a-f891e6-2d2pq\" (UID: \"f238e161-3983-4294-9da6-e99c34417a48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f891e6-2d2pq" Oct 31 00:47:53.354845 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:53.354682 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f238e161-3983-4294-9da6-e99c34417a48-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f891e6-2d2pq\" (UID: \"f238e161-3983-4294-9da6-e99c34417a48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f891e6-2d2pq" Oct 31 00:47:53.354845 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:53.354743 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f238e161-3983-4294-9da6-e99c34417a48-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f891e6-2d2pq\" (UID: \"f238e161-3983-4294-9da6-e99c34417a48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f891e6-2d2pq" Oct 31 00:47:53.372259 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:53.372229 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f891e6-2d2pq"] Oct 31 00:47:53.419553 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:53.419522 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:47:53.419754 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:53.419580 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:47:53.419754 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:53.419595 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfz9z\"" Oct 31 00:47:53.419754 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:53.419607 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:47:53.455958 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:53.455925 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knlcw\" (UniqueName: \"kubernetes.io/projected/f238e161-3983-4294-9da6-e99c34417a48-kube-api-access-knlcw\") pod \"maas-api-revoke-keys-e2e-api-a-f891e6-2d2pq\" (UID: \"f238e161-3983-4294-9da6-e99c34417a48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f891e6-2d2pq" Oct 31 00:47:53.455958 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:53.455973 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f238e161-3983-4294-9da6-e99c34417a48-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f891e6-2d2pq\" (UID: \"f238e161-3983-4294-9da6-e99c34417a48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f891e6-2d2pq" Oct 31 00:47:53.456207 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:53.456002 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f238e161-3983-4294-9da6-e99c34417a48-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f891e6-2d2pq\" (UID: \"f238e161-3983-4294-9da6-e99c34417a48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f891e6-2d2pq" Oct 31 00:47:53.456901 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:53.456828 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f238e161-3983-4294-9da6-e99c34417a48-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f891e6-2d2pq\" (UID: \"f238e161-3983-4294-9da6-e99c34417a48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f891e6-2d2pq" Oct 31 00:47:53.458711 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:53.458689 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f238e161-3983-4294-9da6-e99c34417a48-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f891e6-2d2pq\" (UID: \"f238e161-3983-4294-9da6-e99c34417a48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f891e6-2d2pq" Oct 31 00:47:53.518940 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:53.518908 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knlcw\" (UniqueName: \"kubernetes.io/projected/f238e161-3983-4294-9da6-e99c34417a48-kube-api-access-knlcw\") pod \"maas-api-revoke-keys-e2e-api-a-f891e6-2d2pq\" (UID: \"f238e161-3983-4294-9da6-e99c34417a48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f891e6-2d2pq" Oct 31 00:47:53.626872 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:53.626791 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f891e6-2d2pq" Oct 31 00:47:54.002729 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:47:54.002695 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf238e161_3983_4294_9da6_e99c34417a48.slice/crio-1b1742e0cd63023aa0f103c1f63de272d6af798740764bee8d8146adf75f6ac6 WatchSource:0}: Error finding container 1b1742e0cd63023aa0f103c1f63de272d6af798740764bee8d8146adf75f6ac6: Status 404 returned error can't find the container with id 1b1742e0cd63023aa0f103c1f63de272d6af798740764bee8d8146adf75f6ac6 Oct 31 00:47:54.005606 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:54.005579 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f891e6-2d2pq"] Oct 31 00:47:54.538595 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:54.538563 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f891e6-2d2pq" event={"ID":"f238e161-3983-4294-9da6-e99c34417a48","Type":"ContainerStarted","Data":"b9575b0f0e60615f1565c41ee5035718819a039d3a6f9f6b82aa1309effe4818"} Oct 31 00:47:54.539042 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:54.538623 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f891e6-2d2pq" event={"ID":"f238e161-3983-4294-9da6-e99c34417a48","Type":"ContainerStarted","Data":"1b1742e0cd63023aa0f103c1f63de272d6af798740764bee8d8146adf75f6ac6"} Oct 31 00:47:54.660065 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:54.660008 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f891e6-2d2pq" podStartSLOduration=1.659993856 podStartE2EDuration="1.659993856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:47:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:47:54.657536829 +0000 UTC m=+1967.128706579" watchObservedRunningTime="2026-10-31 00:47:54.659993856 +0000 UTC m=+1967.131163605" Oct 31 00:47:55.555289 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:55.555255 5222 generic.go:356] "Generic (PLEG): container finished" podID="f238e161-3983-4294-9da6-e99c34417a48" containerID="b9575b0f0e60615f1565c41ee5035718819a039d3a6f9f6b82aa1309effe4818" exitCode=0 Oct 31 00:47:55.556191 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:55.556159 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f891e6-2d2pq" event={"ID":"f238e161-3983-4294-9da6-e99c34417a48","Type":"ContainerDied","Data":"b9575b0f0e60615f1565c41ee5035718819a039d3a6f9f6b82aa1309effe4818"} Oct 31 00:47:56.729950 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:56.729922 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f891e6-2d2pq" Oct 31 00:47:56.810459 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:56.810426 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f238e161-3983-4294-9da6-e99c34417a48-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f238e161-3983-4294-9da6-e99c34417a48-maas-api-xks-ca\") pod \"f238e161-3983-4294-9da6-e99c34417a48\" (UID: \"f238e161-3983-4294-9da6-e99c34417a48\") " Oct 31 00:47:56.810654 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:56.810482 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f238e161-3983-4294-9da6-e99c34417a48-kube-api-access-knlcw\" (UniqueName: \"kubernetes.io/projected/f238e161-3983-4294-9da6-e99c34417a48-kube-api-access-knlcw\") pod \"f238e161-3983-4294-9da6-e99c34417a48\" (UID: \"f238e161-3983-4294-9da6-e99c34417a48\") " Oct 31 00:47:56.810654 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:56.810560 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f238e161-3983-4294-9da6-e99c34417a48-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f238e161-3983-4294-9da6-e99c34417a48-maas-api-openshift-ca\") pod \"f238e161-3983-4294-9da6-e99c34417a48\" (UID: \"f238e161-3983-4294-9da6-e99c34417a48\") " Oct 31 00:47:56.810998 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:56.810959 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f238e161-3983-4294-9da6-e99c34417a48-maas-api-openshift-ca" pod "f238e161-3983-4294-9da6-e99c34417a48" (UID: "f238e161-3983-4294-9da6-e99c34417a48"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:47:56.811549 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:56.811522 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f238e161-3983-4294-9da6-e99c34417a48-maas-api-openshift-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:56.812859 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:56.812837 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f238e161-3983-4294-9da6-e99c34417a48-kube-api-access-knlcw" pod "f238e161-3983-4294-9da6-e99c34417a48" (UID: "f238e161-3983-4294-9da6-e99c34417a48"). InnerVolumeSpecName "kube-api-access-knlcw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:47:56.812981 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:56.812957 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f238e161-3983-4294-9da6-e99c34417a48-maas-api-xks-ca" pod "f238e161-3983-4294-9da6-e99c34417a48" (UID: "f238e161-3983-4294-9da6-e99c34417a48"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:47:56.912765 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:56.912723 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f238e161-3983-4294-9da6-e99c34417a48-maas-api-xks-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:56.912765 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:56.912762 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knlcw\" (UniqueName: \"kubernetes.io/projected/f238e161-3983-4294-9da6-e99c34417a48-kube-api-access-knlcw\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:57.573831 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:57.573750 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f891e6-2d2pq" event={"ID":"f238e161-3983-4294-9da6-e99c34417a48","Type":"ContainerDied","Data":"1b1742e0cd63023aa0f103c1f63de272d6af798740764bee8d8146adf75f6ac6"} Oct 31 00:47:57.573831 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:57.573782 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f891e6-2d2pq" Oct 31 00:47:57.573831 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:47:57.573828 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1b1742e0cd63023aa0f103c1f63de272d6af798740764bee8d8146adf75f6ac6" Oct 31 00:48:01.190953 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:01.190912 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d58cd3-rzvp7"] Oct 31 00:48:01.191690 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:01.191675 5222 cpu_manager.go:414] "containerMap: removing container" podUID="f238e161-3983-4294-9da6-e99c34417a48" containerName="revoke-keys" Oct 31 00:48:01.191749 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:01.191692 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f238e161-3983-4294-9da6-e99c34417a48" containerName="revoke-keys" Oct 31 00:48:01.191834 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:01.191825 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="f238e161-3983-4294-9da6-e99c34417a48" containerName="revoke-keys" Oct 31 00:48:01.194696 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:01.194677 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d58cd3-rzvp7" Oct 31 00:48:01.248542 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:01.248506 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d58cd3-rzvp7"] Oct 31 00:48:01.288807 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:01.288775 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvfvz\" (UniqueName: \"kubernetes.io/projected/ca2d3367-eb7f-4c59-bbcf-2981f54a3806-kube-api-access-xvfvz\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-d58cd3-rzvp7\" (UID: \"ca2d3367-eb7f-4c59-bbcf-2981f54a3806\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d58cd3-rzvp7" Oct 31 00:48:01.289033 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:01.288985 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca2d3367-eb7f-4c59-bbcf-2981f54a3806-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-d58cd3-rzvp7\" (UID: \"ca2d3367-eb7f-4c59-bbcf-2981f54a3806\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d58cd3-rzvp7" Oct 31 00:48:01.289103 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:01.289034 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca2d3367-eb7f-4c59-bbcf-2981f54a3806-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-d58cd3-rzvp7\" (UID: \"ca2d3367-eb7f-4c59-bbcf-2981f54a3806\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d58cd3-rzvp7" Oct 31 00:48:01.295283 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:01.295252 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:48:01.295539 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:01.295290 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:48:01.295684 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:01.295326 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfz9z\"" Oct 31 00:48:01.295684 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:01.295339 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:48:01.390758 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:01.390707 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca2d3367-eb7f-4c59-bbcf-2981f54a3806-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-d58cd3-rzvp7\" (UID: \"ca2d3367-eb7f-4c59-bbcf-2981f54a3806\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d58cd3-rzvp7" Oct 31 00:48:01.390956 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:01.390783 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca2d3367-eb7f-4c59-bbcf-2981f54a3806-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-d58cd3-rzvp7\" (UID: \"ca2d3367-eb7f-4c59-bbcf-2981f54a3806\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d58cd3-rzvp7" Oct 31 00:48:01.391039 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:01.391017 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvfvz\" (UniqueName: \"kubernetes.io/projected/ca2d3367-eb7f-4c59-bbcf-2981f54a3806-kube-api-access-xvfvz\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-d58cd3-rzvp7\" (UID: \"ca2d3367-eb7f-4c59-bbcf-2981f54a3806\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d58cd3-rzvp7" Oct 31 00:48:01.391562 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:01.391535 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca2d3367-eb7f-4c59-bbcf-2981f54a3806-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-d58cd3-rzvp7\" (UID: \"ca2d3367-eb7f-4c59-bbcf-2981f54a3806\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d58cd3-rzvp7" Oct 31 00:48:01.393395 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:01.393362 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca2d3367-eb7f-4c59-bbcf-2981f54a3806-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-d58cd3-rzvp7\" (UID: \"ca2d3367-eb7f-4c59-bbcf-2981f54a3806\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d58cd3-rzvp7" Oct 31 00:48:01.454273 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:01.454203 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvfvz\" (UniqueName: \"kubernetes.io/projected/ca2d3367-eb7f-4c59-bbcf-2981f54a3806-kube-api-access-xvfvz\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-d58cd3-rzvp7\" (UID: \"ca2d3367-eb7f-4c59-bbcf-2981f54a3806\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d58cd3-rzvp7" Oct 31 00:48:01.508440 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:01.508394 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d58cd3-rzvp7" Oct 31 00:48:01.669125 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:01.669093 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-vkvcx"] Oct 31 00:48:01.669328 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:01.669298 5222 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-vkvcx" podUID="f45187f1-0814-4c12-bdcb-158eeca665f4" containerName="istio-proxy" containerID="cri-o://3d55a1d7aa5f49d1d7975b99c891dcd99554b570386df2c326ae130bf1efec56" gracePeriod=30 Oct 31 00:48:01.911663 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:48:01.911633 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca2d3367_eb7f_4c59_bbcf_2981f54a3806.slice/crio-87dc52164717f82d0abe490685ef64f2a8bf1477eaeb42d6f3ff7e8e92a0eda1 WatchSource:0}: Error finding container 87dc52164717f82d0abe490685ef64f2a8bf1477eaeb42d6f3ff7e8e92a0eda1: Status 404 returned error can't find the container with id 87dc52164717f82d0abe490685ef64f2a8bf1477eaeb42d6f3ff7e8e92a0eda1 Oct 31 00:48:01.915261 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:01.915233 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d58cd3-rzvp7"] Oct 31 00:48:02.622562 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:02.622520 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d58cd3-rzvp7" event={"ID":"ca2d3367-eb7f-4c59-bbcf-2981f54a3806","Type":"ContainerStarted","Data":"a1e6ade7838d5c818c690a3fb64bc36b8f67c1d3876d415d0f2f853d77edcb9c"} Oct 31 00:48:02.623002 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:02.622588 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d58cd3-rzvp7" event={"ID":"ca2d3367-eb7f-4c59-bbcf-2981f54a3806","Type":"ContainerStarted","Data":"87dc52164717f82d0abe490685ef64f2a8bf1477eaeb42d6f3ff7e8e92a0eda1"} Oct 31 00:48:02.748300 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:02.748245 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d58cd3-rzvp7" podStartSLOduration=1.748230647 podStartE2EDuration="1.748230647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:48:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:48:02.74572821 +0000 UTC m=+1975.216897958" watchObservedRunningTime="2026-10-31 00:48:02.748230647 +0000 UTC m=+1975.219400496" Oct 31 00:48:03.632720 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:03.632686 5222 generic.go:356] "Generic (PLEG): container finished" podID="ca2d3367-eb7f-4c59-bbcf-2981f54a3806" containerID="a1e6ade7838d5c818c690a3fb64bc36b8f67c1d3876d415d0f2f853d77edcb9c" exitCode=0 Oct 31 00:48:03.633167 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:03.632791 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d58cd3-rzvp7" event={"ID":"ca2d3367-eb7f-4c59-bbcf-2981f54a3806","Type":"ContainerDied","Data":"a1e6ade7838d5c818c690a3fb64bc36b8f67c1d3876d415d0f2f853d77edcb9c"} Oct 31 00:48:04.024720 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:04.024679 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f891e6-5b8bd8d58f-wpn4q"] Oct 31 00:48:04.129552 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:04.129495 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f891e6-5cf4b56747-st4dd"] Oct 31 00:48:04.817187 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:04.817160 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d58cd3-rzvp7" Oct 31 00:48:04.955877 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:04.955775 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca2d3367-eb7f-4c59-bbcf-2981f54a3806-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca2d3367-eb7f-4c59-bbcf-2981f54a3806-maas-api-openshift-ca\") pod \"ca2d3367-eb7f-4c59-bbcf-2981f54a3806\" (UID: \"ca2d3367-eb7f-4c59-bbcf-2981f54a3806\") " Oct 31 00:48:04.955877 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:04.955853 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca2d3367-eb7f-4c59-bbcf-2981f54a3806-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca2d3367-eb7f-4c59-bbcf-2981f54a3806-maas-api-xks-ca\") pod \"ca2d3367-eb7f-4c59-bbcf-2981f54a3806\" (UID: \"ca2d3367-eb7f-4c59-bbcf-2981f54a3806\") " Oct 31 00:48:04.956204 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:04.956188 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca2d3367-eb7f-4c59-bbcf-2981f54a3806-kube-api-access-xvfvz\" (UniqueName: \"kubernetes.io/projected/ca2d3367-eb7f-4c59-bbcf-2981f54a3806-kube-api-access-xvfvz\") pod \"ca2d3367-eb7f-4c59-bbcf-2981f54a3806\" (UID: \"ca2d3367-eb7f-4c59-bbcf-2981f54a3806\") " Oct 31 00:48:04.956279 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:04.956239 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca2d3367-eb7f-4c59-bbcf-2981f54a3806-maas-api-openshift-ca" pod "ca2d3367-eb7f-4c59-bbcf-2981f54a3806" (UID: "ca2d3367-eb7f-4c59-bbcf-2981f54a3806"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:48:04.956799 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:04.956776 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca2d3367-eb7f-4c59-bbcf-2981f54a3806-maas-api-openshift-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:04.958873 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:04.958846 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca2d3367-eb7f-4c59-bbcf-2981f54a3806-maas-api-xks-ca" pod "ca2d3367-eb7f-4c59-bbcf-2981f54a3806" (UID: "ca2d3367-eb7f-4c59-bbcf-2981f54a3806"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:48:04.959421 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:04.959054 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca2d3367-eb7f-4c59-bbcf-2981f54a3806-kube-api-access-xvfvz" pod "ca2d3367-eb7f-4c59-bbcf-2981f54a3806" (UID: "ca2d3367-eb7f-4c59-bbcf-2981f54a3806"). InnerVolumeSpecName "kube-api-access-xvfvz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:48:05.058835 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:05.058792 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvfvz\" (UniqueName: \"kubernetes.io/projected/ca2d3367-eb7f-4c59-bbcf-2981f54a3806-kube-api-access-xvfvz\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:05.058835 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:05.058829 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca2d3367-eb7f-4c59-bbcf-2981f54a3806-maas-api-xks-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:05.545405 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:05.543507 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-079967a3-5l6db"] Oct 31 00:48:05.545405 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:05.544694 5222 cpu_manager.go:414] "containerMap: removing container" podUID="ca2d3367-eb7f-4c59-bbcf-2981f54a3806" containerName="revoke-keys" Oct 31 00:48:05.545405 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:05.544723 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca2d3367-eb7f-4c59-bbcf-2981f54a3806" containerName="revoke-keys" Oct 31 00:48:05.545405 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:05.544989 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca2d3367-eb7f-4c59-bbcf-2981f54a3806" containerName="revoke-keys" Oct 31 00:48:05.548502 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:05.548428 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-079967a3-5l6db" Oct 31 00:48:05.588628 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:05.588592 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-079967a3-5l6db"] Oct 31 00:48:05.822904 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:05.822807 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82vkz\" (UniqueName: \"kubernetes.io/projected/2c9a1b97-2895-492c-a5ae-97a6db93bd47-kube-api-access-82vkz\") pod \"maas-api-revoke-sub-079967a3-5l6db\" (UID: \"2c9a1b97-2895-492c-a5ae-97a6db93bd47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-079967a3-5l6db" Oct 31 00:48:05.841755 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:05.840117 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c9a1b97-2895-492c-a5ae-97a6db93bd47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-079967a3-5l6db\" (UID: \"2c9a1b97-2895-492c-a5ae-97a6db93bd47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-079967a3-5l6db" Oct 31 00:48:05.841755 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:05.840343 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c9a1b97-2895-492c-a5ae-97a6db93bd47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-079967a3-5l6db\" (UID: \"2c9a1b97-2895-492c-a5ae-97a6db93bd47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-079967a3-5l6db" Oct 31 00:48:05.882562 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:05.882515 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d58cd3-rzvp7" event={"ID":"ca2d3367-eb7f-4c59-bbcf-2981f54a3806","Type":"ContainerDied","Data":"87dc52164717f82d0abe490685ef64f2a8bf1477eaeb42d6f3ff7e8e92a0eda1"} Oct 31 00:48:05.882777 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:05.882592 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="87dc52164717f82d0abe490685ef64f2a8bf1477eaeb42d6f3ff7e8e92a0eda1" Oct 31 00:48:05.882777 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:05.882686 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d58cd3-rzvp7" Oct 31 00:48:05.944415 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:05.942354 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82vkz\" (UniqueName: \"kubernetes.io/projected/2c9a1b97-2895-492c-a5ae-97a6db93bd47-kube-api-access-82vkz\") pod \"maas-api-revoke-sub-079967a3-5l6db\" (UID: \"2c9a1b97-2895-492c-a5ae-97a6db93bd47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-079967a3-5l6db" Oct 31 00:48:05.944415 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:05.942432 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c9a1b97-2895-492c-a5ae-97a6db93bd47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-079967a3-5l6db\" (UID: \"2c9a1b97-2895-492c-a5ae-97a6db93bd47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-079967a3-5l6db" Oct 31 00:48:05.944415 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:05.942523 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c9a1b97-2895-492c-a5ae-97a6db93bd47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-079967a3-5l6db\" (UID: \"2c9a1b97-2895-492c-a5ae-97a6db93bd47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-079967a3-5l6db" Oct 31 00:48:05.944415 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:05.943488 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c9a1b97-2895-492c-a5ae-97a6db93bd47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-079967a3-5l6db\" (UID: \"2c9a1b97-2895-492c-a5ae-97a6db93bd47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-079967a3-5l6db" Oct 31 00:48:05.944415 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:48:05.944249 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 00:48:05.947083 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:05.947054 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c9a1b97-2895-492c-a5ae-97a6db93bd47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-079967a3-5l6db\" (UID: \"2c9a1b97-2895-492c-a5ae-97a6db93bd47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-079967a3-5l6db" Oct 31 00:48:06.006310 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:06.006273 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82vkz\" (UniqueName: \"kubernetes.io/projected/2c9a1b97-2895-492c-a5ae-97a6db93bd47-kube-api-access-82vkz\") pod \"maas-api-revoke-sub-079967a3-5l6db\" (UID: \"2c9a1b97-2895-492c-a5ae-97a6db93bd47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-079967a3-5l6db" Oct 31 00:48:06.167165 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:06.167076 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-079967a3-5l6db" Oct 31 00:48:06.579961 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:48:06.579929 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c9a1b97_2895_492c_a5ae_97a6db93bd47.slice/crio-25c21cc5854df3048c5d117919aa39fb7fa6c949b2d18222a9c86923b9f00968 WatchSource:0}: Error finding container 25c21cc5854df3048c5d117919aa39fb7fa6c949b2d18222a9c86923b9f00968: Status 404 returned error can't find the container with id 25c21cc5854df3048c5d117919aa39fb7fa6c949b2d18222a9c86923b9f00968 Oct 31 00:48:06.583167 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:06.583141 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-079967a3-5l6db"] Oct 31 00:48:06.895238 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:06.895206 5222 generic.go:356] "Generic (PLEG): container finished" podID="f45187f1-0814-4c12-bdcb-158eeca665f4" containerID="3d55a1d7aa5f49d1d7975b99c891dcd99554b570386df2c326ae130bf1efec56" exitCode=0 Oct 31 00:48:06.895733 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:06.895284 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-vkvcx" event={"ID":"f45187f1-0814-4c12-bdcb-158eeca665f4","Type":"ContainerDied","Data":"3d55a1d7aa5f49d1d7975b99c891dcd99554b570386df2c326ae130bf1efec56"} Oct 31 00:48:06.897035 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:06.897006 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-079967a3-5l6db" event={"ID":"2c9a1b97-2895-492c-a5ae-97a6db93bd47","Type":"ContainerStarted","Data":"25c21cc5854df3048c5d117919aa39fb7fa6c949b2d18222a9c86923b9f00968"} Oct 31 00:48:06.921887 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:06.921862 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-vkvcx" Oct 31 00:48:06.970610 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:06.970581 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f45187f1-0814-4c12-bdcb-158eeca665f4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f45187f1-0814-4c12-bdcb-158eeca665f4-istiod-ca-cert\") pod \"f45187f1-0814-4c12-bdcb-158eeca665f4\" (UID: \"f45187f1-0814-4c12-bdcb-158eeca665f4\") " Oct 31 00:48:06.970779 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:06.970642 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f45187f1-0814-4c12-bdcb-158eeca665f4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f45187f1-0814-4c12-bdcb-158eeca665f4-istio-podinfo\") pod \"f45187f1-0814-4c12-bdcb-158eeca665f4\" (UID: \"f45187f1-0814-4c12-bdcb-158eeca665f4\") " Oct 31 00:48:06.970851 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:06.970768 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f45187f1-0814-4c12-bdcb-158eeca665f4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f45187f1-0814-4c12-bdcb-158eeca665f4-istio-data\") pod \"f45187f1-0814-4c12-bdcb-158eeca665f4\" (UID: \"f45187f1-0814-4c12-bdcb-158eeca665f4\") " Oct 31 00:48:06.970924 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:06.970856 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f45187f1-0814-4c12-bdcb-158eeca665f4-istio-token\" (UniqueName: \"kubernetes.io/projected/f45187f1-0814-4c12-bdcb-158eeca665f4-istio-token\") pod \"f45187f1-0814-4c12-bdcb-158eeca665f4\" (UID: \"f45187f1-0814-4c12-bdcb-158eeca665f4\") " Oct 31 00:48:06.970979 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:06.970949 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f45187f1-0814-4c12-bdcb-158eeca665f4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f45187f1-0814-4c12-bdcb-158eeca665f4-istio-envoy\") pod \"f45187f1-0814-4c12-bdcb-158eeca665f4\" (UID: \"f45187f1-0814-4c12-bdcb-158eeca665f4\") " Oct 31 00:48:06.971036 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:06.970987 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f45187f1-0814-4c12-bdcb-158eeca665f4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f45187f1-0814-4c12-bdcb-158eeca665f4-workload-certs\") pod \"f45187f1-0814-4c12-bdcb-158eeca665f4\" (UID: \"f45187f1-0814-4c12-bdcb-158eeca665f4\") " Oct 31 00:48:06.971036 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:06.971000 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f45187f1-0814-4c12-bdcb-158eeca665f4-istiod-ca-cert" pod "f45187f1-0814-4c12-bdcb-158eeca665f4" (UID: "f45187f1-0814-4c12-bdcb-158eeca665f4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:48:06.971036 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:06.971008 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f45187f1-0814-4c12-bdcb-158eeca665f4-istio-data" pod "f45187f1-0814-4c12-bdcb-158eeca665f4" (UID: "f45187f1-0814-4c12-bdcb-158eeca665f4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:06.971036 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:06.971019 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f45187f1-0814-4c12-bdcb-158eeca665f4-kube-api-access-d7b7s\" (UniqueName: \"kubernetes.io/projected/f45187f1-0814-4c12-bdcb-158eeca665f4-kube-api-access-d7b7s\") pod \"f45187f1-0814-4c12-bdcb-158eeca665f4\" (UID: \"f45187f1-0814-4c12-bdcb-158eeca665f4\") " Oct 31 00:48:06.971234 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:06.971212 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f45187f1-0814-4c12-bdcb-158eeca665f4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f45187f1-0814-4c12-bdcb-158eeca665f4-workload-socket\") pod \"f45187f1-0814-4c12-bdcb-158eeca665f4\" (UID: \"f45187f1-0814-4c12-bdcb-158eeca665f4\") " Oct 31 00:48:06.971310 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:06.971282 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f45187f1-0814-4c12-bdcb-158eeca665f4-workload-certs" pod "f45187f1-0814-4c12-bdcb-158eeca665f4" (UID: "f45187f1-0814-4c12-bdcb-158eeca665f4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:06.971495 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:06.971467 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f45187f1-0814-4c12-bdcb-158eeca665f4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f45187f1-0814-4c12-bdcb-158eeca665f4-credential-socket\") pod \"f45187f1-0814-4c12-bdcb-158eeca665f4\" (UID: \"f45187f1-0814-4c12-bdcb-158eeca665f4\") " Oct 31 00:48:06.971670 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:06.971499 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f45187f1-0814-4c12-bdcb-158eeca665f4-workload-socket" pod "f45187f1-0814-4c12-bdcb-158eeca665f4" (UID: "f45187f1-0814-4c12-bdcb-158eeca665f4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:06.971915 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:06.971744 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f45187f1-0814-4c12-bdcb-158eeca665f4-credential-socket" pod "f45187f1-0814-4c12-bdcb-158eeca665f4" (UID: "f45187f1-0814-4c12-bdcb-158eeca665f4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:06.972066 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:06.972047 5222 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f45187f1-0814-4c12-bdcb-158eeca665f4-istiod-ca-cert\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:06.972139 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:06.972069 5222 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f45187f1-0814-4c12-bdcb-158eeca665f4-istio-data\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:06.972139 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:06.972079 5222 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f45187f1-0814-4c12-bdcb-158eeca665f4-workload-certs\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:06.972139 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:06.972088 5222 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f45187f1-0814-4c12-bdcb-158eeca665f4-workload-socket\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:06.972139 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:06.972098 5222 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f45187f1-0814-4c12-bdcb-158eeca665f4-credential-socket\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:06.973198 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:06.973175 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f45187f1-0814-4c12-bdcb-158eeca665f4-istio-podinfo" pod "f45187f1-0814-4c12-bdcb-158eeca665f4" (UID: "f45187f1-0814-4c12-bdcb-158eeca665f4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:48:06.973723 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:06.973693 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f45187f1-0814-4c12-bdcb-158eeca665f4-istio-envoy" pod "f45187f1-0814-4c12-bdcb-158eeca665f4" (UID: "f45187f1-0814-4c12-bdcb-158eeca665f4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:06.973825 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:06.973812 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f45187f1-0814-4c12-bdcb-158eeca665f4-kube-api-access-d7b7s" pod "f45187f1-0814-4c12-bdcb-158eeca665f4" (UID: "f45187f1-0814-4c12-bdcb-158eeca665f4"). InnerVolumeSpecName "kube-api-access-d7b7s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:48:06.973938 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:06.973916 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f45187f1-0814-4c12-bdcb-158eeca665f4-istio-token" pod "f45187f1-0814-4c12-bdcb-158eeca665f4" (UID: "f45187f1-0814-4c12-bdcb-158eeca665f4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:48:07.074004 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:07.073961 5222 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f45187f1-0814-4c12-bdcb-158eeca665f4-istio-token\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:07.074004 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:07.073996 5222 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f45187f1-0814-4c12-bdcb-158eeca665f4-istio-envoy\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:07.074004 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:07.074006 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7b7s\" (UniqueName: \"kubernetes.io/projected/f45187f1-0814-4c12-bdcb-158eeca665f4-kube-api-access-d7b7s\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:07.074256 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:07.074037 5222 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f45187f1-0814-4c12-bdcb-158eeca665f4-istio-podinfo\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:07.906457 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:07.906336 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-vkvcx" Oct 31 00:48:07.906457 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:07.906347 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-vkvcx" event={"ID":"f45187f1-0814-4c12-bdcb-158eeca665f4","Type":"ContainerDied","Data":"2c763a48762f42eb92ee903b42118beeb1a1c64b77d2d5ddefaca4f09dffbe19"} Oct 31 00:48:07.906457 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:07.906429 5222 scope.go:122] "RemoveContainer" containerID="3d55a1d7aa5f49d1d7975b99c891dcd99554b570386df2c326ae130bf1efec56" Oct 31 00:48:07.908455 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:07.908430 5222 generic.go:356] "Generic (PLEG): container finished" podID="2c9a1b97-2895-492c-a5ae-97a6db93bd47" containerID="f9dc36877834ec53cf93518d59a7ae1f923c22d665dc2aa77b62d7c8c6c8f9e7" exitCode=0 Oct 31 00:48:07.908584 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:07.908513 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-079967a3-5l6db" event={"ID":"2c9a1b97-2895-492c-a5ae-97a6db93bd47","Type":"ContainerDied","Data":"f9dc36877834ec53cf93518d59a7ae1f923c22d665dc2aa77b62d7c8c6c8f9e7"} Oct 31 00:48:08.248563 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:08.248484 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-vkvcx"] Oct 31 00:48:08.253469 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:08.253442 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-vkvcx"] Oct 31 00:48:09.025070 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:09.025022 5222 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-f891e6-5b8bd8d58f-wpn4q" podUID="2b11eda2-8622-4e58-ad66-e13985b1dd36" containerName="payload-processing" containerID="cri-o://0cf16ed2e7b751144ec3a3b69c45e986d1aed2a88fc35c9bb399ca87bab7d2e7" gracePeriod=25 Oct 31 00:48:09.131276 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:09.131236 5222 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f891e6-5cf4b56747-st4dd" podUID="815b9a5c-72ac-447b-bd5f-bb0df408c5f3" containerName="payload-pre-processing" containerID="cri-o://cc59c8674e4d5ee2bad47707cc6c3feded1509019fb145c10934697f4db595aa" gracePeriod=25 Oct 31 00:48:09.213627 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:09.213590 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-079967a3-5l6db" Oct 31 00:48:09.303279 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:09.303251 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f891e6-5b8bd8d58f-wpn4q" Oct 31 00:48:09.306332 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:09.306307 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c9a1b97-2895-492c-a5ae-97a6db93bd47-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c9a1b97-2895-492c-a5ae-97a6db93bd47-maas-api-openshift-ca\") pod \"2c9a1b97-2895-492c-a5ae-97a6db93bd47\" (UID: \"2c9a1b97-2895-492c-a5ae-97a6db93bd47\") " Oct 31 00:48:09.306494 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:09.306448 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c9a1b97-2895-492c-a5ae-97a6db93bd47-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c9a1b97-2895-492c-a5ae-97a6db93bd47-maas-api-xks-ca\") pod \"2c9a1b97-2895-492c-a5ae-97a6db93bd47\" (UID: \"2c9a1b97-2895-492c-a5ae-97a6db93bd47\") " Oct 31 00:48:09.306618 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:09.306603 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c9a1b97-2895-492c-a5ae-97a6db93bd47-kube-api-access-82vkz\" (UniqueName: \"kubernetes.io/projected/2c9a1b97-2895-492c-a5ae-97a6db93bd47-kube-api-access-82vkz\") pod \"2c9a1b97-2895-492c-a5ae-97a6db93bd47\" (UID: \"2c9a1b97-2895-492c-a5ae-97a6db93bd47\") " Oct 31 00:48:09.306746 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:09.306708 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c9a1b97-2895-492c-a5ae-97a6db93bd47-maas-api-openshift-ca" pod "2c9a1b97-2895-492c-a5ae-97a6db93bd47" (UID: "2c9a1b97-2895-492c-a5ae-97a6db93bd47"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:48:09.307182 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:09.307157 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c9a1b97-2895-492c-a5ae-97a6db93bd47-maas-api-openshift-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:09.309083 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:09.309060 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c9a1b97-2895-492c-a5ae-97a6db93bd47-maas-api-xks-ca" pod "2c9a1b97-2895-492c-a5ae-97a6db93bd47" (UID: "2c9a1b97-2895-492c-a5ae-97a6db93bd47"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:48:09.309314 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:09.309292 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c9a1b97-2895-492c-a5ae-97a6db93bd47-kube-api-access-82vkz" pod "2c9a1b97-2895-492c-a5ae-97a6db93bd47" (UID: "2c9a1b97-2895-492c-a5ae-97a6db93bd47"). InnerVolumeSpecName "kube-api-access-82vkz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:48:09.408330 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:09.408303 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b11eda2-8622-4e58-ad66-e13985b1dd36-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b11eda2-8622-4e58-ad66-e13985b1dd36-plugins-config-volume\") pod \"2b11eda2-8622-4e58-ad66-e13985b1dd36\" (UID: \"2b11eda2-8622-4e58-ad66-e13985b1dd36\") " Oct 31 00:48:09.408505 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:09.408475 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b11eda2-8622-4e58-ad66-e13985b1dd36-kube-api-access-m2rfc\" (UniqueName: \"kubernetes.io/projected/2b11eda2-8622-4e58-ad66-e13985b1dd36-kube-api-access-m2rfc\") pod \"2b11eda2-8622-4e58-ad66-e13985b1dd36\" (UID: \"2b11eda2-8622-4e58-ad66-e13985b1dd36\") " Oct 31 00:48:09.408756 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:09.408727 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b11eda2-8622-4e58-ad66-e13985b1dd36-plugins-config-volume" pod "2b11eda2-8622-4e58-ad66-e13985b1dd36" (UID: "2b11eda2-8622-4e58-ad66-e13985b1dd36"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:48:09.409091 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:09.409071 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82vkz\" (UniqueName: \"kubernetes.io/projected/2c9a1b97-2895-492c-a5ae-97a6db93bd47-kube-api-access-82vkz\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:09.409160 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:09.409098 5222 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b11eda2-8622-4e58-ad66-e13985b1dd36-plugins-config-volume\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:09.409160 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:09.409112 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c9a1b97-2895-492c-a5ae-97a6db93bd47-maas-api-xks-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:09.410886 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:09.410865 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b11eda2-8622-4e58-ad66-e13985b1dd36-kube-api-access-m2rfc" pod "2b11eda2-8622-4e58-ad66-e13985b1dd36" (UID: "2b11eda2-8622-4e58-ad66-e13985b1dd36"). InnerVolumeSpecName "kube-api-access-m2rfc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:48:09.413384 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:09.413348 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f891e6-5cf4b56747-st4dd" Oct 31 00:48:09.510258 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:09.510166 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/815b9a5c-72ac-447b-bd5f-bb0df408c5f3-kube-api-access-8xlfc\" (UniqueName: \"kubernetes.io/projected/815b9a5c-72ac-447b-bd5f-bb0df408c5f3-kube-api-access-8xlfc\") pod \"815b9a5c-72ac-447b-bd5f-bb0df408c5f3\" (UID: \"815b9a5c-72ac-447b-bd5f-bb0df408c5f3\") " Oct 31 00:48:09.510258 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:09.510228 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/815b9a5c-72ac-447b-bd5f-bb0df408c5f3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/815b9a5c-72ac-447b-bd5f-bb0df408c5f3-plugins-config-volume\") pod \"815b9a5c-72ac-447b-bd5f-bb0df408c5f3\" (UID: \"815b9a5c-72ac-447b-bd5f-bb0df408c5f3\") " Oct 31 00:48:09.510655 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:09.510625 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/815b9a5c-72ac-447b-bd5f-bb0df408c5f3-plugins-config-volume" pod "815b9a5c-72ac-447b-bd5f-bb0df408c5f3" (UID: "815b9a5c-72ac-447b-bd5f-bb0df408c5f3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:48:09.510791 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:09.510774 5222 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/815b9a5c-72ac-447b-bd5f-bb0df408c5f3-plugins-config-volume\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:09.510857 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:09.510795 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2rfc\" (UniqueName: \"kubernetes.io/projected/2b11eda2-8622-4e58-ad66-e13985b1dd36-kube-api-access-m2rfc\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:09.512627 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:09.512598 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/815b9a5c-72ac-447b-bd5f-bb0df408c5f3-kube-api-access-8xlfc" pod "815b9a5c-72ac-447b-bd5f-bb0df408c5f3" (UID: "815b9a5c-72ac-447b-bd5f-bb0df408c5f3"). InnerVolumeSpecName "kube-api-access-8xlfc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:48:09.611902 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:09.611862 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xlfc\" (UniqueName: \"kubernetes.io/projected/815b9a5c-72ac-447b-bd5f-bb0df408c5f3-kube-api-access-8xlfc\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:09.659169 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:09.659129 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f45187f1-0814-4c12-bdcb-158eeca665f4" path="/var/lib/kubelet/pods/f45187f1-0814-4c12-bdcb-158eeca665f4/volumes" Oct 31 00:48:09.928286 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:09.928254 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-079967a3-5l6db" Oct 31 00:48:09.928286 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:09.928268 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-079967a3-5l6db" event={"ID":"2c9a1b97-2895-492c-a5ae-97a6db93bd47","Type":"ContainerDied","Data":"25c21cc5854df3048c5d117919aa39fb7fa6c949b2d18222a9c86923b9f00968"} Oct 31 00:48:09.928593 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:09.928328 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="25c21cc5854df3048c5d117919aa39fb7fa6c949b2d18222a9c86923b9f00968" Oct 31 00:48:09.929996 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:09.929960 5222 generic.go:356] "Generic (PLEG): container finished" podID="2b11eda2-8622-4e58-ad66-e13985b1dd36" containerID="0cf16ed2e7b751144ec3a3b69c45e986d1aed2a88fc35c9bb399ca87bab7d2e7" exitCode=0 Oct 31 00:48:09.930150 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:09.930007 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f891e6-5b8bd8d58f-wpn4q" Oct 31 00:48:09.930150 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:09.930046 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f891e6-5b8bd8d58f-wpn4q" event={"ID":"2b11eda2-8622-4e58-ad66-e13985b1dd36","Type":"ContainerDied","Data":"0cf16ed2e7b751144ec3a3b69c45e986d1aed2a88fc35c9bb399ca87bab7d2e7"} Oct 31 00:48:09.930150 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:09.930095 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f891e6-5b8bd8d58f-wpn4q" event={"ID":"2b11eda2-8622-4e58-ad66-e13985b1dd36","Type":"ContainerDied","Data":"88531a398b76cbea7bb96fe1bf0ad4917e0518336af53845125d95ed8795ef45"} Oct 31 00:48:09.930150 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:09.930122 5222 scope.go:122] "RemoveContainer" containerID="0cf16ed2e7b751144ec3a3b69c45e986d1aed2a88fc35c9bb399ca87bab7d2e7" Oct 31 00:48:09.933261 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:09.933234 5222 generic.go:356] "Generic (PLEG): container finished" podID="815b9a5c-72ac-447b-bd5f-bb0df408c5f3" containerID="cc59c8674e4d5ee2bad47707cc6c3feded1509019fb145c10934697f4db595aa" exitCode=0 Oct 31 00:48:09.933362 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:09.933291 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f891e6-5cf4b56747-st4dd" Oct 31 00:48:09.933362 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:09.933352 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f891e6-5cf4b56747-st4dd" event={"ID":"815b9a5c-72ac-447b-bd5f-bb0df408c5f3","Type":"ContainerDied","Data":"cc59c8674e4d5ee2bad47707cc6c3feded1509019fb145c10934697f4db595aa"} Oct 31 00:48:09.933553 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:09.933422 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f891e6-5cf4b56747-st4dd" event={"ID":"815b9a5c-72ac-447b-bd5f-bb0df408c5f3","Type":"ContainerDied","Data":"4cf5575c34faa8a9f46c557a7d2f839e9b531878db418a325588fd6734614554"} Oct 31 00:48:09.942037 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:09.942012 5222 scope.go:122] "RemoveContainer" containerID="0cf16ed2e7b751144ec3a3b69c45e986d1aed2a88fc35c9bb399ca87bab7d2e7" Oct 31 00:48:09.942359 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:48:09.942340 5222 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cf16ed2e7b751144ec3a3b69c45e986d1aed2a88fc35c9bb399ca87bab7d2e7\": container with ID starting with 0cf16ed2e7b751144ec3a3b69c45e986d1aed2a88fc35c9bb399ca87bab7d2e7 not found: ID does not exist" containerID="0cf16ed2e7b751144ec3a3b69c45e986d1aed2a88fc35c9bb399ca87bab7d2e7" Oct 31 00:48:09.942429 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:09.942403 5222 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cf16ed2e7b751144ec3a3b69c45e986d1aed2a88fc35c9bb399ca87bab7d2e7"} err="failed to get container status \"0cf16ed2e7b751144ec3a3b69c45e986d1aed2a88fc35c9bb399ca87bab7d2e7\": rpc error: code = NotFound desc = could not find container \"0cf16ed2e7b751144ec3a3b69c45e986d1aed2a88fc35c9bb399ca87bab7d2e7\": container with ID starting with 0cf16ed2e7b751144ec3a3b69c45e986d1aed2a88fc35c9bb399ca87bab7d2e7 not found: ID does not exist" Oct 31 00:48:09.942429 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:09.942421 5222 scope.go:122] "RemoveContainer" containerID="cc59c8674e4d5ee2bad47707cc6c3feded1509019fb145c10934697f4db595aa" Oct 31 00:48:09.952551 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:09.952530 5222 scope.go:122] "RemoveContainer" containerID="cc59c8674e4d5ee2bad47707cc6c3feded1509019fb145c10934697f4db595aa" Oct 31 00:48:09.952908 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:48:09.952885 5222 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc59c8674e4d5ee2bad47707cc6c3feded1509019fb145c10934697f4db595aa\": container with ID starting with cc59c8674e4d5ee2bad47707cc6c3feded1509019fb145c10934697f4db595aa not found: ID does not exist" containerID="cc59c8674e4d5ee2bad47707cc6c3feded1509019fb145c10934697f4db595aa" Oct 31 00:48:09.952962 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:09.952917 5222 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc59c8674e4d5ee2bad47707cc6c3feded1509019fb145c10934697f4db595aa"} err="failed to get container status \"cc59c8674e4d5ee2bad47707cc6c3feded1509019fb145c10934697f4db595aa\": rpc error: code = NotFound desc = could not find container \"cc59c8674e4d5ee2bad47707cc6c3feded1509019fb145c10934697f4db595aa\": container with ID starting with cc59c8674e4d5ee2bad47707cc6c3feded1509019fb145c10934697f4db595aa not found: ID does not exist" Oct 31 00:48:10.102102 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:10.102070 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f891e6-5cf4b56747-st4dd"] Oct 31 00:48:10.153197 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:10.153166 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f891e6-5cf4b56747-st4dd"] Oct 31 00:48:10.270867 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:10.270832 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f891e6-5b8bd8d58f-wpn4q"] Oct 31 00:48:10.276355 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:10.276328 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f891e6-5b8bd8d58f-wpn4q"] Oct 31 00:48:11.658529 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:11.658497 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b11eda2-8622-4e58-ad66-e13985b1dd36" path="/var/lib/kubelet/pods/2b11eda2-8622-4e58-ad66-e13985b1dd36/volumes" Oct 31 00:48:11.658973 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:11.658959 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="815b9a5c-72ac-447b-bd5f-bb0df408c5f3" path="/var/lib/kubelet/pods/815b9a5c-72ac-447b-bd5f-bb0df408c5f3/volumes" Oct 31 00:48:13.275901 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:13.275869 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-d58cd3-766b65cfd7-hjv58"] Oct 31 00:48:13.388704 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:13.388669 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d58cd3-6ff958d595-t9nfb"] Oct 31 00:48:18.276292 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:18.276182 5222 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d58cd3-766b65cfd7-hjv58" podUID="2f5b0e39-04be-4933-9512-8c7ff9bdb9cd" containerName="payload-processing" containerID="cri-o://c2b0365fa6ee04175375ddc641e6734aa5facd2f44004496fbcb88e17baedd30" gracePeriod=25 Oct 31 00:48:18.281418 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:48:18.281242 5222 helpers.go:246] readString: Failed to read "/sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f5b0e39_04be_4933_9512_8c7ff9bdb9cd.slice/crio-c2b0365fa6ee04175375ddc641e6734aa5facd2f44004496fbcb88e17baedd30.scope/cpu.max": open /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f5b0e39_04be_4933_9512_8c7ff9bdb9cd.slice/crio-c2b0365fa6ee04175375ddc641e6734aa5facd2f44004496fbcb88e17baedd30.scope/cpu.max: no such device Oct 31 00:48:18.345281 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:48:18.345247 5222 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f5b0e39_04be_4933_9512_8c7ff9bdb9cd.slice/crio-conmon-c2b0365fa6ee04175375ddc641e6734aa5facd2f44004496fbcb88e17baedd30.scope\": RecentStats: unable to find data in memory cache]" Oct 31 00:48:18.389625 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:18.389570 5222 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d58cd3-6ff958d595-t9nfb" podUID="82e8f109-3468-431f-aabf-ef22d4e59878" containerName="payload-pre-processing" containerID="cri-o://011efb1c2bb3e502346e8bc8f6fcf24e05c48fb3ea1caae656854edeb483cc6f" gracePeriod=25 Oct 31 00:48:18.576673 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:18.576641 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d58cd3-766b65cfd7-hjv58" Oct 31 00:48:18.669061 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:18.669027 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f5b0e39-04be-4933-9512-8c7ff9bdb9cd-kube-api-access-zhrkr\" (UniqueName: \"kubernetes.io/projected/2f5b0e39-04be-4933-9512-8c7ff9bdb9cd-kube-api-access-zhrkr\") pod \"2f5b0e39-04be-4933-9512-8c7ff9bdb9cd\" (UID: \"2f5b0e39-04be-4933-9512-8c7ff9bdb9cd\") " Oct 31 00:48:18.669231 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:18.669140 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f5b0e39-04be-4933-9512-8c7ff9bdb9cd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f5b0e39-04be-4933-9512-8c7ff9bdb9cd-plugins-config-volume\") pod \"2f5b0e39-04be-4933-9512-8c7ff9bdb9cd\" (UID: \"2f5b0e39-04be-4933-9512-8c7ff9bdb9cd\") " Oct 31 00:48:18.669629 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:18.669598 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f5b0e39-04be-4933-9512-8c7ff9bdb9cd-plugins-config-volume" pod "2f5b0e39-04be-4933-9512-8c7ff9bdb9cd" (UID: "2f5b0e39-04be-4933-9512-8c7ff9bdb9cd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:48:18.669787 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:18.669769 5222 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f5b0e39-04be-4933-9512-8c7ff9bdb9cd-plugins-config-volume\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:18.671775 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:18.671745 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f5b0e39-04be-4933-9512-8c7ff9bdb9cd-kube-api-access-zhrkr" pod "2f5b0e39-04be-4933-9512-8c7ff9bdb9cd" (UID: "2f5b0e39-04be-4933-9512-8c7ff9bdb9cd"). InnerVolumeSpecName "kube-api-access-zhrkr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:48:18.704622 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:18.704587 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d58cd3-6ff958d595-t9nfb" Oct 31 00:48:18.771080 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:18.771042 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82e8f109-3468-431f-aabf-ef22d4e59878-kube-api-access-2dfms\" (UniqueName: \"kubernetes.io/projected/82e8f109-3468-431f-aabf-ef22d4e59878-kube-api-access-2dfms\") pod \"82e8f109-3468-431f-aabf-ef22d4e59878\" (UID: \"82e8f109-3468-431f-aabf-ef22d4e59878\") " Oct 31 00:48:18.771491 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:18.771473 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82e8f109-3468-431f-aabf-ef22d4e59878-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82e8f109-3468-431f-aabf-ef22d4e59878-plugins-config-volume\") pod \"82e8f109-3468-431f-aabf-ef22d4e59878\" (UID: \"82e8f109-3468-431f-aabf-ef22d4e59878\") " Oct 31 00:48:18.771871 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:18.771842 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82e8f109-3468-431f-aabf-ef22d4e59878-plugins-config-volume" pod "82e8f109-3468-431f-aabf-ef22d4e59878" (UID: "82e8f109-3468-431f-aabf-ef22d4e59878"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:48:18.772063 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:18.772045 5222 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82e8f109-3468-431f-aabf-ef22d4e59878-plugins-config-volume\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:18.772132 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:18.772069 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhrkr\" (UniqueName: \"kubernetes.io/projected/2f5b0e39-04be-4933-9512-8c7ff9bdb9cd-kube-api-access-zhrkr\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:18.773354 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:18.773332 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82e8f109-3468-431f-aabf-ef22d4e59878-kube-api-access-2dfms" pod "82e8f109-3468-431f-aabf-ef22d4e59878" (UID: "82e8f109-3468-431f-aabf-ef22d4e59878"). InnerVolumeSpecName "kube-api-access-2dfms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:48:18.873751 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:18.873662 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dfms\" (UniqueName: \"kubernetes.io/projected/82e8f109-3468-431f-aabf-ef22d4e59878-kube-api-access-2dfms\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:19.081453 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:19.081415 5222 generic.go:356] "Generic (PLEG): container finished" podID="82e8f109-3468-431f-aabf-ef22d4e59878" containerID="011efb1c2bb3e502346e8bc8f6fcf24e05c48fb3ea1caae656854edeb483cc6f" exitCode=0 Oct 31 00:48:19.081453 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:19.081438 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d58cd3-6ff958d595-t9nfb" Oct 31 00:48:19.081781 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:19.081490 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d58cd3-6ff958d595-t9nfb" event={"ID":"82e8f109-3468-431f-aabf-ef22d4e59878","Type":"ContainerDied","Data":"011efb1c2bb3e502346e8bc8f6fcf24e05c48fb3ea1caae656854edeb483cc6f"} Oct 31 00:48:19.081781 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:19.081546 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d58cd3-6ff958d595-t9nfb" event={"ID":"82e8f109-3468-431f-aabf-ef22d4e59878","Type":"ContainerDied","Data":"c834460a5610ca6cccff679d9c3efc1c61c2c7bee122336ed60ec6de44e8c1f3"} Oct 31 00:48:19.081781 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:19.081575 5222 scope.go:122] "RemoveContainer" containerID="011efb1c2bb3e502346e8bc8f6fcf24e05c48fb3ea1caae656854edeb483cc6f" Oct 31 00:48:19.083316 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:19.083287 5222 generic.go:356] "Generic (PLEG): container finished" podID="2f5b0e39-04be-4933-9512-8c7ff9bdb9cd" containerID="c2b0365fa6ee04175375ddc641e6734aa5facd2f44004496fbcb88e17baedd30" exitCode=0 Oct 31 00:48:19.083465 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:19.083341 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d58cd3-766b65cfd7-hjv58" Oct 31 00:48:19.083465 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:19.083341 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d58cd3-766b65cfd7-hjv58" event={"ID":"2f5b0e39-04be-4933-9512-8c7ff9bdb9cd","Type":"ContainerDied","Data":"c2b0365fa6ee04175375ddc641e6734aa5facd2f44004496fbcb88e17baedd30"} Oct 31 00:48:19.083544 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:19.083516 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d58cd3-766b65cfd7-hjv58" event={"ID":"2f5b0e39-04be-4933-9512-8c7ff9bdb9cd","Type":"ContainerDied","Data":"c1b4857e3de939e13d15ece1e418604bc1801ad5445599ebff631cfe7e2332d8"} Oct 31 00:48:19.094778 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:19.094754 5222 scope.go:122] "RemoveContainer" containerID="011efb1c2bb3e502346e8bc8f6fcf24e05c48fb3ea1caae656854edeb483cc6f" Oct 31 00:48:19.095095 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:48:19.095075 5222 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"011efb1c2bb3e502346e8bc8f6fcf24e05c48fb3ea1caae656854edeb483cc6f\": container with ID starting with 011efb1c2bb3e502346e8bc8f6fcf24e05c48fb3ea1caae656854edeb483cc6f not found: ID does not exist" containerID="011efb1c2bb3e502346e8bc8f6fcf24e05c48fb3ea1caae656854edeb483cc6f" Oct 31 00:48:19.095178 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:19.095102 5222 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"011efb1c2bb3e502346e8bc8f6fcf24e05c48fb3ea1caae656854edeb483cc6f"} err="failed to get container status \"011efb1c2bb3e502346e8bc8f6fcf24e05c48fb3ea1caae656854edeb483cc6f\": rpc error: code = NotFound desc = could not find container \"011efb1c2bb3e502346e8bc8f6fcf24e05c48fb3ea1caae656854edeb483cc6f\": container with ID starting with 011efb1c2bb3e502346e8bc8f6fcf24e05c48fb3ea1caae656854edeb483cc6f not found: ID does not exist" Oct 31 00:48:19.095178 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:19.095123 5222 scope.go:122] "RemoveContainer" containerID="c2b0365fa6ee04175375ddc641e6734aa5facd2f44004496fbcb88e17baedd30" Oct 31 00:48:19.105696 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:19.105673 5222 scope.go:122] "RemoveContainer" containerID="c2b0365fa6ee04175375ddc641e6734aa5facd2f44004496fbcb88e17baedd30" Oct 31 00:48:19.106042 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:48:19.106015 5222 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2b0365fa6ee04175375ddc641e6734aa5facd2f44004496fbcb88e17baedd30\": container with ID starting with c2b0365fa6ee04175375ddc641e6734aa5facd2f44004496fbcb88e17baedd30 not found: ID does not exist" containerID="c2b0365fa6ee04175375ddc641e6734aa5facd2f44004496fbcb88e17baedd30" Oct 31 00:48:19.106138 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:19.106048 5222 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2b0365fa6ee04175375ddc641e6734aa5facd2f44004496fbcb88e17baedd30"} err="failed to get container status \"c2b0365fa6ee04175375ddc641e6734aa5facd2f44004496fbcb88e17baedd30\": rpc error: code = NotFound desc = could not find container \"c2b0365fa6ee04175375ddc641e6734aa5facd2f44004496fbcb88e17baedd30\": container with ID starting with c2b0365fa6ee04175375ddc641e6734aa5facd2f44004496fbcb88e17baedd30 not found: ID does not exist" Oct 31 00:48:19.251868 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:19.251783 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-d58cd3-766b65cfd7-hjv58"] Oct 31 00:48:19.302897 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:19.302862 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-d58cd3-766b65cfd7-hjv58"] Oct 31 00:48:19.417538 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:19.417507 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d58cd3-6ff958d595-t9nfb"] Oct 31 00:48:19.420965 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:19.420939 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d58cd3-6ff958d595-t9nfb"] Oct 31 00:48:19.658037 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:19.657998 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f5b0e39-04be-4933-9512-8c7ff9bdb9cd" path="/var/lib/kubelet/pods/2f5b0e39-04be-4933-9512-8c7ff9bdb9cd/volumes" Oct 31 00:48:19.658575 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:19.658553 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82e8f109-3468-431f-aabf-ef22d4e59878" path="/var/lib/kubelet/pods/82e8f109-3468-431f-aabf-ef22d4e59878/volumes" Oct 31 00:48:24.668452 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:24.668379 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-79tk8"] Oct 31 00:48:35.892718 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:48:35.892672 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 00:48:36.240219 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:36.240129 5222 generic.go:356] "Generic (PLEG): container finished" podID="4c57fa36-80bb-403d-97a3-b18601c386aa" containerID="0d360cc605ccde706b1a6139edee34d8c211f6e0ed82e81d3ee0caa7de814e0c" exitCode=0 Oct 31 00:48:36.240219 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:36.240199 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-2594a9-openshift-default-8694ddf5f7-v4jsh" event={"ID":"4c57fa36-80bb-403d-97a3-b18601c386aa","Type":"ContainerDied","Data":"0d360cc605ccde706b1a6139edee34d8c211f6e0ed82e81d3ee0caa7de814e0c"} Oct 31 00:48:36.240674 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:36.240659 5222 scope.go:122] "RemoveContainer" containerID="0d360cc605ccde706b1a6139edee34d8c211f6e0ed82e81d3ee0caa7de814e0c" Oct 31 00:48:36.242737 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:36.242705 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:48:36.242839 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:36.242779 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:48:36.242839 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:36.242817 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:48:37.273853 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:37.273818 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-2594a9-openshift-default-8694ddf5f7-v4jsh" event={"ID":"4c57fa36-80bb-403d-97a3-b18601c386aa","Type":"ContainerStarted","Data":"564a07ec10feb3039d0b988b7be438f8800118f84e222f998af0fad078434a94"} Oct 31 00:48:37.890143 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:37.890104 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-2594a9-openshift-default-8694ddf5f7-v4jsh" Oct 31 00:48:37.890336 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:37.890149 5222 patch_prober.go:29] interesting pod/e2e-shared-b-w6-2594a9-openshift-default-8694ddf5f7-v4jsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Oct 31 00:48:37.890336 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:37.890213 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-2594a9-openshift-default-8694ddf5f7-v4jsh" podUID="4c57fa36-80bb-403d-97a3-b18601c386aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Oct 31 00:48:38.889514 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:38.889479 5222 patch_prober.go:29] interesting pod/e2e-shared-b-w6-2594a9-openshift-default-8694ddf5f7-v4jsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Oct 31 00:48:38.889934 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:38.889549 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-2594a9-openshift-default-8694ddf5f7-v4jsh" podUID="4c57fa36-80bb-403d-97a3-b18601c386aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Oct 31 00:48:39.889330 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:39.889290 5222 patch_prober.go:29] interesting pod/e2e-shared-b-w6-2594a9-openshift-default-8694ddf5f7-v4jsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Oct 31 00:48:39.889552 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:39.889358 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-2594a9-openshift-default-8694ddf5f7-v4jsh" podUID="4c57fa36-80bb-403d-97a3-b18601c386aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Oct 31 00:48:40.889951 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:40.889912 5222 patch_prober.go:29] interesting pod/e2e-shared-b-w6-2594a9-openshift-default-8694ddf5f7-v4jsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Oct 31 00:48:40.890428 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:40.889979 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-2594a9-openshift-default-8694ddf5f7-v4jsh" podUID="4c57fa36-80bb-403d-97a3-b18601c386aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Oct 31 00:48:41.889837 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:41.889804 5222 patch_prober.go:29] interesting pod/e2e-shared-b-w6-2594a9-openshift-default-8694ddf5f7-v4jsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Oct 31 00:48:41.890029 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:41.889858 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-2594a9-openshift-default-8694ddf5f7-v4jsh" podUID="4c57fa36-80bb-403d-97a3-b18601c386aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Oct 31 00:48:42.889589 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:42.889553 5222 patch_prober.go:29] interesting pod/e2e-shared-b-w6-2594a9-openshift-default-8694ddf5f7-v4jsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Oct 31 00:48:42.889786 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:42.889608 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-2594a9-openshift-default-8694ddf5f7-v4jsh" podUID="4c57fa36-80bb-403d-97a3-b18601c386aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Oct 31 00:48:43.894091 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:43.894061 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-2594a9-openshift-default-8694ddf5f7-v4jsh" Oct 31 00:48:43.894564 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:43.894492 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-2594a9-openshift-default-8694ddf5f7-v4jsh" Oct 31 00:48:43.895157 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:43.895136 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-2594a9-openshift-default-8694ddf5f7-v4jsh" Oct 31 00:48:56.638837 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:56.638804 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-f32ec2-vbs5g"] Oct 31 00:48:56.639584 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:56.639567 5222 cpu_manager.go:414] "containerMap: removing container" podUID="f45187f1-0814-4c12-bdcb-158eeca665f4" containerName="istio-proxy" Oct 31 00:48:56.639743 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:56.639586 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f45187f1-0814-4c12-bdcb-158eeca665f4" containerName="istio-proxy" Oct 31 00:48:56.639743 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:56.639604 5222 cpu_manager.go:414] "containerMap: removing container" podUID="2c9a1b97-2895-492c-a5ae-97a6db93bd47" containerName="revoke-keys" Oct 31 00:48:56.639743 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:56.639609 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c9a1b97-2895-492c-a5ae-97a6db93bd47" containerName="revoke-keys" Oct 31 00:48:56.639743 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:56.639634 5222 cpu_manager.go:414] "containerMap: removing container" podUID="815b9a5c-72ac-447b-bd5f-bb0df408c5f3" containerName="payload-pre-processing" Oct 31 00:48:56.639743 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:56.639639 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="815b9a5c-72ac-447b-bd5f-bb0df408c5f3" containerName="payload-pre-processing" Oct 31 00:48:56.639743 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:56.639656 5222 cpu_manager.go:414] "containerMap: removing container" podUID="2b11eda2-8622-4e58-ad66-e13985b1dd36" containerName="payload-processing" Oct 31 00:48:56.639743 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:56.639661 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b11eda2-8622-4e58-ad66-e13985b1dd36" containerName="payload-processing" Oct 31 00:48:56.639743 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:56.639671 5222 cpu_manager.go:414] "containerMap: removing container" podUID="2f5b0e39-04be-4933-9512-8c7ff9bdb9cd" containerName="payload-processing" Oct 31 00:48:56.639743 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:56.639676 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f5b0e39-04be-4933-9512-8c7ff9bdb9cd" containerName="payload-processing" Oct 31 00:48:56.639743 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:56.639694 5222 cpu_manager.go:414] "containerMap: removing container" podUID="82e8f109-3468-431f-aabf-ef22d4e59878" containerName="payload-pre-processing" Oct 31 00:48:56.639743 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:56.639698 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82e8f109-3468-431f-aabf-ef22d4e59878" containerName="payload-pre-processing" Oct 31 00:48:56.640167 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:56.639796 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="f45187f1-0814-4c12-bdcb-158eeca665f4" containerName="istio-proxy" Oct 31 00:48:56.640167 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:56.639807 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c9a1b97-2895-492c-a5ae-97a6db93bd47" containerName="revoke-keys" Oct 31 00:48:56.640167 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:56.639816 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="815b9a5c-72ac-447b-bd5f-bb0df408c5f3" containerName="payload-pre-processing" Oct 31 00:48:56.640167 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:56.639824 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b11eda2-8622-4e58-ad66-e13985b1dd36" containerName="payload-processing" Oct 31 00:48:56.640167 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:56.639831 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f5b0e39-04be-4933-9512-8c7ff9bdb9cd" containerName="payload-processing" Oct 31 00:48:56.640167 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:56.639839 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="82e8f109-3468-431f-aabf-ef22d4e59878" containerName="payload-pre-processing" Oct 31 00:48:56.646141 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:56.646124 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-f32ec2-vbs5g" Oct 31 00:48:56.701948 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:56.701909 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-f32ec2-vbs5g"] Oct 31 00:48:56.703657 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:56.703620 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdfnw\" (UniqueName: \"kubernetes.io/projected/a119a257-ffd3-4573-ac30-d8ebee73908a-kube-api-access-zdfnw\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-f32ec2-vbs5g\" (UID: \"a119a257-ffd3-4573-ac30-d8ebee73908a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-f32ec2-vbs5g" Oct 31 00:48:56.703843 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:56.703816 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a119a257-ffd3-4573-ac30-d8ebee73908a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-f32ec2-vbs5g\" (UID: \"a119a257-ffd3-4573-ac30-d8ebee73908a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-f32ec2-vbs5g" Oct 31 00:48:56.704142 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:56.704121 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a119a257-ffd3-4573-ac30-d8ebee73908a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-f32ec2-vbs5g\" (UID: \"a119a257-ffd3-4573-ac30-d8ebee73908a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-f32ec2-vbs5g" Oct 31 00:48:56.748522 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:56.748488 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:48:56.748522 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:56.748508 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:48:56.748760 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:56.748554 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:48:56.750859 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:56.750844 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfz9z\"" Oct 31 00:48:56.805020 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:56.804987 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a119a257-ffd3-4573-ac30-d8ebee73908a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-f32ec2-vbs5g\" (UID: \"a119a257-ffd3-4573-ac30-d8ebee73908a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-f32ec2-vbs5g" Oct 31 00:48:56.805206 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:56.805155 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a119a257-ffd3-4573-ac30-d8ebee73908a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-f32ec2-vbs5g\" (UID: \"a119a257-ffd3-4573-ac30-d8ebee73908a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-f32ec2-vbs5g" Oct 31 00:48:56.805276 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:56.805219 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdfnw\" (UniqueName: \"kubernetes.io/projected/a119a257-ffd3-4573-ac30-d8ebee73908a-kube-api-access-zdfnw\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-f32ec2-vbs5g\" (UID: \"a119a257-ffd3-4573-ac30-d8ebee73908a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-f32ec2-vbs5g" Oct 31 00:48:56.805923 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:56.805895 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a119a257-ffd3-4573-ac30-d8ebee73908a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-f32ec2-vbs5g\" (UID: \"a119a257-ffd3-4573-ac30-d8ebee73908a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-f32ec2-vbs5g" Oct 31 00:48:56.808058 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:56.808035 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a119a257-ffd3-4573-ac30-d8ebee73908a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-f32ec2-vbs5g\" (UID: \"a119a257-ffd3-4573-ac30-d8ebee73908a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-f32ec2-vbs5g" Oct 31 00:48:56.867605 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:56.867573 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdfnw\" (UniqueName: \"kubernetes.io/projected/a119a257-ffd3-4573-ac30-d8ebee73908a-kube-api-access-zdfnw\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-f32ec2-vbs5g\" (UID: \"a119a257-ffd3-4573-ac30-d8ebee73908a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-f32ec2-vbs5g" Oct 31 00:48:56.960901 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:56.960807 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-f32ec2-vbs5g" Oct 31 00:48:57.344165 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:48:57.344123 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda119a257_ffd3_4573_ac30_d8ebee73908a.slice/crio-1238be50c9cd1a9c6ccc5d63fe013e00e584a8bf3c3ca686a24b0192835c9f72 WatchSource:0}: Error finding container 1238be50c9cd1a9c6ccc5d63fe013e00e584a8bf3c3ca686a24b0192835c9f72: Status 404 returned error can't find the container with id 1238be50c9cd1a9c6ccc5d63fe013e00e584a8bf3c3ca686a24b0192835c9f72 Oct 31 00:48:57.346611 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:57.346588 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-f32ec2-vbs5g"] Oct 31 00:48:57.431930 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:57.431892 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-f32ec2-vbs5g" event={"ID":"a119a257-ffd3-4573-ac30-d8ebee73908a","Type":"ContainerStarted","Data":"1238be50c9cd1a9c6ccc5d63fe013e00e584a8bf3c3ca686a24b0192835c9f72"} Oct 31 00:48:58.441853 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:58.441817 5222 generic.go:356] "Generic (PLEG): container finished" podID="a119a257-ffd3-4573-ac30-d8ebee73908a" containerID="c6dfa239dbf3b9a61e5fba7509c5c1050ad1b5b51d04c39111f44f5a75cf7405" exitCode=0 Oct 31 00:48:58.442340 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:58.441910 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-f32ec2-vbs5g" event={"ID":"a119a257-ffd3-4573-ac30-d8ebee73908a","Type":"ContainerDied","Data":"c6dfa239dbf3b9a61e5fba7509c5c1050ad1b5b51d04c39111f44f5a75cf7405"} Oct 31 00:48:59.624721 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:59.624693 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-f32ec2-vbs5g" Oct 31 00:48:59.745585 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:59.745500 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a119a257-ffd3-4573-ac30-d8ebee73908a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a119a257-ffd3-4573-ac30-d8ebee73908a-maas-api-xks-ca\") pod \"a119a257-ffd3-4573-ac30-d8ebee73908a\" (UID: \"a119a257-ffd3-4573-ac30-d8ebee73908a\") " Oct 31 00:48:59.745762 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:59.745628 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a119a257-ffd3-4573-ac30-d8ebee73908a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a119a257-ffd3-4573-ac30-d8ebee73908a-maas-api-openshift-ca\") pod \"a119a257-ffd3-4573-ac30-d8ebee73908a\" (UID: \"a119a257-ffd3-4573-ac30-d8ebee73908a\") " Oct 31 00:48:59.745762 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:59.745670 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a119a257-ffd3-4573-ac30-d8ebee73908a-kube-api-access-zdfnw\" (UniqueName: \"kubernetes.io/projected/a119a257-ffd3-4573-ac30-d8ebee73908a-kube-api-access-zdfnw\") pod \"a119a257-ffd3-4573-ac30-d8ebee73908a\" (UID: \"a119a257-ffd3-4573-ac30-d8ebee73908a\") " Oct 31 00:48:59.746207 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:59.746170 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a119a257-ffd3-4573-ac30-d8ebee73908a-maas-api-openshift-ca" pod "a119a257-ffd3-4573-ac30-d8ebee73908a" (UID: "a119a257-ffd3-4573-ac30-d8ebee73908a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:48:59.747190 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:59.747162 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a119a257-ffd3-4573-ac30-d8ebee73908a-maas-api-openshift-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:59.748225 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:59.748202 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a119a257-ffd3-4573-ac30-d8ebee73908a-maas-api-xks-ca" pod "a119a257-ffd3-4573-ac30-d8ebee73908a" (UID: "a119a257-ffd3-4573-ac30-d8ebee73908a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:48:59.748652 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:59.748630 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a119a257-ffd3-4573-ac30-d8ebee73908a-kube-api-access-zdfnw" pod "a119a257-ffd3-4573-ac30-d8ebee73908a" (UID: "a119a257-ffd3-4573-ac30-d8ebee73908a"). InnerVolumeSpecName "kube-api-access-zdfnw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:48:59.848737 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:59.848704 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a119a257-ffd3-4573-ac30-d8ebee73908a-maas-api-xks-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:59.848737 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:48:59.848733 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdfnw\" (UniqueName: \"kubernetes.io/projected/a119a257-ffd3-4573-ac30-d8ebee73908a-kube-api-access-zdfnw\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:00.459355 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:00.459321 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-f32ec2-vbs5g" Oct 31 00:49:00.459572 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:00.459352 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-f32ec2-vbs5g" event={"ID":"a119a257-ffd3-4573-ac30-d8ebee73908a","Type":"ContainerDied","Data":"1238be50c9cd1a9c6ccc5d63fe013e00e584a8bf3c3ca686a24b0192835c9f72"} Oct 31 00:49:00.459572 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:00.459441 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1238be50c9cd1a9c6ccc5d63fe013e00e584a8bf3c3ca686a24b0192835c9f72" Oct 31 00:49:05.892136 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:49:05.892097 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 00:49:09.322746 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:09.322707 5222 scope.go:122] "RemoveContainer" containerID="c370cc5ff634f8403704b66ecb4f320bcd409cf7fbc2b96d004f878b2ce452a4" Oct 31 00:49:09.339261 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:09.339211 5222 scope.go:122] "RemoveContainer" containerID="aff2ed7c4ce564a2e82fe8b353a3dda60a100554b03193279a855b854c85675a" Oct 31 00:49:09.895397 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:09.894411 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c0f56c-j4m87"] Oct 31 00:49:09.895632 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:09.895573 5222 cpu_manager.go:414] "containerMap: removing container" podUID="a119a257-ffd3-4573-ac30-d8ebee73908a" containerName="revoke-keys" Oct 31 00:49:09.895632 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:09.895594 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a119a257-ffd3-4573-ac30-d8ebee73908a" containerName="revoke-keys" Oct 31 00:49:09.904445 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:09.895782 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="a119a257-ffd3-4573-ac30-d8ebee73908a" containerName="revoke-keys" Oct 31 00:49:09.932490 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:09.908204 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c0f56c-j4m87" Oct 31 00:49:09.932490 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:09.917821 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04c68b33-8c78-44f2-91bc-f957fcfcec73-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c0f56c-j4m87\" (UID: \"04c68b33-8c78-44f2-91bc-f957fcfcec73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c0f56c-j4m87" Oct 31 00:49:09.932490 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:09.918174 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ww5fz\" (UniqueName: \"kubernetes.io/projected/04c68b33-8c78-44f2-91bc-f957fcfcec73-kube-api-access-ww5fz\") pod \"maas-api-revoke-keys-e2e-api-a-c0f56c-j4m87\" (UID: \"04c68b33-8c78-44f2-91bc-f957fcfcec73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c0f56c-j4m87" Oct 31 00:49:09.932490 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:09.918221 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04c68b33-8c78-44f2-91bc-f957fcfcec73-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c0f56c-j4m87\" (UID: \"04c68b33-8c78-44f2-91bc-f957fcfcec73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c0f56c-j4m87" Oct 31 00:49:09.932490 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:09.926495 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c0f56c-j4m87"] Oct 31 00:49:09.991133 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:09.983452 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:49:09.991133 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:09.983512 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:49:09.991133 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:09.980442 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfz9z\"" Oct 31 00:49:10.015879 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:10.014657 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:49:10.042396 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:10.036651 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ww5fz\" (UniqueName: \"kubernetes.io/projected/04c68b33-8c78-44f2-91bc-f957fcfcec73-kube-api-access-ww5fz\") pod \"maas-api-revoke-keys-e2e-api-a-c0f56c-j4m87\" (UID: \"04c68b33-8c78-44f2-91bc-f957fcfcec73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c0f56c-j4m87" Oct 31 00:49:10.042396 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:10.036718 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04c68b33-8c78-44f2-91bc-f957fcfcec73-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c0f56c-j4m87\" (UID: \"04c68b33-8c78-44f2-91bc-f957fcfcec73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c0f56c-j4m87" Oct 31 00:49:10.042396 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:10.036854 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04c68b33-8c78-44f2-91bc-f957fcfcec73-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c0f56c-j4m87\" (UID: \"04c68b33-8c78-44f2-91bc-f957fcfcec73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c0f56c-j4m87" Oct 31 00:49:10.074418 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:10.073768 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04c68b33-8c78-44f2-91bc-f957fcfcec73-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c0f56c-j4m87\" (UID: \"04c68b33-8c78-44f2-91bc-f957fcfcec73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c0f56c-j4m87" Oct 31 00:49:10.080108 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:10.080068 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04c68b33-8c78-44f2-91bc-f957fcfcec73-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c0f56c-j4m87\" (UID: \"04c68b33-8c78-44f2-91bc-f957fcfcec73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c0f56c-j4m87" Oct 31 00:49:10.107939 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:10.107906 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ww5fz\" (UniqueName: \"kubernetes.io/projected/04c68b33-8c78-44f2-91bc-f957fcfcec73-kube-api-access-ww5fz\") pod \"maas-api-revoke-keys-e2e-api-a-c0f56c-j4m87\" (UID: \"04c68b33-8c78-44f2-91bc-f957fcfcec73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c0f56c-j4m87" Oct 31 00:49:10.281247 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:10.281207 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c0f56c-j4m87" Oct 31 00:49:10.692388 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:49:10.683591 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04c68b33_8c78_44f2_91bc_f957fcfcec73.slice/crio-28d98c7c4a1316ca36e49ccecaa2548c229d2014d5b4bb4072472812e96bdba2 WatchSource:0}: Error finding container 28d98c7c4a1316ca36e49ccecaa2548c229d2014d5b4bb4072472812e96bdba2: Status 404 returned error can't find the container with id 28d98c7c4a1316ca36e49ccecaa2548c229d2014d5b4bb4072472812e96bdba2 Oct 31 00:49:10.692388 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:10.688007 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c0f56c-j4m87"] Oct 31 00:49:11.617073 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:11.617037 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c0f56c-j4m87" event={"ID":"04c68b33-8c78-44f2-91bc-f957fcfcec73","Type":"ContainerStarted","Data":"126086a618ae38e81bfd9a69d5f2908a55876013eaa7adff275f8a656cdfe091"} Oct 31 00:49:11.617228 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:11.617101 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c0f56c-j4m87" event={"ID":"04c68b33-8c78-44f2-91bc-f957fcfcec73","Type":"ContainerStarted","Data":"28d98c7c4a1316ca36e49ccecaa2548c229d2014d5b4bb4072472812e96bdba2"} Oct 31 00:49:11.735280 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:11.735222 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c0f56c-j4m87" podStartSLOduration=2.73520701 podStartE2EDuration="2.73520701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:49:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:49:11.73457971 +0000 UTC m=+2044.205749462" watchObservedRunningTime="2026-10-31 00:49:11.73520701 +0000 UTC m=+2044.206376808" Oct 31 00:49:12.627713 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:12.627672 5222 generic.go:356] "Generic (PLEG): container finished" podID="04c68b33-8c78-44f2-91bc-f957fcfcec73" containerID="126086a618ae38e81bfd9a69d5f2908a55876013eaa7adff275f8a656cdfe091" exitCode=0 Oct 31 00:49:12.627901 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:12.627706 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c0f56c-j4m87" event={"ID":"04c68b33-8c78-44f2-91bc-f957fcfcec73","Type":"ContainerDied","Data":"126086a618ae38e81bfd9a69d5f2908a55876013eaa7adff275f8a656cdfe091"} Oct 31 00:49:13.818143 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:13.818116 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c0f56c-j4m87" Oct 31 00:49:13.897896 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:13.897821 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04c68b33-8c78-44f2-91bc-f957fcfcec73-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04c68b33-8c78-44f2-91bc-f957fcfcec73-maas-api-openshift-ca\") pod \"04c68b33-8c78-44f2-91bc-f957fcfcec73\" (UID: \"04c68b33-8c78-44f2-91bc-f957fcfcec73\") " Oct 31 00:49:13.898036 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:13.897912 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04c68b33-8c78-44f2-91bc-f957fcfcec73-kube-api-access-ww5fz\" (UniqueName: \"kubernetes.io/projected/04c68b33-8c78-44f2-91bc-f957fcfcec73-kube-api-access-ww5fz\") pod \"04c68b33-8c78-44f2-91bc-f957fcfcec73\" (UID: \"04c68b33-8c78-44f2-91bc-f957fcfcec73\") " Oct 31 00:49:13.898036 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:13.897949 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04c68b33-8c78-44f2-91bc-f957fcfcec73-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04c68b33-8c78-44f2-91bc-f957fcfcec73-maas-api-xks-ca\") pod \"04c68b33-8c78-44f2-91bc-f957fcfcec73\" (UID: \"04c68b33-8c78-44f2-91bc-f957fcfcec73\") " Oct 31 00:49:13.898265 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:13.898235 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04c68b33-8c78-44f2-91bc-f957fcfcec73-maas-api-openshift-ca" pod "04c68b33-8c78-44f2-91bc-f957fcfcec73" (UID: "04c68b33-8c78-44f2-91bc-f957fcfcec73"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:49:13.898554 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:13.898534 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04c68b33-8c78-44f2-91bc-f957fcfcec73-maas-api-openshift-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:13.900210 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:13.900185 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04c68b33-8c78-44f2-91bc-f957fcfcec73-maas-api-xks-ca" pod "04c68b33-8c78-44f2-91bc-f957fcfcec73" (UID: "04c68b33-8c78-44f2-91bc-f957fcfcec73"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:49:13.900287 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:13.900250 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04c68b33-8c78-44f2-91bc-f957fcfcec73-kube-api-access-ww5fz" pod "04c68b33-8c78-44f2-91bc-f957fcfcec73" (UID: "04c68b33-8c78-44f2-91bc-f957fcfcec73"). InnerVolumeSpecName "kube-api-access-ww5fz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:49:13.999986 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:13.999943 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ww5fz\" (UniqueName: \"kubernetes.io/projected/04c68b33-8c78-44f2-91bc-f957fcfcec73-kube-api-access-ww5fz\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:13.999986 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:13.999979 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04c68b33-8c78-44f2-91bc-f957fcfcec73-maas-api-xks-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:14.643618 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:14.643589 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c0f56c-j4m87" Oct 31 00:49:14.643816 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:14.643629 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c0f56c-j4m87" event={"ID":"04c68b33-8c78-44f2-91bc-f957fcfcec73","Type":"ContainerDied","Data":"28d98c7c4a1316ca36e49ccecaa2548c229d2014d5b4bb4072472812e96bdba2"} Oct 31 00:49:14.643816 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:14.643682 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="28d98c7c4a1316ca36e49ccecaa2548c229d2014d5b4bb4072472812e96bdba2" Oct 31 00:49:16.662848 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:16.662814 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-e030e0-j5qhc"] Oct 31 00:49:16.663560 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:16.663541 5222 cpu_manager.go:414] "containerMap: removing container" podUID="04c68b33-8c78-44f2-91bc-f957fcfcec73" containerName="revoke-keys" Oct 31 00:49:16.663560 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:16.663560 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04c68b33-8c78-44f2-91bc-f957fcfcec73" containerName="revoke-keys" Oct 31 00:49:16.663711 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:16.663702 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="04c68b33-8c78-44f2-91bc-f957fcfcec73" containerName="revoke-keys" Oct 31 00:49:16.667697 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:16.667678 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-e030e0-j5qhc" Oct 31 00:49:16.721059 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:16.721021 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-e030e0-j5qhc"] Oct 31 00:49:16.735739 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:16.735697 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f0919d7-e8d7-4f73-88ac-278d2203d1d4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-e030e0-j5qhc\" (UID: \"9f0919d7-e8d7-4f73-88ac-278d2203d1d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-e030e0-j5qhc" Oct 31 00:49:16.736001 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:16.735976 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gq7qd\" (UniqueName: \"kubernetes.io/projected/9f0919d7-e8d7-4f73-88ac-278d2203d1d4-kube-api-access-gq7qd\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-e030e0-j5qhc\" (UID: \"9f0919d7-e8d7-4f73-88ac-278d2203d1d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-e030e0-j5qhc" Oct 31 00:49:16.736065 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:16.736015 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f0919d7-e8d7-4f73-88ac-278d2203d1d4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-e030e0-j5qhc\" (UID: \"9f0919d7-e8d7-4f73-88ac-278d2203d1d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-e030e0-j5qhc" Oct 31 00:49:16.767355 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:16.767321 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:49:16.767355 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:16.767363 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:49:16.767652 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:16.767362 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:49:16.767652 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:16.767461 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfz9z\"" Oct 31 00:49:16.837362 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:16.837331 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq7qd\" (UniqueName: \"kubernetes.io/projected/9f0919d7-e8d7-4f73-88ac-278d2203d1d4-kube-api-access-gq7qd\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-e030e0-j5qhc\" (UID: \"9f0919d7-e8d7-4f73-88ac-278d2203d1d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-e030e0-j5qhc" Oct 31 00:49:16.837576 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:16.837401 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f0919d7-e8d7-4f73-88ac-278d2203d1d4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-e030e0-j5qhc\" (UID: \"9f0919d7-e8d7-4f73-88ac-278d2203d1d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-e030e0-j5qhc" Oct 31 00:49:16.837576 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:16.837561 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f0919d7-e8d7-4f73-88ac-278d2203d1d4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-e030e0-j5qhc\" (UID: \"9f0919d7-e8d7-4f73-88ac-278d2203d1d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-e030e0-j5qhc" Oct 31 00:49:16.838097 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:16.838069 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f0919d7-e8d7-4f73-88ac-278d2203d1d4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-e030e0-j5qhc\" (UID: \"9f0919d7-e8d7-4f73-88ac-278d2203d1d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-e030e0-j5qhc" Oct 31 00:49:16.840147 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:16.840126 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f0919d7-e8d7-4f73-88ac-278d2203d1d4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-e030e0-j5qhc\" (UID: \"9f0919d7-e8d7-4f73-88ac-278d2203d1d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-e030e0-j5qhc" Oct 31 00:49:16.900285 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:16.900245 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gq7qd\" (UniqueName: \"kubernetes.io/projected/9f0919d7-e8d7-4f73-88ac-278d2203d1d4-kube-api-access-gq7qd\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-e030e0-j5qhc\" (UID: \"9f0919d7-e8d7-4f73-88ac-278d2203d1d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-e030e0-j5qhc" Oct 31 00:49:16.983093 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:16.982990 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-e030e0-j5qhc" Oct 31 00:49:17.365808 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:49:17.365776 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f0919d7_e8d7_4f73_88ac_278d2203d1d4.slice/crio-a5199efb81455540b4e5e0c9b0a67de8cd3c06d57c8a95a245f4ded25c6a0085 WatchSource:0}: Error finding container a5199efb81455540b4e5e0c9b0a67de8cd3c06d57c8a95a245f4ded25c6a0085: Status 404 returned error can't find the container with id a5199efb81455540b4e5e0c9b0a67de8cd3c06d57c8a95a245f4ded25c6a0085 Oct 31 00:49:17.369013 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:17.368986 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-e030e0-j5qhc"] Oct 31 00:49:17.667982 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:17.667950 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-e030e0-j5qhc" event={"ID":"9f0919d7-e8d7-4f73-88ac-278d2203d1d4","Type":"ContainerStarted","Data":"a5199efb81455540b4e5e0c9b0a67de8cd3c06d57c8a95a245f4ded25c6a0085"} Oct 31 00:49:18.677477 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:18.677362 5222 generic.go:356] "Generic (PLEG): container finished" podID="9f0919d7-e8d7-4f73-88ac-278d2203d1d4" containerID="066c39a7d74a193b214d15154a3d2a8f1abf5c269245146d331a9cf30c43c50b" exitCode=0 Oct 31 00:49:18.677876 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:18.677398 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-e030e0-j5qhc" event={"ID":"9f0919d7-e8d7-4f73-88ac-278d2203d1d4","Type":"ContainerDied","Data":"066c39a7d74a193b214d15154a3d2a8f1abf5c269245146d331a9cf30c43c50b"} Oct 31 00:49:19.910625 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:19.910595 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-e030e0-j5qhc" Oct 31 00:49:19.987096 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:19.987012 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f0919d7-e8d7-4f73-88ac-278d2203d1d4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f0919d7-e8d7-4f73-88ac-278d2203d1d4-maas-api-xks-ca\") pod \"9f0919d7-e8d7-4f73-88ac-278d2203d1d4\" (UID: \"9f0919d7-e8d7-4f73-88ac-278d2203d1d4\") " Oct 31 00:49:19.987239 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:19.987160 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f0919d7-e8d7-4f73-88ac-278d2203d1d4-kube-api-access-gq7qd\" (UniqueName: \"kubernetes.io/projected/9f0919d7-e8d7-4f73-88ac-278d2203d1d4-kube-api-access-gq7qd\") pod \"9f0919d7-e8d7-4f73-88ac-278d2203d1d4\" (UID: \"9f0919d7-e8d7-4f73-88ac-278d2203d1d4\") " Oct 31 00:49:19.987239 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:19.987212 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f0919d7-e8d7-4f73-88ac-278d2203d1d4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f0919d7-e8d7-4f73-88ac-278d2203d1d4-maas-api-openshift-ca\") pod \"9f0919d7-e8d7-4f73-88ac-278d2203d1d4\" (UID: \"9f0919d7-e8d7-4f73-88ac-278d2203d1d4\") " Oct 31 00:49:19.987643 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:19.987619 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f0919d7-e8d7-4f73-88ac-278d2203d1d4-maas-api-openshift-ca" pod "9f0919d7-e8d7-4f73-88ac-278d2203d1d4" (UID: "9f0919d7-e8d7-4f73-88ac-278d2203d1d4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:49:19.989216 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:19.989193 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f0919d7-e8d7-4f73-88ac-278d2203d1d4-maas-api-xks-ca" pod "9f0919d7-e8d7-4f73-88ac-278d2203d1d4" (UID: "9f0919d7-e8d7-4f73-88ac-278d2203d1d4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:49:19.989523 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:19.989498 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f0919d7-e8d7-4f73-88ac-278d2203d1d4-kube-api-access-gq7qd" pod "9f0919d7-e8d7-4f73-88ac-278d2203d1d4" (UID: "9f0919d7-e8d7-4f73-88ac-278d2203d1d4"). InnerVolumeSpecName "kube-api-access-gq7qd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:49:20.089049 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:20.089009 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f0919d7-e8d7-4f73-88ac-278d2203d1d4-maas-api-openshift-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:20.089049 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:20.089042 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f0919d7-e8d7-4f73-88ac-278d2203d1d4-maas-api-xks-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:20.089049 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:20.089055 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gq7qd\" (UniqueName: \"kubernetes.io/projected/9f0919d7-e8d7-4f73-88ac-278d2203d1d4-kube-api-access-gq7qd\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:20.698911 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:20.698871 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-e030e0-j5qhc" event={"ID":"9f0919d7-e8d7-4f73-88ac-278d2203d1d4","Type":"ContainerDied","Data":"a5199efb81455540b4e5e0c9b0a67de8cd3c06d57c8a95a245f4ded25c6a0085"} Oct 31 00:49:20.698911 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:20.698887 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-e030e0-j5qhc" Oct 31 00:49:20.699205 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:20.698936 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a5199efb81455540b4e5e0c9b0a67de8cd3c06d57c8a95a245f4ded25c6a0085" Oct 31 00:49:21.208906 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:21.208872 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c0f56c-679fccf4f-wqpxd"] Oct 31 00:49:21.310433 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:21.310362 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c0f56c-6bbdcff4c-dswhj"] Oct 31 00:49:26.214608 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:26.214556 5222 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-c0f56c-679fccf4f-wqpxd" podUID="24572644-6def-4e05-bc4b-3ad70a7ca70d" containerName="payload-processing" containerID="cri-o://69293cfe73f8cab07b9b7df9fe15c8611230e494942f7ffc599c6d9d17e91002" gracePeriod=25 Oct 31 00:49:26.311254 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:26.311210 5222 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c0f56c-6bbdcff4c-dswhj" podUID="445117f4-90c9-4084-9847-30d6e03e1d2e" containerName="payload-pre-processing" containerID="cri-o://c6d945ad8eb46cbe9a1244b07388903932fa13f5ce2839db3cd58918d1402c0a" gracePeriod=25 Oct 31 00:49:26.634060 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:26.634029 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c0f56c-6bbdcff4c-dswhj" Oct 31 00:49:26.675704 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:26.675679 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c0f56c-679fccf4f-wqpxd" Oct 31 00:49:26.730736 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:26.730701 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/445117f4-90c9-4084-9847-30d6e03e1d2e-kube-api-access-x7hbk\" (UniqueName: \"kubernetes.io/projected/445117f4-90c9-4084-9847-30d6e03e1d2e-kube-api-access-x7hbk\") pod \"445117f4-90c9-4084-9847-30d6e03e1d2e\" (UID: \"445117f4-90c9-4084-9847-30d6e03e1d2e\") " Oct 31 00:49:26.730888 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:26.730759 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24572644-6def-4e05-bc4b-3ad70a7ca70d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24572644-6def-4e05-bc4b-3ad70a7ca70d-plugins-config-volume\") pod \"24572644-6def-4e05-bc4b-3ad70a7ca70d\" (UID: \"24572644-6def-4e05-bc4b-3ad70a7ca70d\") " Oct 31 00:49:26.730970 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:26.730954 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/445117f4-90c9-4084-9847-30d6e03e1d2e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/445117f4-90c9-4084-9847-30d6e03e1d2e-plugins-config-volume\") pod \"445117f4-90c9-4084-9847-30d6e03e1d2e\" (UID: \"445117f4-90c9-4084-9847-30d6e03e1d2e\") " Oct 31 00:49:26.731022 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:26.730984 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24572644-6def-4e05-bc4b-3ad70a7ca70d-kube-api-access-frx5d\" (UniqueName: \"kubernetes.io/projected/24572644-6def-4e05-bc4b-3ad70a7ca70d-kube-api-access-frx5d\") pod \"24572644-6def-4e05-bc4b-3ad70a7ca70d\" (UID: \"24572644-6def-4e05-bc4b-3ad70a7ca70d\") " Oct 31 00:49:26.731269 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:26.731166 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24572644-6def-4e05-bc4b-3ad70a7ca70d-plugins-config-volume" pod "24572644-6def-4e05-bc4b-3ad70a7ca70d" (UID: "24572644-6def-4e05-bc4b-3ad70a7ca70d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:49:26.731403 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:26.731360 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/445117f4-90c9-4084-9847-30d6e03e1d2e-plugins-config-volume" pod "445117f4-90c9-4084-9847-30d6e03e1d2e" (UID: "445117f4-90c9-4084-9847-30d6e03e1d2e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:49:26.731726 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:26.731706 5222 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/445117f4-90c9-4084-9847-30d6e03e1d2e-plugins-config-volume\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:26.731791 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:26.731731 5222 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24572644-6def-4e05-bc4b-3ad70a7ca70d-plugins-config-volume\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:26.733126 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:26.733098 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/445117f4-90c9-4084-9847-30d6e03e1d2e-kube-api-access-x7hbk" pod "445117f4-90c9-4084-9847-30d6e03e1d2e" (UID: "445117f4-90c9-4084-9847-30d6e03e1d2e"). InnerVolumeSpecName "kube-api-access-x7hbk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:49:26.733218 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:26.733172 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24572644-6def-4e05-bc4b-3ad70a7ca70d-kube-api-access-frx5d" pod "24572644-6def-4e05-bc4b-3ad70a7ca70d" (UID: "24572644-6def-4e05-bc4b-3ad70a7ca70d"). InnerVolumeSpecName "kube-api-access-frx5d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:49:26.797442 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:26.796611 5222 generic.go:356] "Generic (PLEG): container finished" podID="24572644-6def-4e05-bc4b-3ad70a7ca70d" containerID="69293cfe73f8cab07b9b7df9fe15c8611230e494942f7ffc599c6d9d17e91002" exitCode=0 Oct 31 00:49:26.797442 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:26.796702 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c0f56c-679fccf4f-wqpxd" Oct 31 00:49:26.797442 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:26.796780 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c0f56c-679fccf4f-wqpxd" event={"ID":"24572644-6def-4e05-bc4b-3ad70a7ca70d","Type":"ContainerDied","Data":"69293cfe73f8cab07b9b7df9fe15c8611230e494942f7ffc599c6d9d17e91002"} Oct 31 00:49:26.797442 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:26.796833 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c0f56c-679fccf4f-wqpxd" event={"ID":"24572644-6def-4e05-bc4b-3ad70a7ca70d","Type":"ContainerDied","Data":"f07ec7bbf5ee3bb687dd289df4a7647929c8abccfc07cdb4bd719336a3311962"} Oct 31 00:49:26.797442 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:26.796862 5222 scope.go:122] "RemoveContainer" containerID="69293cfe73f8cab07b9b7df9fe15c8611230e494942f7ffc599c6d9d17e91002" Oct 31 00:49:26.799624 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:26.799594 5222 generic.go:356] "Generic (PLEG): container finished" podID="445117f4-90c9-4084-9847-30d6e03e1d2e" containerID="c6d945ad8eb46cbe9a1244b07388903932fa13f5ce2839db3cd58918d1402c0a" exitCode=0 Oct 31 00:49:26.799727 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:26.799655 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c0f56c-6bbdcff4c-dswhj" Oct 31 00:49:26.799727 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:26.799669 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c0f56c-6bbdcff4c-dswhj" event={"ID":"445117f4-90c9-4084-9847-30d6e03e1d2e","Type":"ContainerDied","Data":"c6d945ad8eb46cbe9a1244b07388903932fa13f5ce2839db3cd58918d1402c0a"} Oct 31 00:49:26.799727 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:26.799718 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c0f56c-6bbdcff4c-dswhj" event={"ID":"445117f4-90c9-4084-9847-30d6e03e1d2e","Type":"ContainerDied","Data":"dcc71deb89eb7fa36bae7195b51f29adaa815d628942a76a5030273178dc220d"} Oct 31 00:49:26.808980 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:26.808950 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d58cd3-rzvp7"] Oct 31 00:49:26.810698 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:26.810671 5222 scope.go:122] "RemoveContainer" containerID="69293cfe73f8cab07b9b7df9fe15c8611230e494942f7ffc599c6d9d17e91002" Oct 31 00:49:26.811006 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:49:26.810987 5222 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69293cfe73f8cab07b9b7df9fe15c8611230e494942f7ffc599c6d9d17e91002\": container with ID starting with 69293cfe73f8cab07b9b7df9fe15c8611230e494942f7ffc599c6d9d17e91002 not found: ID does not exist" containerID="69293cfe73f8cab07b9b7df9fe15c8611230e494942f7ffc599c6d9d17e91002" Oct 31 00:49:26.811071 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:26.811013 5222 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69293cfe73f8cab07b9b7df9fe15c8611230e494942f7ffc599c6d9d17e91002"} err="failed to get container status \"69293cfe73f8cab07b9b7df9fe15c8611230e494942f7ffc599c6d9d17e91002\": rpc error: code = NotFound desc = could not find container \"69293cfe73f8cab07b9b7df9fe15c8611230e494942f7ffc599c6d9d17e91002\": container with ID starting with 69293cfe73f8cab07b9b7df9fe15c8611230e494942f7ffc599c6d9d17e91002 not found: ID does not exist" Oct 31 00:49:26.811071 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:26.811034 5222 scope.go:122] "RemoveContainer" containerID="c6d945ad8eb46cbe9a1244b07388903932fa13f5ce2839db3cd58918d1402c0a" Oct 31 00:49:26.812962 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:26.812938 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d58cd3-rzvp7"] Oct 31 00:49:26.821104 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:26.821085 5222 scope.go:122] "RemoveContainer" containerID="c6d945ad8eb46cbe9a1244b07388903932fa13f5ce2839db3cd58918d1402c0a" Oct 31 00:49:26.821468 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:49:26.821442 5222 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6d945ad8eb46cbe9a1244b07388903932fa13f5ce2839db3cd58918d1402c0a\": container with ID starting with c6d945ad8eb46cbe9a1244b07388903932fa13f5ce2839db3cd58918d1402c0a not found: ID does not exist" containerID="c6d945ad8eb46cbe9a1244b07388903932fa13f5ce2839db3cd58918d1402c0a" Oct 31 00:49:26.821564 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:26.821474 5222 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6d945ad8eb46cbe9a1244b07388903932fa13f5ce2839db3cd58918d1402c0a"} err="failed to get container status \"c6d945ad8eb46cbe9a1244b07388903932fa13f5ce2839db3cd58918d1402c0a\": rpc error: code = NotFound desc = could not find container \"c6d945ad8eb46cbe9a1244b07388903932fa13f5ce2839db3cd58918d1402c0a\": container with ID starting with c6d945ad8eb46cbe9a1244b07388903932fa13f5ce2839db3cd58918d1402c0a not found: ID does not exist" Oct 31 00:49:26.833598 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:26.833564 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frx5d\" (UniqueName: \"kubernetes.io/projected/24572644-6def-4e05-bc4b-3ad70a7ca70d-kube-api-access-frx5d\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:26.833598 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:26.833597 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7hbk\" (UniqueName: \"kubernetes.io/projected/445117f4-90c9-4084-9847-30d6e03e1d2e-kube-api-access-x7hbk\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:26.987781 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:26.987752 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c0f56c-6bbdcff4c-dswhj"] Oct 31 00:49:26.991596 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:26.991569 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c0f56c-6bbdcff4c-dswhj"] Oct 31 00:49:27.104467 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:27.104356 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c0f56c-679fccf4f-wqpxd"] Oct 31 00:49:27.107997 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:27.107965 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c0f56c-679fccf4f-wqpxd"] Oct 31 00:49:27.659713 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:27.659674 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24572644-6def-4e05-bc4b-3ad70a7ca70d" path="/var/lib/kubelet/pods/24572644-6def-4e05-bc4b-3ad70a7ca70d/volumes" Oct 31 00:49:27.660124 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:27.660110 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="445117f4-90c9-4084-9847-30d6e03e1d2e" path="/var/lib/kubelet/pods/445117f4-90c9-4084-9847-30d6e03e1d2e/volumes" Oct 31 00:49:27.660571 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:27.660554 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca2d3367-eb7f-4c59-bbcf-2981f54a3806" path="/var/lib/kubelet/pods/ca2d3367-eb7f-4c59-bbcf-2981f54a3806/volumes" Oct 31 00:49:28.266639 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:28.266595 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c0f56c-j4m87"] Oct 31 00:49:28.317977 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:28.317934 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c0f56c-j4m87"] Oct 31 00:49:28.321308 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:28.321282 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-e030e0-57f56bb655-wfdmp"] Oct 31 00:49:28.424585 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:28.424409 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-e030e0-5686f5c779-sx2sx"] Oct 31 00:49:29.659335 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:29.659289 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04c68b33-8c78-44f2-91bc-f957fcfcec73" path="/var/lib/kubelet/pods/04c68b33-8c78-44f2-91bc-f957fcfcec73/volumes" Oct 31 00:49:33.321816 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:33.321772 5222 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-e030e0-57f56bb655-wfdmp" podUID="c6fa3226-46ba-4b95-a4c4-bdba6073277c" containerName="payload-processing" containerID="cri-o://724027af61a844b561486b6eedbc57fac3ec9035ac2f0a27d03227eddbb1fb51" gracePeriod=25 Oct 31 00:49:33.425261 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:33.425199 5222 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-e030e0-5686f5c779-sx2sx" podUID="60ee69ce-043d-4529-b9be-7a36cd7468ca" containerName="payload-pre-processing" containerID="cri-o://0289592cd71f72ab252d00b52aca1bcb06f9d0c043f2672bb14f21f81581b17a" gracePeriod=25 Oct 31 00:49:33.619066 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:33.619034 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-e030e0-57f56bb655-wfdmp" Oct 31 00:49:33.671972 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:33.671930 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6fa3226-46ba-4b95-a4c4-bdba6073277c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6fa3226-46ba-4b95-a4c4-bdba6073277c-plugins-config-volume\") pod \"c6fa3226-46ba-4b95-a4c4-bdba6073277c\" (UID: \"c6fa3226-46ba-4b95-a4c4-bdba6073277c\") " Oct 31 00:49:33.672169 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:33.672066 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6fa3226-46ba-4b95-a4c4-bdba6073277c-kube-api-access-prtg2\" (UniqueName: \"kubernetes.io/projected/c6fa3226-46ba-4b95-a4c4-bdba6073277c-kube-api-access-prtg2\") pod \"c6fa3226-46ba-4b95-a4c4-bdba6073277c\" (UID: \"c6fa3226-46ba-4b95-a4c4-bdba6073277c\") " Oct 31 00:49:33.673921 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:33.673887 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6fa3226-46ba-4b95-a4c4-bdba6073277c-plugins-config-volume" pod "c6fa3226-46ba-4b95-a4c4-bdba6073277c" (UID: "c6fa3226-46ba-4b95-a4c4-bdba6073277c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:49:33.676651 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:33.676618 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6fa3226-46ba-4b95-a4c4-bdba6073277c-kube-api-access-prtg2" pod "c6fa3226-46ba-4b95-a4c4-bdba6073277c" (UID: "c6fa3226-46ba-4b95-a4c4-bdba6073277c"). InnerVolumeSpecName "kube-api-access-prtg2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:49:33.725099 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:33.725070 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-e030e0-5686f5c779-sx2sx" Oct 31 00:49:33.730587 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:33.730557 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-d58cd3-openshift-default-d9c46776f-vklx7"] Oct 31 00:49:33.730822 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:33.730797 5222 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-d58cd3-openshift-default-d9c46776f-vklx7" podUID="608e314b-554b-40a2-90f7-fbdefd2b27b6" containerName="istio-proxy" containerID="cri-o://ed21fb6353a1a845beb2ff9ed88d4338be5a28ac3df2de4357a2f56e0b70f15a" gracePeriod=30 Oct 31 00:49:33.773687 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:33.773650 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60ee69ce-043d-4529-b9be-7a36cd7468ca-kube-api-access-gmfks\" (UniqueName: \"kubernetes.io/projected/60ee69ce-043d-4529-b9be-7a36cd7468ca-kube-api-access-gmfks\") pod \"60ee69ce-043d-4529-b9be-7a36cd7468ca\" (UID: \"60ee69ce-043d-4529-b9be-7a36cd7468ca\") " Oct 31 00:49:33.773896 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:33.773757 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60ee69ce-043d-4529-b9be-7a36cd7468ca-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60ee69ce-043d-4529-b9be-7a36cd7468ca-plugins-config-volume\") pod \"60ee69ce-043d-4529-b9be-7a36cd7468ca\" (UID: \"60ee69ce-043d-4529-b9be-7a36cd7468ca\") " Oct 31 00:49:33.774331 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:33.774296 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60ee69ce-043d-4529-b9be-7a36cd7468ca-plugins-config-volume" pod "60ee69ce-043d-4529-b9be-7a36cd7468ca" (UID: "60ee69ce-043d-4529-b9be-7a36cd7468ca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:49:33.774704 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:33.774674 5222 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60ee69ce-043d-4529-b9be-7a36cd7468ca-plugins-config-volume\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:33.774834 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:33.774708 5222 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6fa3226-46ba-4b95-a4c4-bdba6073277c-plugins-config-volume\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:33.774834 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:33.774724 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prtg2\" (UniqueName: \"kubernetes.io/projected/c6fa3226-46ba-4b95-a4c4-bdba6073277c-kube-api-access-prtg2\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:33.776214 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:33.776189 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60ee69ce-043d-4529-b9be-7a36cd7468ca-kube-api-access-gmfks" pod "60ee69ce-043d-4529-b9be-7a36cd7468ca" (UID: "60ee69ce-043d-4529-b9be-7a36cd7468ca"). InnerVolumeSpecName "kube-api-access-gmfks". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:49:33.880484 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:33.879609 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmfks\" (UniqueName: \"kubernetes.io/projected/60ee69ce-043d-4529-b9be-7a36cd7468ca-kube-api-access-gmfks\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:33.953135 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:33.953100 5222 generic.go:356] "Generic (PLEG): container finished" podID="60ee69ce-043d-4529-b9be-7a36cd7468ca" containerID="0289592cd71f72ab252d00b52aca1bcb06f9d0c043f2672bb14f21f81581b17a" exitCode=0 Oct 31 00:49:33.953490 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:33.953465 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-e030e0-5686f5c779-sx2sx" event={"ID":"60ee69ce-043d-4529-b9be-7a36cd7468ca","Type":"ContainerDied","Data":"0289592cd71f72ab252d00b52aca1bcb06f9d0c043f2672bb14f21f81581b17a"} Oct 31 00:49:33.953660 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:33.953644 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-e030e0-5686f5c779-sx2sx" event={"ID":"60ee69ce-043d-4529-b9be-7a36cd7468ca","Type":"ContainerDied","Data":"db72535b020a8b2e6fe8639a7bd0cd9a5829d593fa9c5350b86f8514423e3049"} Oct 31 00:49:33.953776 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:33.953765 5222 scope.go:122] "RemoveContainer" containerID="0289592cd71f72ab252d00b52aca1bcb06f9d0c043f2672bb14f21f81581b17a" Oct 31 00:49:33.954014 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:33.953999 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-e030e0-5686f5c779-sx2sx" Oct 31 00:49:33.970823 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:33.970787 5222 generic.go:356] "Generic (PLEG): container finished" podID="c6fa3226-46ba-4b95-a4c4-bdba6073277c" containerID="724027af61a844b561486b6eedbc57fac3ec9035ac2f0a27d03227eddbb1fb51" exitCode=0 Oct 31 00:49:33.970997 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:33.970858 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-e030e0-57f56bb655-wfdmp" event={"ID":"c6fa3226-46ba-4b95-a4c4-bdba6073277c","Type":"ContainerDied","Data":"724027af61a844b561486b6eedbc57fac3ec9035ac2f0a27d03227eddbb1fb51"} Oct 31 00:49:33.970997 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:33.970910 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-e030e0-57f56bb655-wfdmp" event={"ID":"c6fa3226-46ba-4b95-a4c4-bdba6073277c","Type":"ContainerDied","Data":"0f0078002388c1704b31f05dca9e07049eec892873c7a05288b4f1b33a6d5807"} Oct 31 00:49:33.970997 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:33.970943 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-e030e0-57f56bb655-wfdmp" Oct 31 00:49:33.989428 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:33.989407 5222 scope.go:122] "RemoveContainer" containerID="0289592cd71f72ab252d00b52aca1bcb06f9d0c043f2672bb14f21f81581b17a" Oct 31 00:49:33.989752 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:49:33.989726 5222 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0289592cd71f72ab252d00b52aca1bcb06f9d0c043f2672bb14f21f81581b17a\": container with ID starting with 0289592cd71f72ab252d00b52aca1bcb06f9d0c043f2672bb14f21f81581b17a not found: ID does not exist" containerID="0289592cd71f72ab252d00b52aca1bcb06f9d0c043f2672bb14f21f81581b17a" Oct 31 00:49:33.989807 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:33.989762 5222 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0289592cd71f72ab252d00b52aca1bcb06f9d0c043f2672bb14f21f81581b17a"} err="failed to get container status \"0289592cd71f72ab252d00b52aca1bcb06f9d0c043f2672bb14f21f81581b17a\": rpc error: code = NotFound desc = could not find container \"0289592cd71f72ab252d00b52aca1bcb06f9d0c043f2672bb14f21f81581b17a\": container with ID starting with 0289592cd71f72ab252d00b52aca1bcb06f9d0c043f2672bb14f21f81581b17a not found: ID does not exist" Oct 31 00:49:33.989807 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:33.989781 5222 scope.go:122] "RemoveContainer" containerID="724027af61a844b561486b6eedbc57fac3ec9035ac2f0a27d03227eddbb1fb51" Oct 31 00:49:34.000492 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:34.000472 5222 scope.go:122] "RemoveContainer" containerID="724027af61a844b561486b6eedbc57fac3ec9035ac2f0a27d03227eddbb1fb51" Oct 31 00:49:34.000836 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:49:34.000817 5222 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"724027af61a844b561486b6eedbc57fac3ec9035ac2f0a27d03227eddbb1fb51\": container with ID starting with 724027af61a844b561486b6eedbc57fac3ec9035ac2f0a27d03227eddbb1fb51 not found: ID does not exist" containerID="724027af61a844b561486b6eedbc57fac3ec9035ac2f0a27d03227eddbb1fb51" Oct 31 00:49:34.000886 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:34.000845 5222 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"724027af61a844b561486b6eedbc57fac3ec9035ac2f0a27d03227eddbb1fb51"} err="failed to get container status \"724027af61a844b561486b6eedbc57fac3ec9035ac2f0a27d03227eddbb1fb51\": rpc error: code = NotFound desc = could not find container \"724027af61a844b561486b6eedbc57fac3ec9035ac2f0a27d03227eddbb1fb51\": container with ID starting with 724027af61a844b561486b6eedbc57fac3ec9035ac2f0a27d03227eddbb1fb51 not found: ID does not exist" Oct 31 00:49:34.212333 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:34.212251 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-e030e0-5686f5c779-sx2sx"] Oct 31 00:49:34.222622 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:34.222579 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-e030e0-5686f5c779-sx2sx"] Oct 31 00:49:34.334231 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:34.334180 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-e030e0-57f56bb655-wfdmp"] Oct 31 00:49:34.338189 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:34.338164 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-e030e0-57f56bb655-wfdmp"] Oct 31 00:49:34.418393 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:34.417499 5222 patch_prober.go:29] interesting pod/e2e-shared-a-w2-d58cd3-openshift-default-d9c46776f-vklx7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:49:34.418393 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:34.417591 5222 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-d58cd3-openshift-default-d9c46776f-vklx7" podUID="608e314b-554b-40a2-90f7-fbdefd2b27b6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:49:34.621199 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:34.621160 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-dcdbd0-4bv7s"] Oct 31 00:49:34.621987 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:34.621963 5222 cpu_manager.go:414] "containerMap: removing container" podUID="445117f4-90c9-4084-9847-30d6e03e1d2e" containerName="payload-pre-processing" Oct 31 00:49:34.621987 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:34.621983 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="445117f4-90c9-4084-9847-30d6e03e1d2e" containerName="payload-pre-processing" Oct 31 00:49:34.622161 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:34.622006 5222 cpu_manager.go:414] "containerMap: removing container" podUID="60ee69ce-043d-4529-b9be-7a36cd7468ca" containerName="payload-pre-processing" Oct 31 00:49:34.622161 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:34.622011 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60ee69ce-043d-4529-b9be-7a36cd7468ca" containerName="payload-pre-processing" Oct 31 00:49:34.622161 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:34.622020 5222 cpu_manager.go:414] "containerMap: removing container" podUID="c6fa3226-46ba-4b95-a4c4-bdba6073277c" containerName="payload-processing" Oct 31 00:49:34.622161 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:34.622027 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6fa3226-46ba-4b95-a4c4-bdba6073277c" containerName="payload-processing" Oct 31 00:49:34.622161 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:34.622055 5222 cpu_manager.go:414] "containerMap: removing container" podUID="9f0919d7-e8d7-4f73-88ac-278d2203d1d4" containerName="revoke-keys" Oct 31 00:49:34.622161 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:34.622059 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f0919d7-e8d7-4f73-88ac-278d2203d1d4" containerName="revoke-keys" Oct 31 00:49:34.622161 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:34.622088 5222 cpu_manager.go:414] "containerMap: removing container" podUID="24572644-6def-4e05-bc4b-3ad70a7ca70d" containerName="payload-processing" Oct 31 00:49:34.622161 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:34.622092 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24572644-6def-4e05-bc4b-3ad70a7ca70d" containerName="payload-processing" Oct 31 00:49:34.622544 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:34.622187 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="445117f4-90c9-4084-9847-30d6e03e1d2e" containerName="payload-pre-processing" Oct 31 00:49:34.622544 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:34.622196 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="60ee69ce-043d-4529-b9be-7a36cd7468ca" containerName="payload-pre-processing" Oct 31 00:49:34.622544 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:34.622202 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6fa3226-46ba-4b95-a4c4-bdba6073277c" containerName="payload-processing" Oct 31 00:49:34.622544 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:34.622212 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f0919d7-e8d7-4f73-88ac-278d2203d1d4" containerName="revoke-keys" Oct 31 00:49:34.622544 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:34.622222 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="24572644-6def-4e05-bc4b-3ad70a7ca70d" containerName="payload-processing" Oct 31 00:49:34.627399 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:34.627357 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-dcdbd0-4bv7s" Oct 31 00:49:34.679424 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:34.679390 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-dcdbd0-4bv7s"] Oct 31 00:49:34.681518 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:34.681490 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfz9z\"" Oct 31 00:49:34.681689 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:34.681496 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:49:34.681689 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:34.681566 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:49:34.681689 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:34.681611 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:49:34.702316 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:34.702272 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c1b1ee2-7349-41c4-9fde-3bbf2aa4a8d8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-dcdbd0-4bv7s\" (UID: \"5c1b1ee2-7349-41c4-9fde-3bbf2aa4a8d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-dcdbd0-4bv7s" Oct 31 00:49:34.702547 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:34.702450 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bktc\" (UniqueName: \"kubernetes.io/projected/5c1b1ee2-7349-41c4-9fde-3bbf2aa4a8d8-kube-api-access-6bktc\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-dcdbd0-4bv7s\" (UID: \"5c1b1ee2-7349-41c4-9fde-3bbf2aa4a8d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-dcdbd0-4bv7s" Oct 31 00:49:34.702732 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:34.702713 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c1b1ee2-7349-41c4-9fde-3bbf2aa4a8d8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-dcdbd0-4bv7s\" (UID: \"5c1b1ee2-7349-41c4-9fde-3bbf2aa4a8d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-dcdbd0-4bv7s" Oct 31 00:49:34.784901 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:34.784860 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-79tk8"] Oct 31 00:49:34.804123 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:34.804081 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c1b1ee2-7349-41c4-9fde-3bbf2aa4a8d8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-dcdbd0-4bv7s\" (UID: \"5c1b1ee2-7349-41c4-9fde-3bbf2aa4a8d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-dcdbd0-4bv7s" Oct 31 00:49:34.804316 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:34.804243 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c1b1ee2-7349-41c4-9fde-3bbf2aa4a8d8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-dcdbd0-4bv7s\" (UID: \"5c1b1ee2-7349-41c4-9fde-3bbf2aa4a8d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-dcdbd0-4bv7s" Oct 31 00:49:34.804316 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:34.804297 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bktc\" (UniqueName: \"kubernetes.io/projected/5c1b1ee2-7349-41c4-9fde-3bbf2aa4a8d8-kube-api-access-6bktc\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-dcdbd0-4bv7s\" (UID: \"5c1b1ee2-7349-41c4-9fde-3bbf2aa4a8d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-dcdbd0-4bv7s" Oct 31 00:49:34.805074 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:34.805052 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c1b1ee2-7349-41c4-9fde-3bbf2aa4a8d8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-dcdbd0-4bv7s\" (UID: \"5c1b1ee2-7349-41c4-9fde-3bbf2aa4a8d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-dcdbd0-4bv7s" Oct 31 00:49:34.807267 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:34.807244 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c1b1ee2-7349-41c4-9fde-3bbf2aa4a8d8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-dcdbd0-4bv7s\" (UID: \"5c1b1ee2-7349-41c4-9fde-3bbf2aa4a8d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-dcdbd0-4bv7s" Oct 31 00:49:34.867029 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:34.866994 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bktc\" (UniqueName: \"kubernetes.io/projected/5c1b1ee2-7349-41c4-9fde-3bbf2aa4a8d8-kube-api-access-6bktc\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-dcdbd0-4bv7s\" (UID: \"5c1b1ee2-7349-41c4-9fde-3bbf2aa4a8d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-dcdbd0-4bv7s" Oct 31 00:49:34.942819 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:34.942734 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-dcdbd0-4bv7s" Oct 31 00:49:35.324506 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:49:35.324467 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c1b1ee2_7349_41c4_9fde_3bbf2aa4a8d8.slice/crio-5da385dda0f381aabdabb4e1a4883213db6247e92288702871f2a2d1bf3c4879 WatchSource:0}: Error finding container 5da385dda0f381aabdabb4e1a4883213db6247e92288702871f2a2d1bf3c4879: Status 404 returned error can't find the container with id 5da385dda0f381aabdabb4e1a4883213db6247e92288702871f2a2d1bf3c4879 Oct 31 00:49:35.327424 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:35.327397 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-dcdbd0-4bv7s"] Oct 31 00:49:35.659697 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:35.659660 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60ee69ce-043d-4529-b9be-7a36cd7468ca" path="/var/lib/kubelet/pods/60ee69ce-043d-4529-b9be-7a36cd7468ca/volumes" Oct 31 00:49:35.660200 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:35.660181 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6fa3226-46ba-4b95-a4c4-bdba6073277c" path="/var/lib/kubelet/pods/c6fa3226-46ba-4b95-a4c4-bdba6073277c/volumes" Oct 31 00:49:35.965440 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:49:35.964978 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 00:49:35.996393 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:35.995137 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-dcdbd0-4bv7s" event={"ID":"5c1b1ee2-7349-41c4-9fde-3bbf2aa4a8d8","Type":"ContainerStarted","Data":"2338a6eb69d1b44faecf36f9529877bf6e608849ea8187d55d0efae85bedd3b8"} Oct 31 00:49:35.996393 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:35.995218 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-dcdbd0-4bv7s" event={"ID":"5c1b1ee2-7349-41c4-9fde-3bbf2aa4a8d8","Type":"ContainerStarted","Data":"5da385dda0f381aabdabb4e1a4883213db6247e92288702871f2a2d1bf3c4879"} Oct 31 00:49:36.118183 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:36.118120 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-dcdbd0-4bv7s" podStartSLOduration=2.118103762 podStartE2EDuration="2.118103762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:49:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:49:36.114492529 +0000 UTC m=+2068.585662281" watchObservedRunningTime="2026-10-31 00:49:36.118103762 +0000 UTC m=+2068.589273510" Oct 31 00:49:37.007678 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:37.007638 5222 generic.go:356] "Generic (PLEG): container finished" podID="5c1b1ee2-7349-41c4-9fde-3bbf2aa4a8d8" containerID="2338a6eb69d1b44faecf36f9529877bf6e608849ea8187d55d0efae85bedd3b8" exitCode=0 Oct 31 00:49:37.008125 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:37.007727 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-dcdbd0-4bv7s" event={"ID":"5c1b1ee2-7349-41c4-9fde-3bbf2aa4a8d8","Type":"ContainerDied","Data":"2338a6eb69d1b44faecf36f9529877bf6e608849ea8187d55d0efae85bedd3b8"} Oct 31 00:49:38.602203 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:38.602175 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-dcdbd0-4bv7s" Oct 31 00:49:38.631331 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:38.631291 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c1b1ee2-7349-41c4-9fde-3bbf2aa4a8d8-kube-api-access-6bktc\" (UniqueName: \"kubernetes.io/projected/5c1b1ee2-7349-41c4-9fde-3bbf2aa4a8d8-kube-api-access-6bktc\") pod \"5c1b1ee2-7349-41c4-9fde-3bbf2aa4a8d8\" (UID: \"5c1b1ee2-7349-41c4-9fde-3bbf2aa4a8d8\") " Oct 31 00:49:38.631526 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:38.631356 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c1b1ee2-7349-41c4-9fde-3bbf2aa4a8d8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c1b1ee2-7349-41c4-9fde-3bbf2aa4a8d8-maas-api-openshift-ca\") pod \"5c1b1ee2-7349-41c4-9fde-3bbf2aa4a8d8\" (UID: \"5c1b1ee2-7349-41c4-9fde-3bbf2aa4a8d8\") " Oct 31 00:49:38.631526 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:38.631489 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c1b1ee2-7349-41c4-9fde-3bbf2aa4a8d8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c1b1ee2-7349-41c4-9fde-3bbf2aa4a8d8-maas-api-xks-ca\") pod \"5c1b1ee2-7349-41c4-9fde-3bbf2aa4a8d8\" (UID: \"5c1b1ee2-7349-41c4-9fde-3bbf2aa4a8d8\") " Oct 31 00:49:38.631767 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:38.631737 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c1b1ee2-7349-41c4-9fde-3bbf2aa4a8d8-maas-api-openshift-ca" pod "5c1b1ee2-7349-41c4-9fde-3bbf2aa4a8d8" (UID: "5c1b1ee2-7349-41c4-9fde-3bbf2aa4a8d8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:49:38.632125 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:38.632094 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c1b1ee2-7349-41c4-9fde-3bbf2aa4a8d8-maas-api-openshift-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:38.633733 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:38.633700 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c1b1ee2-7349-41c4-9fde-3bbf2aa4a8d8-maas-api-xks-ca" pod "5c1b1ee2-7349-41c4-9fde-3bbf2aa4a8d8" (UID: "5c1b1ee2-7349-41c4-9fde-3bbf2aa4a8d8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:49:38.633824 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:38.633793 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c1b1ee2-7349-41c4-9fde-3bbf2aa4a8d8-kube-api-access-6bktc" pod "5c1b1ee2-7349-41c4-9fde-3bbf2aa4a8d8" (UID: "5c1b1ee2-7349-41c4-9fde-3bbf2aa4a8d8"). InnerVolumeSpecName "kube-api-access-6bktc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:49:38.735409 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:38.735341 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bktc\" (UniqueName: \"kubernetes.io/projected/5c1b1ee2-7349-41c4-9fde-3bbf2aa4a8d8-kube-api-access-6bktc\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:38.735634 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:38.735610 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c1b1ee2-7349-41c4-9fde-3bbf2aa4a8d8-maas-api-xks-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:38.825839 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:38.825752 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-4ab8e5-openshift-default-6c89ffbf45-mtckf"] Oct 31 00:49:38.827284 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:38.827263 5222 cpu_manager.go:414] "containerMap: removing container" podUID="5c1b1ee2-7349-41c4-9fde-3bbf2aa4a8d8" containerName="revoke-keys" Oct 31 00:49:38.828136 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:38.827968 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c1b1ee2-7349-41c4-9fde-3bbf2aa4a8d8" containerName="revoke-keys" Oct 31 00:49:38.828241 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:38.828223 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c1b1ee2-7349-41c4-9fde-3bbf2aa4a8d8" containerName="revoke-keys" Oct 31 00:49:38.832703 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:38.832683 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4ab8e5-openshift-default-6c89ffbf45-mtckf" Oct 31 00:49:38.971403 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:38.923896 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-4ab8e5-openshift-default-dockercfg-p6sgb\"" Oct 31 00:49:38.971403 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:38.939708 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c46a2765-b445-4cd6-9b3a-1aef0a68b895-workload-socket\") pod \"e2e-inf-a-4ab8e5-openshift-default-6c89ffbf45-mtckf\" (UID: \"c46a2765-b445-4cd6-9b3a-1aef0a68b895\") " pod="openshift-ingress/e2e-inf-a-4ab8e5-openshift-default-6c89ffbf45-mtckf" Oct 31 00:49:38.971403 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:38.939810 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c46a2765-b445-4cd6-9b3a-1aef0a68b895-istiod-ca-cert\") pod \"e2e-inf-a-4ab8e5-openshift-default-6c89ffbf45-mtckf\" (UID: \"c46a2765-b445-4cd6-9b3a-1aef0a68b895\") " pod="openshift-ingress/e2e-inf-a-4ab8e5-openshift-default-6c89ffbf45-mtckf" Oct 31 00:49:38.971403 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:38.939847 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c46a2765-b445-4cd6-9b3a-1aef0a68b895-istio-token\") pod \"e2e-inf-a-4ab8e5-openshift-default-6c89ffbf45-mtckf\" (UID: \"c46a2765-b445-4cd6-9b3a-1aef0a68b895\") " pod="openshift-ingress/e2e-inf-a-4ab8e5-openshift-default-6c89ffbf45-mtckf" Oct 31 00:49:38.971403 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:38.939886 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c46a2765-b445-4cd6-9b3a-1aef0a68b895-istio-podinfo\") pod \"e2e-inf-a-4ab8e5-openshift-default-6c89ffbf45-mtckf\" (UID: \"c46a2765-b445-4cd6-9b3a-1aef0a68b895\") " pod="openshift-ingress/e2e-inf-a-4ab8e5-openshift-default-6c89ffbf45-mtckf" Oct 31 00:49:38.971403 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:38.939927 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c46a2765-b445-4cd6-9b3a-1aef0a68b895-istio-envoy\") pod \"e2e-inf-a-4ab8e5-openshift-default-6c89ffbf45-mtckf\" (UID: \"c46a2765-b445-4cd6-9b3a-1aef0a68b895\") " pod="openshift-ingress/e2e-inf-a-4ab8e5-openshift-default-6c89ffbf45-mtckf" Oct 31 00:49:38.971403 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:38.940032 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c46a2765-b445-4cd6-9b3a-1aef0a68b895-credential-socket\") pod \"e2e-inf-a-4ab8e5-openshift-default-6c89ffbf45-mtckf\" (UID: \"c46a2765-b445-4cd6-9b3a-1aef0a68b895\") " pod="openshift-ingress/e2e-inf-a-4ab8e5-openshift-default-6c89ffbf45-mtckf" Oct 31 00:49:38.971403 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:38.940075 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c46a2765-b445-4cd6-9b3a-1aef0a68b895-workload-certs\") pod \"e2e-inf-a-4ab8e5-openshift-default-6c89ffbf45-mtckf\" (UID: \"c46a2765-b445-4cd6-9b3a-1aef0a68b895\") " pod="openshift-ingress/e2e-inf-a-4ab8e5-openshift-default-6c89ffbf45-mtckf" Oct 31 00:49:38.971403 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:38.940149 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dzcm\" (UniqueName: \"kubernetes.io/projected/c46a2765-b445-4cd6-9b3a-1aef0a68b895-kube-api-access-6dzcm\") pod \"e2e-inf-a-4ab8e5-openshift-default-6c89ffbf45-mtckf\" (UID: \"c46a2765-b445-4cd6-9b3a-1aef0a68b895\") " pod="openshift-ingress/e2e-inf-a-4ab8e5-openshift-default-6c89ffbf45-mtckf" Oct 31 00:49:38.971403 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:38.940309 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c46a2765-b445-4cd6-9b3a-1aef0a68b895-istio-data\") pod \"e2e-inf-a-4ab8e5-openshift-default-6c89ffbf45-mtckf\" (UID: \"c46a2765-b445-4cd6-9b3a-1aef0a68b895\") " pod="openshift-ingress/e2e-inf-a-4ab8e5-openshift-default-6c89ffbf45-mtckf" Oct 31 00:49:39.013403 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:38.996308 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-4ab8e5-openshift-default-6c89ffbf45-mtckf"] Oct 31 00:49:39.081445 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:39.058450 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c46a2765-b445-4cd6-9b3a-1aef0a68b895-workload-socket\") pod \"e2e-inf-a-4ab8e5-openshift-default-6c89ffbf45-mtckf\" (UID: \"c46a2765-b445-4cd6-9b3a-1aef0a68b895\") " pod="openshift-ingress/e2e-inf-a-4ab8e5-openshift-default-6c89ffbf45-mtckf" Oct 31 00:49:39.081864 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:39.081824 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c46a2765-b445-4cd6-9b3a-1aef0a68b895-istiod-ca-cert\") pod \"e2e-inf-a-4ab8e5-openshift-default-6c89ffbf45-mtckf\" (UID: \"c46a2765-b445-4cd6-9b3a-1aef0a68b895\") " pod="openshift-ingress/e2e-inf-a-4ab8e5-openshift-default-6c89ffbf45-mtckf" Oct 31 00:49:39.082057 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:39.082037 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c46a2765-b445-4cd6-9b3a-1aef0a68b895-istio-token\") pod \"e2e-inf-a-4ab8e5-openshift-default-6c89ffbf45-mtckf\" (UID: \"c46a2765-b445-4cd6-9b3a-1aef0a68b895\") " pod="openshift-ingress/e2e-inf-a-4ab8e5-openshift-default-6c89ffbf45-mtckf" Oct 31 00:49:39.163770 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:39.163724 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c46a2765-b445-4cd6-9b3a-1aef0a68b895-istio-podinfo\") pod \"e2e-inf-a-4ab8e5-openshift-default-6c89ffbf45-mtckf\" (UID: \"c46a2765-b445-4cd6-9b3a-1aef0a68b895\") " pod="openshift-ingress/e2e-inf-a-4ab8e5-openshift-default-6c89ffbf45-mtckf" Oct 31 00:49:39.164068 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:39.164041 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c46a2765-b445-4cd6-9b3a-1aef0a68b895-istio-envoy\") pod \"e2e-inf-a-4ab8e5-openshift-default-6c89ffbf45-mtckf\" (UID: \"c46a2765-b445-4cd6-9b3a-1aef0a68b895\") " pod="openshift-ingress/e2e-inf-a-4ab8e5-openshift-default-6c89ffbf45-mtckf" Oct 31 00:49:39.164297 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:39.164282 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c46a2765-b445-4cd6-9b3a-1aef0a68b895-credential-socket\") pod \"e2e-inf-a-4ab8e5-openshift-default-6c89ffbf45-mtckf\" (UID: \"c46a2765-b445-4cd6-9b3a-1aef0a68b895\") " pod="openshift-ingress/e2e-inf-a-4ab8e5-openshift-default-6c89ffbf45-mtckf" Oct 31 00:49:39.164473 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:39.164454 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c46a2765-b445-4cd6-9b3a-1aef0a68b895-workload-certs\") pod \"e2e-inf-a-4ab8e5-openshift-default-6c89ffbf45-mtckf\" (UID: \"c46a2765-b445-4cd6-9b3a-1aef0a68b895\") " pod="openshift-ingress/e2e-inf-a-4ab8e5-openshift-default-6c89ffbf45-mtckf" Oct 31 00:49:39.176684 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:39.176643 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dzcm\" (UniqueName: \"kubernetes.io/projected/c46a2765-b445-4cd6-9b3a-1aef0a68b895-kube-api-access-6dzcm\") pod \"e2e-inf-a-4ab8e5-openshift-default-6c89ffbf45-mtckf\" (UID: \"c46a2765-b445-4cd6-9b3a-1aef0a68b895\") " pod="openshift-ingress/e2e-inf-a-4ab8e5-openshift-default-6c89ffbf45-mtckf" Oct 31 00:49:39.177130 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:39.177097 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c46a2765-b445-4cd6-9b3a-1aef0a68b895-istio-data\") pod \"e2e-inf-a-4ab8e5-openshift-default-6c89ffbf45-mtckf\" (UID: \"c46a2765-b445-4cd6-9b3a-1aef0a68b895\") " pod="openshift-ingress/e2e-inf-a-4ab8e5-openshift-default-6c89ffbf45-mtckf" Oct 31 00:49:39.250117 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:39.083576 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c46a2765-b445-4cd6-9b3a-1aef0a68b895-istiod-ca-cert\") pod \"e2e-inf-a-4ab8e5-openshift-default-6c89ffbf45-mtckf\" (UID: \"c46a2765-b445-4cd6-9b3a-1aef0a68b895\") " pod="openshift-ingress/e2e-inf-a-4ab8e5-openshift-default-6c89ffbf45-mtckf" Oct 31 00:49:39.252329 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:39.195106 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c46a2765-b445-4cd6-9b3a-1aef0a68b895-credential-socket\") pod \"e2e-inf-a-4ab8e5-openshift-default-6c89ffbf45-mtckf\" (UID: \"c46a2765-b445-4cd6-9b3a-1aef0a68b895\") " pod="openshift-ingress/e2e-inf-a-4ab8e5-openshift-default-6c89ffbf45-mtckf" Oct 31 00:49:39.252574 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:39.083864 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c46a2765-b445-4cd6-9b3a-1aef0a68b895-workload-socket\") pod \"e2e-inf-a-4ab8e5-openshift-default-6c89ffbf45-mtckf\" (UID: \"c46a2765-b445-4cd6-9b3a-1aef0a68b895\") " pod="openshift-ingress/e2e-inf-a-4ab8e5-openshift-default-6c89ffbf45-mtckf" Oct 31 00:49:39.252701 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:39.238079 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c46a2765-b445-4cd6-9b3a-1aef0a68b895-workload-certs\") pod \"e2e-inf-a-4ab8e5-openshift-default-6c89ffbf45-mtckf\" (UID: \"c46a2765-b445-4cd6-9b3a-1aef0a68b895\") " pod="openshift-ingress/e2e-inf-a-4ab8e5-openshift-default-6c89ffbf45-mtckf" Oct 31 00:49:39.252789 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:39.238689 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c46a2765-b445-4cd6-9b3a-1aef0a68b895-istio-data\") pod \"e2e-inf-a-4ab8e5-openshift-default-6c89ffbf45-mtckf\" (UID: \"c46a2765-b445-4cd6-9b3a-1aef0a68b895\") " pod="openshift-ingress/e2e-inf-a-4ab8e5-openshift-default-6c89ffbf45-mtckf" Oct 31 00:49:39.379329 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:39.379188 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c46a2765-b445-4cd6-9b3a-1aef0a68b895-istio-envoy\") pod \"e2e-inf-a-4ab8e5-openshift-default-6c89ffbf45-mtckf\" (UID: \"c46a2765-b445-4cd6-9b3a-1aef0a68b895\") " pod="openshift-ingress/e2e-inf-a-4ab8e5-openshift-default-6c89ffbf45-mtckf" Oct 31 00:49:39.381490 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:39.381448 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dzcm\" (UniqueName: \"kubernetes.io/projected/c46a2765-b445-4cd6-9b3a-1aef0a68b895-kube-api-access-6dzcm\") pod \"e2e-inf-a-4ab8e5-openshift-default-6c89ffbf45-mtckf\" (UID: \"c46a2765-b445-4cd6-9b3a-1aef0a68b895\") " pod="openshift-ingress/e2e-inf-a-4ab8e5-openshift-default-6c89ffbf45-mtckf" Oct 31 00:49:39.395647 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:39.395609 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c46a2765-b445-4cd6-9b3a-1aef0a68b895-istio-podinfo\") pod \"e2e-inf-a-4ab8e5-openshift-default-6c89ffbf45-mtckf\" (UID: \"c46a2765-b445-4cd6-9b3a-1aef0a68b895\") " pod="openshift-ingress/e2e-inf-a-4ab8e5-openshift-default-6c89ffbf45-mtckf" Oct 31 00:49:39.397086 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:39.397042 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c46a2765-b445-4cd6-9b3a-1aef0a68b895-istio-token\") pod \"e2e-inf-a-4ab8e5-openshift-default-6c89ffbf45-mtckf\" (UID: \"c46a2765-b445-4cd6-9b3a-1aef0a68b895\") " pod="openshift-ingress/e2e-inf-a-4ab8e5-openshift-default-6c89ffbf45-mtckf" Oct 31 00:49:39.402107 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:39.402074 5222 generic.go:356] "Generic (PLEG): container finished" podID="608e314b-554b-40a2-90f7-fbdefd2b27b6" containerID="ed21fb6353a1a845beb2ff9ed88d4338be5a28ac3df2de4357a2f56e0b70f15a" exitCode=0 Oct 31 00:49:39.402329 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:39.402311 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-d58cd3-openshift-default-d9c46776f-vklx7" event={"ID":"608e314b-554b-40a2-90f7-fbdefd2b27b6","Type":"ContainerDied","Data":"ed21fb6353a1a845beb2ff9ed88d4338be5a28ac3df2de4357a2f56e0b70f15a"} Oct 31 00:49:39.402431 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:39.402386 5222 scope.go:122] "RemoveContainer" containerID="ff6a9379b42cb98396f10d68f9835af0c5dbe25bc8a4554dd8bcfeb16eb7bbee" Oct 31 00:49:39.407427 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:39.406889 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-dcdbd0-4bv7s" event={"ID":"5c1b1ee2-7349-41c4-9fde-3bbf2aa4a8d8","Type":"ContainerDied","Data":"5da385dda0f381aabdabb4e1a4883213db6247e92288702871f2a2d1bf3c4879"} Oct 31 00:49:39.407427 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:39.406959 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5da385dda0f381aabdabb4e1a4883213db6247e92288702871f2a2d1bf3c4879" Oct 31 00:49:39.407427 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:39.407038 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-dcdbd0-4bv7s" Oct 31 00:49:39.578718 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:39.578685 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4ab8e5-openshift-default-6c89ffbf45-mtckf" Oct 31 00:49:39.931312 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:39.931282 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-d58cd3-openshift-default-d9c46776f-vklx7" Oct 31 00:49:39.979655 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:39.979623 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/608e314b-554b-40a2-90f7-fbdefd2b27b6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/608e314b-554b-40a2-90f7-fbdefd2b27b6-istio-data\") pod \"608e314b-554b-40a2-90f7-fbdefd2b27b6\" (UID: \"608e314b-554b-40a2-90f7-fbdefd2b27b6\") " Oct 31 00:49:39.979841 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:39.979749 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/608e314b-554b-40a2-90f7-fbdefd2b27b6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/608e314b-554b-40a2-90f7-fbdefd2b27b6-istio-podinfo\") pod \"608e314b-554b-40a2-90f7-fbdefd2b27b6\" (UID: \"608e314b-554b-40a2-90f7-fbdefd2b27b6\") " Oct 31 00:49:39.979841 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:39.979836 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/608e314b-554b-40a2-90f7-fbdefd2b27b6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/608e314b-554b-40a2-90f7-fbdefd2b27b6-credential-socket\") pod \"608e314b-554b-40a2-90f7-fbdefd2b27b6\" (UID: \"608e314b-554b-40a2-90f7-fbdefd2b27b6\") " Oct 31 00:49:39.980000 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:39.979864 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/608e314b-554b-40a2-90f7-fbdefd2b27b6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/608e314b-554b-40a2-90f7-fbdefd2b27b6-workload-socket\") pod \"608e314b-554b-40a2-90f7-fbdefd2b27b6\" (UID: \"608e314b-554b-40a2-90f7-fbdefd2b27b6\") " Oct 31 00:49:39.980000 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:39.979926 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/608e314b-554b-40a2-90f7-fbdefd2b27b6-istio-token\" (UniqueName: \"kubernetes.io/projected/608e314b-554b-40a2-90f7-fbdefd2b27b6-istio-token\") pod \"608e314b-554b-40a2-90f7-fbdefd2b27b6\" (UID: \"608e314b-554b-40a2-90f7-fbdefd2b27b6\") " Oct 31 00:49:39.980000 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:39.979971 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/608e314b-554b-40a2-90f7-fbdefd2b27b6-kube-api-access-nwhns\" (UniqueName: \"kubernetes.io/projected/608e314b-554b-40a2-90f7-fbdefd2b27b6-kube-api-access-nwhns\") pod \"608e314b-554b-40a2-90f7-fbdefd2b27b6\" (UID: \"608e314b-554b-40a2-90f7-fbdefd2b27b6\") " Oct 31 00:49:39.980000 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:39.979992 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/608e314b-554b-40a2-90f7-fbdefd2b27b6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/608e314b-554b-40a2-90f7-fbdefd2b27b6-istiod-ca-cert\") pod \"608e314b-554b-40a2-90f7-fbdefd2b27b6\" (UID: \"608e314b-554b-40a2-90f7-fbdefd2b27b6\") " Oct 31 00:49:39.980223 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:39.980018 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/608e314b-554b-40a2-90f7-fbdefd2b27b6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/608e314b-554b-40a2-90f7-fbdefd2b27b6-workload-certs\") pod \"608e314b-554b-40a2-90f7-fbdefd2b27b6\" (UID: \"608e314b-554b-40a2-90f7-fbdefd2b27b6\") " Oct 31 00:49:39.980223 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:39.980046 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/608e314b-554b-40a2-90f7-fbdefd2b27b6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/608e314b-554b-40a2-90f7-fbdefd2b27b6-istio-envoy\") pod \"608e314b-554b-40a2-90f7-fbdefd2b27b6\" (UID: \"608e314b-554b-40a2-90f7-fbdefd2b27b6\") " Oct 31 00:49:39.980932 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:39.980907 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/608e314b-554b-40a2-90f7-fbdefd2b27b6-workload-socket" pod "608e314b-554b-40a2-90f7-fbdefd2b27b6" (UID: "608e314b-554b-40a2-90f7-fbdefd2b27b6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:49:39.981106 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:39.981089 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/608e314b-554b-40a2-90f7-fbdefd2b27b6-credential-socket" pod "608e314b-554b-40a2-90f7-fbdefd2b27b6" (UID: "608e314b-554b-40a2-90f7-fbdefd2b27b6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:49:39.981241 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:39.981224 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/608e314b-554b-40a2-90f7-fbdefd2b27b6-istio-data" pod "608e314b-554b-40a2-90f7-fbdefd2b27b6" (UID: "608e314b-554b-40a2-90f7-fbdefd2b27b6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:49:39.981981 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:39.981935 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/608e314b-554b-40a2-90f7-fbdefd2b27b6-istiod-ca-cert" pod "608e314b-554b-40a2-90f7-fbdefd2b27b6" (UID: "608e314b-554b-40a2-90f7-fbdefd2b27b6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:49:39.982200 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:39.982097 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/608e314b-554b-40a2-90f7-fbdefd2b27b6-workload-certs" pod "608e314b-554b-40a2-90f7-fbdefd2b27b6" (UID: "608e314b-554b-40a2-90f7-fbdefd2b27b6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:49:39.983418 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:39.983395 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/608e314b-554b-40a2-90f7-fbdefd2b27b6-istio-envoy" pod "608e314b-554b-40a2-90f7-fbdefd2b27b6" (UID: "608e314b-554b-40a2-90f7-fbdefd2b27b6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:49:39.983624 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:39.983598 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/608e314b-554b-40a2-90f7-fbdefd2b27b6-istio-podinfo" pod "608e314b-554b-40a2-90f7-fbdefd2b27b6" (UID: "608e314b-554b-40a2-90f7-fbdefd2b27b6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:49:39.984061 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:39.984032 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/608e314b-554b-40a2-90f7-fbdefd2b27b6-istio-token" pod "608e314b-554b-40a2-90f7-fbdefd2b27b6" (UID: "608e314b-554b-40a2-90f7-fbdefd2b27b6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:49:39.986304 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:39.986268 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/608e314b-554b-40a2-90f7-fbdefd2b27b6-kube-api-access-nwhns" pod "608e314b-554b-40a2-90f7-fbdefd2b27b6" (UID: "608e314b-554b-40a2-90f7-fbdefd2b27b6"). InnerVolumeSpecName "kube-api-access-nwhns". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:49:40.082188 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:40.082089 5222 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/608e314b-554b-40a2-90f7-fbdefd2b27b6-credential-socket\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:40.082188 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:40.082121 5222 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/608e314b-554b-40a2-90f7-fbdefd2b27b6-workload-socket\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:40.082188 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:40.082132 5222 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/608e314b-554b-40a2-90f7-fbdefd2b27b6-istio-token\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:40.082188 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:40.082141 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwhns\" (UniqueName: \"kubernetes.io/projected/608e314b-554b-40a2-90f7-fbdefd2b27b6-kube-api-access-nwhns\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:40.082188 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:40.082151 5222 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/608e314b-554b-40a2-90f7-fbdefd2b27b6-istiod-ca-cert\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:40.082188 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:40.082159 5222 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/608e314b-554b-40a2-90f7-fbdefd2b27b6-workload-certs\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:40.082188 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:40.082167 5222 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/608e314b-554b-40a2-90f7-fbdefd2b27b6-istio-envoy\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:40.082188 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:40.082175 5222 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/608e314b-554b-40a2-90f7-fbdefd2b27b6-istio-data\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:40.082188 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:40.082184 5222 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/608e314b-554b-40a2-90f7-fbdefd2b27b6-istio-podinfo\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:40.353895 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:40.353701 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f891e6-2d2pq"] Oct 31 00:49:40.372965 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:40.372883 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f891e6-2d2pq"] Oct 31 00:49:40.429156 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:40.429090 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:49:40.429326 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:40.429238 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:49:40.429326 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:40.429312 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:49:40.432361 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:40.432259 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-4ab8e5-openshift-default-6c89ffbf45-mtckf"] Oct 31 00:49:40.434345 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:40.434311 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4ab8e5-openshift-default-6c89ffbf45-mtckf" event={"ID":"c46a2765-b445-4cd6-9b3a-1aef0a68b895","Type":"ContainerStarted","Data":"c143921677f0e227a7738500706e8b2bba156f895a540c22426ce5d0921242de"} Oct 31 00:49:40.471339 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:40.471296 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-d58cd3-openshift-default-d9c46776f-vklx7" Oct 31 00:49:40.471677 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:40.471345 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-d58cd3-openshift-default-d9c46776f-vklx7" event={"ID":"608e314b-554b-40a2-90f7-fbdefd2b27b6","Type":"ContainerDied","Data":"cbc38540d09d92baa1f9a0da46b3893782114a87d23edf07f41a54ead8ce6e3b"} Oct 31 00:49:40.471758 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:40.471744 5222 scope.go:122] "RemoveContainer" containerID="ed21fb6353a1a845beb2ff9ed88d4338be5a28ac3df2de4357a2f56e0b70f15a" Oct 31 00:49:40.671653 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:40.671448 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-d58cd3-openshift-default-d9c46776f-vklx7"] Oct 31 00:49:40.676979 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:40.676951 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-d58cd3-openshift-default-d9c46776f-vklx7"] Oct 31 00:49:41.499684 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:41.499636 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4ab8e5-openshift-default-6c89ffbf45-mtckf" event={"ID":"c46a2765-b445-4cd6-9b3a-1aef0a68b895","Type":"ContainerStarted","Data":"ffd419175db9ab219d8aa6bb9ebd361568b3e73b48ccb04f0f9d7ff01734c8bd"} Oct 31 00:49:41.571225 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:41.571191 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7d5f4de7-npdwh"] Oct 31 00:49:41.576419 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:41.576363 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7d5f4de7-npdwh"] Oct 31 00:49:41.579042 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:41.579017 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-4ab8e5-openshift-default-6c89ffbf45-mtckf" Oct 31 00:49:41.584623 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:41.584599 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-4ab8e5-openshift-default-6c89ffbf45-mtckf" Oct 31 00:49:41.633502 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:41.633442 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-4ab8e5-openshift-default-6c89ffbf45-mtckf" podStartSLOduration=3.633425484 podStartE2EDuration="3.633425484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:49:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:49:41.630546468 +0000 UTC m=+2074.101716219" watchObservedRunningTime="2026-10-31 00:49:41.633425484 +0000 UTC m=+2074.104595234" Oct 31 00:49:41.670792 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:41.670759 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10937a00-0cff-4b28-9a52-6cef09efe247" path="/var/lib/kubelet/pods/10937a00-0cff-4b28-9a52-6cef09efe247/volumes" Oct 31 00:49:41.671294 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:41.671280 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="608e314b-554b-40a2-90f7-fbdefd2b27b6" path="/var/lib/kubelet/pods/608e314b-554b-40a2-90f7-fbdefd2b27b6/volumes" Oct 31 00:49:41.671780 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:41.671765 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f238e161-3983-4294-9da6-e99c34417a48" path="/var/lib/kubelet/pods/f238e161-3983-4294-9da6-e99c34417a48/volumes" Oct 31 00:49:42.512618 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:42.512586 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-4ab8e5-openshift-default-6c89ffbf45-mtckf" Oct 31 00:49:42.513715 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:42.513695 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-4ab8e5-openshift-default-6c89ffbf45-mtckf" Oct 31 00:49:46.201993 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:46.201899 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-dcdbd0-bc7d86cfc-rs4qx"] Oct 31 00:49:46.264600 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:46.264550 5222 patch_prober.go:29] interesting pod/e2e-api-b-50a19d-openshift-default-68c6d7d74c-24fwv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:49:46.264773 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:46.264658 5222 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-50a19d-openshift-default-68c6d7d74c-24fwv" podUID="1f2eb643-034b-4b9a-bd2f-6ddb034008b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:46.303456 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:46.303423 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-dcdbd0-7cc8947c99-k4hnc"] Oct 31 00:49:47.558280 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:47.558249 5222 generic.go:356] "Generic (PLEG): container finished" podID="1f2eb643-034b-4b9a-bd2f-6ddb034008b8" containerID="31c580e668900e489b539a14c43cd7cf2fc958b6bd1ddb1e866d31c62799d239" exitCode=0 Oct 31 00:49:47.558770 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:47.558320 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-50a19d-openshift-default-68c6d7d74c-24fwv" event={"ID":"1f2eb643-034b-4b9a-bd2f-6ddb034008b8","Type":"ContainerDied","Data":"31c580e668900e489b539a14c43cd7cf2fc958b6bd1ddb1e866d31c62799d239"} Oct 31 00:49:47.558831 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:47.558799 5222 scope.go:122] "RemoveContainer" containerID="31c580e668900e489b539a14c43cd7cf2fc958b6bd1ddb1e866d31c62799d239" Oct 31 00:49:47.560701 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:47.560665 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:49:47.560819 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:47.560754 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:49:47.560819 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:47.560808 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:49:48.016785 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:48.016744 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f891e6-openshift-default-565fdf54d7-56xb6"] Oct 31 00:49:48.017044 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:48.017014 5222 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-f891e6-openshift-default-565fdf54d7-56xb6" podUID="18761079-b183-45ca-9b61-cd9717b0195a" containerName="istio-proxy" containerID="cri-o://55c5be21442828cfe86324df5a0b3caed961495f1c97af0b2ba6f559e359e926" gracePeriod=30 Oct 31 00:49:48.569490 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:48.569453 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-50a19d-openshift-default-68c6d7d74c-24fwv" event={"ID":"1f2eb643-034b-4b9a-bd2f-6ddb034008b8","Type":"ContainerStarted","Data":"5c11287883f26b13ebd3c0fbefb7ffe9f512dd4615040c56d0e120124faccc7b"} Oct 31 00:49:48.840241 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:48.836725 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-50a19d-openshift-default-68c6d7d74c-24fwv" Oct 31 00:49:48.840241 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:48.839984 5222 patch_prober.go:29] interesting pod/e2e-api-b-50a19d-openshift-default-68c6d7d74c-24fwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body= Oct 31 00:49:48.840241 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:48.840048 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-50a19d-openshift-default-68c6d7d74c-24fwv" podUID="1f2eb643-034b-4b9a-bd2f-6ddb034008b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" Oct 31 00:49:49.834511 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:49.834471 5222 patch_prober.go:29] interesting pod/e2e-api-b-50a19d-openshift-default-68c6d7d74c-24fwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body= Oct 31 00:49:49.834926 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:49.834527 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-50a19d-openshift-default-68c6d7d74c-24fwv" podUID="1f2eb643-034b-4b9a-bd2f-6ddb034008b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" Oct 31 00:49:50.422222 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:50.422186 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4ab8e5-79cc98db87-r95fp"] Oct 31 00:49:50.422994 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:50.422972 5222 cpu_manager.go:414] "containerMap: removing container" podUID="608e314b-554b-40a2-90f7-fbdefd2b27b6" containerName="istio-proxy" Oct 31 00:49:50.423125 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:50.422995 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="608e314b-554b-40a2-90f7-fbdefd2b27b6" containerName="istio-proxy" Oct 31 00:49:50.423125 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:50.423013 5222 cpu_manager.go:414] "containerMap: removing container" podUID="608e314b-554b-40a2-90f7-fbdefd2b27b6" containerName="istio-proxy" Oct 31 00:49:50.423125 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:50.423020 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="608e314b-554b-40a2-90f7-fbdefd2b27b6" containerName="istio-proxy" Oct 31 00:49:50.423290 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:50.423197 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="608e314b-554b-40a2-90f7-fbdefd2b27b6" containerName="istio-proxy" Oct 31 00:49:50.423290 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:50.423212 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="608e314b-554b-40a2-90f7-fbdefd2b27b6" containerName="istio-proxy" Oct 31 00:49:50.427010 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:50.426984 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4ab8e5-79cc98db87-r95fp" Oct 31 00:49:50.488200 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:50.488149 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4ab8e5-79cc98db87-r95fp"] Oct 31 00:49:50.543398 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:50.538002 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krfp7\" (UniqueName: \"kubernetes.io/projected/3132fd63-bf3d-4386-a31c-3c2ff47efd20-kube-api-access-krfp7\") pod \"payload-pre-processing-e2e-inf-a-4ab8e5-79cc98db87-r95fp\" (UID: \"3132fd63-bf3d-4386-a31c-3c2ff47efd20\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4ab8e5-79cc98db87-r95fp" Oct 31 00:49:50.543398 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:50.538056 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3132fd63-bf3d-4386-a31c-3c2ff47efd20-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-4ab8e5-79cc98db87-r95fp\" (UID: \"3132fd63-bf3d-4386-a31c-3c2ff47efd20\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4ab8e5-79cc98db87-r95fp" Oct 31 00:49:50.572010 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:50.571661 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-4ab8e5\"" Oct 31 00:49:50.572010 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:50.571777 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-4ab8e5-dockercfg-6x4c7\"" Oct 31 00:49:50.677484 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:50.639458 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krfp7\" (UniqueName: \"kubernetes.io/projected/3132fd63-bf3d-4386-a31c-3c2ff47efd20-kube-api-access-krfp7\") pod \"payload-pre-processing-e2e-inf-a-4ab8e5-79cc98db87-r95fp\" (UID: \"3132fd63-bf3d-4386-a31c-3c2ff47efd20\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4ab8e5-79cc98db87-r95fp" Oct 31 00:49:50.677484 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:50.639509 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3132fd63-bf3d-4386-a31c-3c2ff47efd20-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-4ab8e5-79cc98db87-r95fp\" (UID: \"3132fd63-bf3d-4386-a31c-3c2ff47efd20\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4ab8e5-79cc98db87-r95fp" Oct 31 00:49:50.677484 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:50.640694 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3132fd63-bf3d-4386-a31c-3c2ff47efd20-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-4ab8e5-79cc98db87-r95fp\" (UID: \"3132fd63-bf3d-4386-a31c-3c2ff47efd20\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4ab8e5-79cc98db87-r95fp" Oct 31 00:49:50.703618 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:50.703582 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4ab8e5-b77879c55-qptzw"] Oct 31 00:49:50.715319 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:50.715287 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-4ab8e5-b77879c55-qptzw" Oct 31 00:49:50.735828 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:50.735795 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4ab8e5-b77879c55-qptzw"] Oct 31 00:49:50.738282 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:50.738245 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krfp7\" (UniqueName: \"kubernetes.io/projected/3132fd63-bf3d-4386-a31c-3c2ff47efd20-kube-api-access-krfp7\") pod \"payload-pre-processing-e2e-inf-a-4ab8e5-79cc98db87-r95fp\" (UID: \"3132fd63-bf3d-4386-a31c-3c2ff47efd20\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4ab8e5-79cc98db87-r95fp" Oct 31 00:49:50.741469 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:50.741349 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8t7mp\" (UniqueName: \"kubernetes.io/projected/81a85359-4e26-4ae9-8251-11e56381e3bd-kube-api-access-8t7mp\") pod \"payload-processing-e2e-inf-a-4ab8e5-b77879c55-qptzw\" (UID: \"81a85359-4e26-4ae9-8251-11e56381e3bd\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4ab8e5-b77879c55-qptzw" Oct 31 00:49:50.741614 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:50.741554 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81a85359-4e26-4ae9-8251-11e56381e3bd-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-4ab8e5-b77879c55-qptzw\" (UID: \"81a85359-4e26-4ae9-8251-11e56381e3bd\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4ab8e5-b77879c55-qptzw" Oct 31 00:49:50.751949 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:50.751923 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4ab8e5-79cc98db87-r95fp" Oct 31 00:49:50.834652 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:50.834622 5222 patch_prober.go:29] interesting pod/e2e-api-b-50a19d-openshift-default-68c6d7d74c-24fwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body= Oct 31 00:49:50.835071 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:50.834696 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-50a19d-openshift-default-68c6d7d74c-24fwv" podUID="1f2eb643-034b-4b9a-bd2f-6ddb034008b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" Oct 31 00:49:50.871886 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:50.870635 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8t7mp\" (UniqueName: \"kubernetes.io/projected/81a85359-4e26-4ae9-8251-11e56381e3bd-kube-api-access-8t7mp\") pod \"payload-processing-e2e-inf-a-4ab8e5-b77879c55-qptzw\" (UID: \"81a85359-4e26-4ae9-8251-11e56381e3bd\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4ab8e5-b77879c55-qptzw" Oct 31 00:49:50.871886 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:50.870731 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81a85359-4e26-4ae9-8251-11e56381e3bd-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-4ab8e5-b77879c55-qptzw\" (UID: \"81a85359-4e26-4ae9-8251-11e56381e3bd\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4ab8e5-b77879c55-qptzw" Oct 31 00:49:50.871886 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:50.871806 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81a85359-4e26-4ae9-8251-11e56381e3bd-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-4ab8e5-b77879c55-qptzw\" (UID: \"81a85359-4e26-4ae9-8251-11e56381e3bd\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4ab8e5-b77879c55-qptzw" Oct 31 00:49:50.933970 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:50.933889 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8t7mp\" (UniqueName: \"kubernetes.io/projected/81a85359-4e26-4ae9-8251-11e56381e3bd-kube-api-access-8t7mp\") pod \"payload-processing-e2e-inf-a-4ab8e5-b77879c55-qptzw\" (UID: \"81a85359-4e26-4ae9-8251-11e56381e3bd\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4ab8e5-b77879c55-qptzw" Oct 31 00:49:51.062936 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:51.062684 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-4ab8e5-b77879c55-qptzw" Oct 31 00:49:51.168753 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:51.168491 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4ab8e5-79cc98db87-r95fp"] Oct 31 00:49:51.203286 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:51.203246 5222 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-dcdbd0-bc7d86cfc-rs4qx" podUID="8899ca28-d521-4591-b372-dcef03e946e1" containerName="payload-processing" containerID="cri-o://b311048e31633670fe1f6caffdfe4629820a8ca6413b72583f6476e109a0444a" gracePeriod=25 Oct 31 00:49:51.304294 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:51.304247 5222 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-dcdbd0-7cc8947c99-k4hnc" podUID="bb6ff052-f89c-4b7f-945f-01ffd909308c" containerName="payload-pre-processing" containerID="cri-o://1d1a4dc4adedb20f0377dc7d82e5807701afda3799a04c7dd514c5f4d6cdf98c" gracePeriod=25 Oct 31 00:49:51.475916 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:51.475880 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4ab8e5-b77879c55-qptzw"] Oct 31 00:49:51.479131 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:49:51.478432 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81a85359_4e26_4ae9_8251_11e56381e3bd.slice/crio-d746f722099ad1b04b64a27477416c8318c5ff5fa18dfec01280e6db6dd4faa4 WatchSource:0}: Error finding container d746f722099ad1b04b64a27477416c8318c5ff5fa18dfec01280e6db6dd4faa4: Status 404 returned error can't find the container with id d746f722099ad1b04b64a27477416c8318c5ff5fa18dfec01280e6db6dd4faa4 Oct 31 00:49:51.514364 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:51.514338 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-dcdbd0-bc7d86cfc-rs4qx" Oct 31 00:49:51.586801 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:51.585199 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8899ca28-d521-4591-b372-dcef03e946e1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8899ca28-d521-4591-b372-dcef03e946e1-plugins-config-volume\") pod \"8899ca28-d521-4591-b372-dcef03e946e1\" (UID: \"8899ca28-d521-4591-b372-dcef03e946e1\") " Oct 31 00:49:51.586801 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:51.585420 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8899ca28-d521-4591-b372-dcef03e946e1-kube-api-access-nwd7h\" (UniqueName: \"kubernetes.io/projected/8899ca28-d521-4591-b372-dcef03e946e1-kube-api-access-nwd7h\") pod \"8899ca28-d521-4591-b372-dcef03e946e1\" (UID: \"8899ca28-d521-4591-b372-dcef03e946e1\") " Oct 31 00:49:51.587076 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:51.587045 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8899ca28-d521-4591-b372-dcef03e946e1-plugins-config-volume" pod "8899ca28-d521-4591-b372-dcef03e946e1" (UID: "8899ca28-d521-4591-b372-dcef03e946e1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:49:51.589994 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:51.589266 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8899ca28-d521-4591-b372-dcef03e946e1-kube-api-access-nwd7h" pod "8899ca28-d521-4591-b372-dcef03e946e1" (UID: "8899ca28-d521-4591-b372-dcef03e946e1"). InnerVolumeSpecName "kube-api-access-nwd7h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:49:51.683616 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:51.683570 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-dcdbd0-7cc8947c99-k4hnc" Oct 31 00:49:51.687238 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:51.687216 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwd7h\" (UniqueName: \"kubernetes.io/projected/8899ca28-d521-4591-b372-dcef03e946e1-kube-api-access-nwd7h\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:51.687432 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:51.687419 5222 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8899ca28-d521-4591-b372-dcef03e946e1-plugins-config-volume\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:51.714711 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:51.714680 5222 generic.go:356] "Generic (PLEG): container finished" podID="8899ca28-d521-4591-b372-dcef03e946e1" containerID="b311048e31633670fe1f6caffdfe4629820a8ca6413b72583f6476e109a0444a" exitCode=0 Oct 31 00:49:51.714911 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:51.714724 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-dcdbd0-bc7d86cfc-rs4qx" Oct 31 00:49:51.714911 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:51.714756 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-dcdbd0-bc7d86cfc-rs4qx" event={"ID":"8899ca28-d521-4591-b372-dcef03e946e1","Type":"ContainerDied","Data":"b311048e31633670fe1f6caffdfe4629820a8ca6413b72583f6476e109a0444a"} Oct 31 00:49:51.714911 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:51.714807 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-dcdbd0-bc7d86cfc-rs4qx" event={"ID":"8899ca28-d521-4591-b372-dcef03e946e1","Type":"ContainerDied","Data":"fa290622518adecc96355a6282f549a8e33a8629f1129e2a8fec58ad27b6cfcb"} Oct 31 00:49:51.714911 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:51.714828 5222 scope.go:122] "RemoveContainer" containerID="b311048e31633670fe1f6caffdfe4629820a8ca6413b72583f6476e109a0444a" Oct 31 00:49:51.728839 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:51.728805 5222 generic.go:356] "Generic (PLEG): container finished" podID="bb6ff052-f89c-4b7f-945f-01ffd909308c" containerID="1d1a4dc4adedb20f0377dc7d82e5807701afda3799a04c7dd514c5f4d6cdf98c" exitCode=0 Oct 31 00:49:51.728999 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:51.728861 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-dcdbd0-7cc8947c99-k4hnc" Oct 31 00:49:51.728999 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:51.728974 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-dcdbd0-7cc8947c99-k4hnc" event={"ID":"bb6ff052-f89c-4b7f-945f-01ffd909308c","Type":"ContainerDied","Data":"1d1a4dc4adedb20f0377dc7d82e5807701afda3799a04c7dd514c5f4d6cdf98c"} Oct 31 00:49:51.729101 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:51.729066 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-dcdbd0-7cc8947c99-k4hnc" event={"ID":"bb6ff052-f89c-4b7f-945f-01ffd909308c","Type":"ContainerDied","Data":"95fd6f16f6ffa572c4cb4024218c79d1ad35b92f67df3cf1d45d57117524ed6e"} Oct 31 00:49:51.730906 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:51.730877 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-4ab8e5-b77879c55-qptzw" event={"ID":"81a85359-4e26-4ae9-8251-11e56381e3bd","Type":"ContainerStarted","Data":"d52f3b5a282eff0303b6cbf6bb425d4ff66ce65fedff6b076dc6dab8aafdf9b8"} Oct 31 00:49:51.731035 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:51.730928 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-4ab8e5-b77879c55-qptzw" event={"ID":"81a85359-4e26-4ae9-8251-11e56381e3bd","Type":"ContainerStarted","Data":"d746f722099ad1b04b64a27477416c8318c5ff5fa18dfec01280e6db6dd4faa4"} Oct 31 00:49:51.731035 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:51.730986 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-4ab8e5-b77879c55-qptzw" Oct 31 00:49:51.733121 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:51.733004 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4ab8e5-79cc98db87-r95fp" event={"ID":"3132fd63-bf3d-4386-a31c-3c2ff47efd20","Type":"ContainerStarted","Data":"3c703c647fd747d85b363db7d63fc973555d90660597e3d8c2d54dff6fa6890e"} Oct 31 00:49:51.733121 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:51.733042 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4ab8e5-79cc98db87-r95fp" event={"ID":"3132fd63-bf3d-4386-a31c-3c2ff47efd20","Type":"ContainerStarted","Data":"99a3e8360803ebf10da834ae17ba93d4376bddf2c216eb3ef51bcfbd5b6b7c59"} Oct 31 00:49:51.733121 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:51.733074 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4ab8e5-79cc98db87-r95fp" Oct 31 00:49:51.773967 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:51.773787 5222 scope.go:122] "RemoveContainer" containerID="b311048e31633670fe1f6caffdfe4629820a8ca6413b72583f6476e109a0444a" Oct 31 00:49:51.774175 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:49:51.774146 5222 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b311048e31633670fe1f6caffdfe4629820a8ca6413b72583f6476e109a0444a\": container with ID starting with b311048e31633670fe1f6caffdfe4629820a8ca6413b72583f6476e109a0444a not found: ID does not exist" containerID="b311048e31633670fe1f6caffdfe4629820a8ca6413b72583f6476e109a0444a" Oct 31 00:49:51.774222 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:51.774189 5222 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b311048e31633670fe1f6caffdfe4629820a8ca6413b72583f6476e109a0444a"} err="failed to get container status \"b311048e31633670fe1f6caffdfe4629820a8ca6413b72583f6476e109a0444a\": rpc error: code = NotFound desc = could not find container \"b311048e31633670fe1f6caffdfe4629820a8ca6413b72583f6476e109a0444a\": container with ID starting with b311048e31633670fe1f6caffdfe4629820a8ca6413b72583f6476e109a0444a not found: ID does not exist" Oct 31 00:49:51.774222 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:51.774214 5222 scope.go:122] "RemoveContainer" containerID="1d1a4dc4adedb20f0377dc7d82e5807701afda3799a04c7dd514c5f4d6cdf98c" Oct 31 00:49:51.784964 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:51.784941 5222 scope.go:122] "RemoveContainer" containerID="1d1a4dc4adedb20f0377dc7d82e5807701afda3799a04c7dd514c5f4d6cdf98c" Oct 31 00:49:51.785308 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:49:51.785276 5222 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d1a4dc4adedb20f0377dc7d82e5807701afda3799a04c7dd514c5f4d6cdf98c\": container with ID starting with 1d1a4dc4adedb20f0377dc7d82e5807701afda3799a04c7dd514c5f4d6cdf98c not found: ID does not exist" containerID="1d1a4dc4adedb20f0377dc7d82e5807701afda3799a04c7dd514c5f4d6cdf98c" Oct 31 00:49:51.785428 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:51.785309 5222 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d1a4dc4adedb20f0377dc7d82e5807701afda3799a04c7dd514c5f4d6cdf98c"} err="failed to get container status \"1d1a4dc4adedb20f0377dc7d82e5807701afda3799a04c7dd514c5f4d6cdf98c\": rpc error: code = NotFound desc = could not find container \"1d1a4dc4adedb20f0377dc7d82e5807701afda3799a04c7dd514c5f4d6cdf98c\": container with ID starting with 1d1a4dc4adedb20f0377dc7d82e5807701afda3799a04c7dd514c5f4d6cdf98c not found: ID does not exist" Oct 31 00:49:51.788278 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:51.788256 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb6ff052-f89c-4b7f-945f-01ffd909308c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb6ff052-f89c-4b7f-945f-01ffd909308c-plugins-config-volume\") pod \"bb6ff052-f89c-4b7f-945f-01ffd909308c\" (UID: \"bb6ff052-f89c-4b7f-945f-01ffd909308c\") " Oct 31 00:49:51.788337 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:51.788326 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb6ff052-f89c-4b7f-945f-01ffd909308c-kube-api-access-bjlwb\" (UniqueName: \"kubernetes.io/projected/bb6ff052-f89c-4b7f-945f-01ffd909308c-kube-api-access-bjlwb\") pod \"bb6ff052-f89c-4b7f-945f-01ffd909308c\" (UID: \"bb6ff052-f89c-4b7f-945f-01ffd909308c\") " Oct 31 00:49:51.788724 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:51.788694 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb6ff052-f89c-4b7f-945f-01ffd909308c-plugins-config-volume" pod "bb6ff052-f89c-4b7f-945f-01ffd909308c" (UID: "bb6ff052-f89c-4b7f-945f-01ffd909308c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:49:51.788984 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:51.788967 5222 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb6ff052-f89c-4b7f-945f-01ffd909308c-plugins-config-volume\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:51.790950 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:51.790931 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb6ff052-f89c-4b7f-945f-01ffd909308c-kube-api-access-bjlwb" pod "bb6ff052-f89c-4b7f-945f-01ffd909308c" (UID: "bb6ff052-f89c-4b7f-945f-01ffd909308c"). InnerVolumeSpecName "kube-api-access-bjlwb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:49:51.834698 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:51.834664 5222 patch_prober.go:29] interesting pod/e2e-api-b-50a19d-openshift-default-68c6d7d74c-24fwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body= Oct 31 00:49:51.835108 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:51.834730 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-50a19d-openshift-default-68c6d7d74c-24fwv" podUID="1f2eb643-034b-4b9a-bd2f-6ddb034008b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" Oct 31 00:49:51.890682 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:51.890649 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjlwb\" (UniqueName: \"kubernetes.io/projected/bb6ff052-f89c-4b7f-945f-01ffd909308c-kube-api-access-bjlwb\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:52.027822 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:52.027769 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4ab8e5-79cc98db87-r95fp" podStartSLOduration=2.027754118 podStartE2EDuration="2.027754118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:49:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:49:52.025418513 +0000 UTC m=+2084.496588253" watchObservedRunningTime="2026-10-31 00:49:52.027754118 +0000 UTC m=+2084.498923866" Oct 31 00:49:52.204090 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:52.204046 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-dcdbd0-bc7d86cfc-rs4qx"] Oct 31 00:49:52.206313 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:52.205267 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-dcdbd0-bc7d86cfc-rs4qx"] Oct 31 00:49:52.291112 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:52.291001 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-4ab8e5-b77879c55-qptzw" podStartSLOduration=2.290987267 podStartE2EDuration="2.290987267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:49:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:49:52.288466921 +0000 UTC m=+2084.759636670" watchObservedRunningTime="2026-10-31 00:49:52.290987267 +0000 UTC m=+2084.762157076" Oct 31 00:49:52.474545 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:52.474486 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-dcdbd0-7cc8947c99-k4hnc"] Oct 31 00:49:52.477152 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:52.476994 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-dcdbd0-7cc8947c99-k4hnc"] Oct 31 00:49:52.834589 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:52.834547 5222 patch_prober.go:29] interesting pod/e2e-api-b-50a19d-openshift-default-68c6d7d74c-24fwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body= Oct 31 00:49:52.834818 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:52.834618 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-50a19d-openshift-default-68c6d7d74c-24fwv" podUID="1f2eb643-034b-4b9a-bd2f-6ddb034008b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" Oct 31 00:49:52.866692 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:52.866653 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-982ae3-k6nrb"] Oct 31 00:49:52.867350 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:52.867327 5222 cpu_manager.go:414] "containerMap: removing container" podUID="bb6ff052-f89c-4b7f-945f-01ffd909308c" containerName="payload-pre-processing" Oct 31 00:49:52.867350 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:52.867348 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb6ff052-f89c-4b7f-945f-01ffd909308c" containerName="payload-pre-processing" Oct 31 00:49:52.867548 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:52.867403 5222 cpu_manager.go:414] "containerMap: removing container" podUID="8899ca28-d521-4591-b372-dcef03e946e1" containerName="payload-processing" Oct 31 00:49:52.867548 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:52.867410 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8899ca28-d521-4591-b372-dcef03e946e1" containerName="payload-processing" Oct 31 00:49:52.867548 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:52.867537 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="8899ca28-d521-4591-b372-dcef03e946e1" containerName="payload-processing" Oct 31 00:49:52.867712 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:52.867553 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb6ff052-f89c-4b7f-945f-01ffd909308c" containerName="payload-pre-processing" Oct 31 00:49:52.872703 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:52.872680 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-982ae3-k6nrb" Oct 31 00:49:52.913429 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:52.913394 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-982ae3-k6nrb"] Oct 31 00:49:52.926418 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:52.926364 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:49:52.926589 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:52.926442 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:49:52.958010 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:52.957706 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:49:52.962599 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:52.962561 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfz9z\"" Oct 31 00:49:53.008186 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:53.008151 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlwj2\" (UniqueName: \"kubernetes.io/projected/9b5e141c-81c8-497a-aa91-c9e30c14fe30-kube-api-access-mlwj2\") pod \"maas-api-revoke-keys-e2e-worker-w1-982ae3-k6nrb\" (UID: \"9b5e141c-81c8-497a-aa91-c9e30c14fe30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-982ae3-k6nrb" Oct 31 00:49:53.008404 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:53.008342 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b5e141c-81c8-497a-aa91-c9e30c14fe30-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-982ae3-k6nrb\" (UID: \"9b5e141c-81c8-497a-aa91-c9e30c14fe30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-982ae3-k6nrb" Oct 31 00:49:53.008493 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:53.008445 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b5e141c-81c8-497a-aa91-c9e30c14fe30-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-982ae3-k6nrb\" (UID: \"9b5e141c-81c8-497a-aa91-c9e30c14fe30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-982ae3-k6nrb" Oct 31 00:49:53.110063 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:53.109985 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlwj2\" (UniqueName: \"kubernetes.io/projected/9b5e141c-81c8-497a-aa91-c9e30c14fe30-kube-api-access-mlwj2\") pod \"maas-api-revoke-keys-e2e-worker-w1-982ae3-k6nrb\" (UID: \"9b5e141c-81c8-497a-aa91-c9e30c14fe30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-982ae3-k6nrb" Oct 31 00:49:53.110213 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:53.110118 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b5e141c-81c8-497a-aa91-c9e30c14fe30-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-982ae3-k6nrb\" (UID: \"9b5e141c-81c8-497a-aa91-c9e30c14fe30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-982ae3-k6nrb" Oct 31 00:49:53.110213 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:53.110159 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b5e141c-81c8-497a-aa91-c9e30c14fe30-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-982ae3-k6nrb\" (UID: \"9b5e141c-81c8-497a-aa91-c9e30c14fe30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-982ae3-k6nrb" Oct 31 00:49:53.110918 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:53.110898 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b5e141c-81c8-497a-aa91-c9e30c14fe30-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-982ae3-k6nrb\" (UID: \"9b5e141c-81c8-497a-aa91-c9e30c14fe30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-982ae3-k6nrb" Oct 31 00:49:53.112810 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:53.112779 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b5e141c-81c8-497a-aa91-c9e30c14fe30-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-982ae3-k6nrb\" (UID: \"9b5e141c-81c8-497a-aa91-c9e30c14fe30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-982ae3-k6nrb" Oct 31 00:49:53.174427 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:53.173546 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlwj2\" (UniqueName: \"kubernetes.io/projected/9b5e141c-81c8-497a-aa91-c9e30c14fe30-kube-api-access-mlwj2\") pod \"maas-api-revoke-keys-e2e-worker-w1-982ae3-k6nrb\" (UID: \"9b5e141c-81c8-497a-aa91-c9e30c14fe30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-982ae3-k6nrb" Oct 31 00:49:53.187079 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:53.187044 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-982ae3-k6nrb" Oct 31 00:49:53.376668 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:53.376582 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f891e6-openshift-default-565fdf54d7-56xb6" Oct 31 00:49:53.515845 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:53.515807 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18761079-b183-45ca-9b61-cd9717b0195a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18761079-b183-45ca-9b61-cd9717b0195a-workload-certs\") pod \"18761079-b183-45ca-9b61-cd9717b0195a\" (UID: \"18761079-b183-45ca-9b61-cd9717b0195a\") " Oct 31 00:49:53.516039 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:53.515918 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18761079-b183-45ca-9b61-cd9717b0195a-kube-api-access-jh2fd\" (UniqueName: \"kubernetes.io/projected/18761079-b183-45ca-9b61-cd9717b0195a-kube-api-access-jh2fd\") pod \"18761079-b183-45ca-9b61-cd9717b0195a\" (UID: \"18761079-b183-45ca-9b61-cd9717b0195a\") " Oct 31 00:49:53.516039 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:53.515990 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18761079-b183-45ca-9b61-cd9717b0195a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18761079-b183-45ca-9b61-cd9717b0195a-istiod-ca-cert\") pod \"18761079-b183-45ca-9b61-cd9717b0195a\" (UID: \"18761079-b183-45ca-9b61-cd9717b0195a\") " Oct 31 00:49:53.516175 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:53.516083 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18761079-b183-45ca-9b61-cd9717b0195a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18761079-b183-45ca-9b61-cd9717b0195a-istio-data\") pod \"18761079-b183-45ca-9b61-cd9717b0195a\" (UID: \"18761079-b183-45ca-9b61-cd9717b0195a\") " Oct 31 00:49:53.516175 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:53.516127 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18761079-b183-45ca-9b61-cd9717b0195a-workload-certs" pod "18761079-b183-45ca-9b61-cd9717b0195a" (UID: "18761079-b183-45ca-9b61-cd9717b0195a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:49:53.516175 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:53.516153 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18761079-b183-45ca-9b61-cd9717b0195a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18761079-b183-45ca-9b61-cd9717b0195a-workload-socket\") pod \"18761079-b183-45ca-9b61-cd9717b0195a\" (UID: \"18761079-b183-45ca-9b61-cd9717b0195a\") " Oct 31 00:49:53.516338 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:53.516227 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18761079-b183-45ca-9b61-cd9717b0195a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18761079-b183-45ca-9b61-cd9717b0195a-istio-envoy\") pod \"18761079-b183-45ca-9b61-cd9717b0195a\" (UID: \"18761079-b183-45ca-9b61-cd9717b0195a\") " Oct 31 00:49:53.516338 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:53.516269 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18761079-b183-45ca-9b61-cd9717b0195a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18761079-b183-45ca-9b61-cd9717b0195a-credential-socket\") pod \"18761079-b183-45ca-9b61-cd9717b0195a\" (UID: \"18761079-b183-45ca-9b61-cd9717b0195a\") " Oct 31 00:49:53.516488 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:53.516440 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/18761079-b183-45ca-9b61-cd9717b0195a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18761079-b183-45ca-9b61-cd9717b0195a-istio-podinfo\") pod \"18761079-b183-45ca-9b61-cd9717b0195a\" (UID: \"18761079-b183-45ca-9b61-cd9717b0195a\") " Oct 31 00:49:53.516549 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:53.516479 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18761079-b183-45ca-9b61-cd9717b0195a-istiod-ca-cert" pod "18761079-b183-45ca-9b61-cd9717b0195a" (UID: "18761079-b183-45ca-9b61-cd9717b0195a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:49:53.516549 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:53.516483 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18761079-b183-45ca-9b61-cd9717b0195a-workload-socket" pod "18761079-b183-45ca-9b61-cd9717b0195a" (UID: "18761079-b183-45ca-9b61-cd9717b0195a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:49:53.516549 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:53.516500 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18761079-b183-45ca-9b61-cd9717b0195a-istio-token\" (UniqueName: \"kubernetes.io/projected/18761079-b183-45ca-9b61-cd9717b0195a-istio-token\") pod \"18761079-b183-45ca-9b61-cd9717b0195a\" (UID: \"18761079-b183-45ca-9b61-cd9717b0195a\") " Oct 31 00:49:53.516710 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:53.516686 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18761079-b183-45ca-9b61-cd9717b0195a-istio-data" pod "18761079-b183-45ca-9b61-cd9717b0195a" (UID: "18761079-b183-45ca-9b61-cd9717b0195a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:49:53.516792 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:53.516768 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18761079-b183-45ca-9b61-cd9717b0195a-credential-socket" pod "18761079-b183-45ca-9b61-cd9717b0195a" (UID: "18761079-b183-45ca-9b61-cd9717b0195a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:49:53.517217 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:53.517188 5222 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18761079-b183-45ca-9b61-cd9717b0195a-workload-socket\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:53.517217 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:53.517218 5222 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18761079-b183-45ca-9b61-cd9717b0195a-credential-socket\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:53.517409 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:53.517233 5222 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18761079-b183-45ca-9b61-cd9717b0195a-workload-certs\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:53.517409 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:53.517248 5222 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18761079-b183-45ca-9b61-cd9717b0195a-istiod-ca-cert\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:53.517409 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:53.517260 5222 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18761079-b183-45ca-9b61-cd9717b0195a-istio-data\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:53.520111 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:53.520088 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/18761079-b183-45ca-9b61-cd9717b0195a-istio-podinfo" pod "18761079-b183-45ca-9b61-cd9717b0195a" (UID: "18761079-b183-45ca-9b61-cd9717b0195a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:49:53.520223 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:53.520129 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18761079-b183-45ca-9b61-cd9717b0195a-istio-token" pod "18761079-b183-45ca-9b61-cd9717b0195a" (UID: "18761079-b183-45ca-9b61-cd9717b0195a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:49:53.520288 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:53.520236 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18761079-b183-45ca-9b61-cd9717b0195a-istio-envoy" pod "18761079-b183-45ca-9b61-cd9717b0195a" (UID: "18761079-b183-45ca-9b61-cd9717b0195a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:49:53.522252 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:53.521713 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18761079-b183-45ca-9b61-cd9717b0195a-kube-api-access-jh2fd" pod "18761079-b183-45ca-9b61-cd9717b0195a" (UID: "18761079-b183-45ca-9b61-cd9717b0195a"). InnerVolumeSpecName "kube-api-access-jh2fd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:49:53.615329 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:49:53.615130 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b5e141c_81c8_497a_aa91_c9e30c14fe30.slice/crio-ea3e1960a6b9930f772c2abd1c362ac000e84fdad1f8fc56f934ded713ece7c6 WatchSource:0}: Error finding container ea3e1960a6b9930f772c2abd1c362ac000e84fdad1f8fc56f934ded713ece7c6: Status 404 returned error can't find the container with id ea3e1960a6b9930f772c2abd1c362ac000e84fdad1f8fc56f934ded713ece7c6 Oct 31 00:49:53.618161 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:53.618134 5222 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18761079-b183-45ca-9b61-cd9717b0195a-istio-podinfo\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:53.618280 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:53.618164 5222 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18761079-b183-45ca-9b61-cd9717b0195a-istio-token\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:53.618280 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:53.618179 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jh2fd\" (UniqueName: \"kubernetes.io/projected/18761079-b183-45ca-9b61-cd9717b0195a-kube-api-access-jh2fd\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:53.618280 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:53.618195 5222 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18761079-b183-45ca-9b61-cd9717b0195a-istio-envoy\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:53.618724 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:53.618702 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-982ae3-k6nrb"] Oct 31 00:49:53.670597 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:53.670562 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8899ca28-d521-4591-b372-dcef03e946e1" path="/var/lib/kubelet/pods/8899ca28-d521-4591-b372-dcef03e946e1/volumes" Oct 31 00:49:53.671032 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:53.671016 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb6ff052-f89c-4b7f-945f-01ffd909308c" path="/var/lib/kubelet/pods/bb6ff052-f89c-4b7f-945f-01ffd909308c/volumes" Oct 31 00:49:53.775239 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:53.775206 5222 generic.go:356] "Generic (PLEG): container finished" podID="18761079-b183-45ca-9b61-cd9717b0195a" containerID="55c5be21442828cfe86324df5a0b3caed961495f1c97af0b2ba6f559e359e926" exitCode=0 Oct 31 00:49:53.775456 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:53.775256 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f891e6-openshift-default-565fdf54d7-56xb6" Oct 31 00:49:53.775456 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:53.775294 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f891e6-openshift-default-565fdf54d7-56xb6" event={"ID":"18761079-b183-45ca-9b61-cd9717b0195a","Type":"ContainerDied","Data":"55c5be21442828cfe86324df5a0b3caed961495f1c97af0b2ba6f559e359e926"} Oct 31 00:49:53.775456 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:53.775353 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f891e6-openshift-default-565fdf54d7-56xb6" event={"ID":"18761079-b183-45ca-9b61-cd9717b0195a","Type":"ContainerDied","Data":"64f112b6f660b30d55fb28d7b2a1385fc252f2088e5d3db8357f9e292174f936"} Oct 31 00:49:53.775456 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:53.775414 5222 scope.go:122] "RemoveContainer" containerID="55c5be21442828cfe86324df5a0b3caed961495f1c97af0b2ba6f559e359e926" Oct 31 00:49:53.777240 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:53.777215 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-982ae3-k6nrb" event={"ID":"9b5e141c-81c8-497a-aa91-c9e30c14fe30","Type":"ContainerStarted","Data":"ea3e1960a6b9930f772c2abd1c362ac000e84fdad1f8fc56f934ded713ece7c6"} Oct 31 00:49:53.786529 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:53.786509 5222 scope.go:122] "RemoveContainer" containerID="17f34d286c445ba41db967e1cc0a46cb4287bfcc862122233af07db17a2e4fc3" Oct 31 00:49:53.797484 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:53.797463 5222 scope.go:122] "RemoveContainer" containerID="55c5be21442828cfe86324df5a0b3caed961495f1c97af0b2ba6f559e359e926" Oct 31 00:49:53.797789 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:49:53.797770 5222 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55c5be21442828cfe86324df5a0b3caed961495f1c97af0b2ba6f559e359e926\": container with ID starting with 55c5be21442828cfe86324df5a0b3caed961495f1c97af0b2ba6f559e359e926 not found: ID does not exist" containerID="55c5be21442828cfe86324df5a0b3caed961495f1c97af0b2ba6f559e359e926" Oct 31 00:49:53.797840 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:53.797798 5222 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55c5be21442828cfe86324df5a0b3caed961495f1c97af0b2ba6f559e359e926"} err="failed to get container status \"55c5be21442828cfe86324df5a0b3caed961495f1c97af0b2ba6f559e359e926\": rpc error: code = NotFound desc = could not find container \"55c5be21442828cfe86324df5a0b3caed961495f1c97af0b2ba6f559e359e926\": container with ID starting with 55c5be21442828cfe86324df5a0b3caed961495f1c97af0b2ba6f559e359e926 not found: ID does not exist" Oct 31 00:49:53.797840 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:53.797814 5222 scope.go:122] "RemoveContainer" containerID="17f34d286c445ba41db967e1cc0a46cb4287bfcc862122233af07db17a2e4fc3" Oct 31 00:49:53.798039 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:49:53.798017 5222 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17f34d286c445ba41db967e1cc0a46cb4287bfcc862122233af07db17a2e4fc3\": container with ID starting with 17f34d286c445ba41db967e1cc0a46cb4287bfcc862122233af07db17a2e4fc3 not found: ID does not exist" containerID="17f34d286c445ba41db967e1cc0a46cb4287bfcc862122233af07db17a2e4fc3" Oct 31 00:49:53.798094 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:53.798049 5222 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17f34d286c445ba41db967e1cc0a46cb4287bfcc862122233af07db17a2e4fc3"} err="failed to get container status \"17f34d286c445ba41db967e1cc0a46cb4287bfcc862122233af07db17a2e4fc3\": rpc error: code = NotFound desc = could not find container \"17f34d286c445ba41db967e1cc0a46cb4287bfcc862122233af07db17a2e4fc3\": container with ID starting with 17f34d286c445ba41db967e1cc0a46cb4287bfcc862122233af07db17a2e4fc3 not found: ID does not exist" Oct 31 00:49:53.834318 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:53.834281 5222 patch_prober.go:29] interesting pod/e2e-api-b-50a19d-openshift-default-68c6d7d74c-24fwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body= Oct 31 00:49:53.834512 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:53.834345 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-50a19d-openshift-default-68c6d7d74c-24fwv" podUID="1f2eb643-034b-4b9a-bd2f-6ddb034008b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" Oct 31 00:49:53.964462 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:53.963430 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f891e6-openshift-default-565fdf54d7-56xb6"] Oct 31 00:49:53.967379 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:53.967329 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-f891e6-openshift-default-565fdf54d7-56xb6"] Oct 31 00:49:54.788694 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:54.788654 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-982ae3-k6nrb" event={"ID":"9b5e141c-81c8-497a-aa91-c9e30c14fe30","Type":"ContainerStarted","Data":"11a7dff0b5e0f347d74ad8a8ef7e72fb5229cc58de19ac5416cc2ad439bf177f"} Oct 31 00:49:54.828583 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:54.828545 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-982ae3/distinct-w1-982ae3-kserve-84b6f97b84-4hhxk"] Oct 31 00:49:54.838907 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:54.838884 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-50a19d-openshift-default-68c6d7d74c-24fwv" Oct 31 00:49:54.839209 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:54.839190 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-50a19d-openshift-default-68c6d7d74c-24fwv" Oct 31 00:49:54.840197 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:54.840173 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-50a19d-openshift-default-68c6d7d74c-24fwv" Oct 31 00:49:54.840677 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:54.840641 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-982ae3/premium-w1-982ae3-kserve-5c8755f445-qm9bh"] Oct 31 00:49:54.913440 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:54.913385 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-982ae3-k6nrb" podStartSLOduration=2.913353004 podStartE2EDuration="2.913353004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:49:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:49:54.909687897 +0000 UTC m=+2087.380857649" watchObservedRunningTime="2026-10-31 00:49:54.913353004 +0000 UTC m=+2087.384522752" Oct 31 00:49:55.286229 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:55.286185 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7d5f4de7-ffd6698f9-2r7sb"] Oct 31 00:49:55.288364 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:55.288336 5222 cpu_manager.go:414] "containerMap: removing container" podUID="18761079-b183-45ca-9b61-cd9717b0195a" containerName="istio-proxy" Oct 31 00:49:55.288592 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:55.288574 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18761079-b183-45ca-9b61-cd9717b0195a" containerName="istio-proxy" Oct 31 00:49:55.288767 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:55.288742 5222 cpu_manager.go:414] "containerMap: removing container" podUID="18761079-b183-45ca-9b61-cd9717b0195a" containerName="istio-proxy" Oct 31 00:49:55.292551 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:55.292520 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18761079-b183-45ca-9b61-cd9717b0195a" containerName="istio-proxy" Oct 31 00:49:55.293006 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:55.292985 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="18761079-b183-45ca-9b61-cd9717b0195a" containerName="istio-proxy" Oct 31 00:49:55.293088 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:55.293022 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="18761079-b183-45ca-9b61-cd9717b0195a" containerName="istio-proxy" Oct 31 00:49:55.297554 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:55.297528 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7d5f4de7-ffd6698f9-2r7sb" Oct 31 00:49:55.383455 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:55.381552 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7d5f4de7-ffd6698f9-2r7sb"] Oct 31 00:49:55.426401 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:55.420710 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-7d5f4de7\"" Oct 31 00:49:55.426401 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:55.421125 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-7d5f4de7-dockercfg-hvkcf\"" Oct 31 00:49:55.446168 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:55.439738 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-425zb\" (UniqueName: \"kubernetes.io/projected/64f10e01-81b0-4bff-a597-19b2a0106639-kube-api-access-425zb\") pod \"payload-pre-processing-e2e-mt-7d5f4de7-ffd6698f9-2r7sb\" (UID: \"64f10e01-81b0-4bff-a597-19b2a0106639\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7d5f4de7-ffd6698f9-2r7sb" Oct 31 00:49:55.446168 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:55.439870 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64f10e01-81b0-4bff-a597-19b2a0106639-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-7d5f4de7-ffd6698f9-2r7sb\" (UID: \"64f10e01-81b0-4bff-a597-19b2a0106639\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7d5f4de7-ffd6698f9-2r7sb" Oct 31 00:49:55.495751 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:55.495715 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7d5f4de7-cf6c94fdc-zvq97"] Oct 31 00:49:55.502737 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:55.502708 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7d5f4de7-cf6c94fdc-zvq97"] Oct 31 00:49:55.502937 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:55.502845 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7d5f4de7-cf6c94fdc-zvq97" Oct 31 00:49:55.541932 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:55.541841 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fcdf\" (UniqueName: \"kubernetes.io/projected/cb02bae7-ad38-4a84-b668-2871c1c581e9-kube-api-access-7fcdf\") pod \"payload-processing-e2e-mt-7d5f4de7-cf6c94fdc-zvq97\" (UID: \"cb02bae7-ad38-4a84-b668-2871c1c581e9\") " pod="openshift-ingress/payload-processing-e2e-mt-7d5f4de7-cf6c94fdc-zvq97" Oct 31 00:49:55.542311 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:55.542289 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb02bae7-ad38-4a84-b668-2871c1c581e9-plugins-config-volume\") pod \"payload-processing-e2e-mt-7d5f4de7-cf6c94fdc-zvq97\" (UID: \"cb02bae7-ad38-4a84-b668-2871c1c581e9\") " pod="openshift-ingress/payload-processing-e2e-mt-7d5f4de7-cf6c94fdc-zvq97" Oct 31 00:49:55.542588 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:55.542570 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-425zb\" (UniqueName: \"kubernetes.io/projected/64f10e01-81b0-4bff-a597-19b2a0106639-kube-api-access-425zb\") pod \"payload-pre-processing-e2e-mt-7d5f4de7-ffd6698f9-2r7sb\" (UID: \"64f10e01-81b0-4bff-a597-19b2a0106639\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7d5f4de7-ffd6698f9-2r7sb" Oct 31 00:49:55.542802 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:55.542788 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64f10e01-81b0-4bff-a597-19b2a0106639-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-7d5f4de7-ffd6698f9-2r7sb\" (UID: \"64f10e01-81b0-4bff-a597-19b2a0106639\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7d5f4de7-ffd6698f9-2r7sb" Oct 31 00:49:55.544264 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:55.544238 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64f10e01-81b0-4bff-a597-19b2a0106639-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-7d5f4de7-ffd6698f9-2r7sb\" (UID: \"64f10e01-81b0-4bff-a597-19b2a0106639\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7d5f4de7-ffd6698f9-2r7sb" Oct 31 00:49:55.634433 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:55.627107 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-425zb\" (UniqueName: \"kubernetes.io/projected/64f10e01-81b0-4bff-a597-19b2a0106639-kube-api-access-425zb\") pod \"payload-pre-processing-e2e-mt-7d5f4de7-ffd6698f9-2r7sb\" (UID: \"64f10e01-81b0-4bff-a597-19b2a0106639\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7d5f4de7-ffd6698f9-2r7sb" Oct 31 00:49:55.657555 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:55.644106 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fcdf\" (UniqueName: \"kubernetes.io/projected/cb02bae7-ad38-4a84-b668-2871c1c581e9-kube-api-access-7fcdf\") pod \"payload-processing-e2e-mt-7d5f4de7-cf6c94fdc-zvq97\" (UID: \"cb02bae7-ad38-4a84-b668-2871c1c581e9\") " pod="openshift-ingress/payload-processing-e2e-mt-7d5f4de7-cf6c94fdc-zvq97" Oct 31 00:49:55.658056 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:55.658026 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb02bae7-ad38-4a84-b668-2871c1c581e9-plugins-config-volume\") pod \"payload-processing-e2e-mt-7d5f4de7-cf6c94fdc-zvq97\" (UID: \"cb02bae7-ad38-4a84-b668-2871c1c581e9\") " pod="openshift-ingress/payload-processing-e2e-mt-7d5f4de7-cf6c94fdc-zvq97" Oct 31 00:49:55.676393 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:55.659229 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb02bae7-ad38-4a84-b668-2871c1c581e9-plugins-config-volume\") pod \"payload-processing-e2e-mt-7d5f4de7-cf6c94fdc-zvq97\" (UID: \"cb02bae7-ad38-4a84-b668-2871c1c581e9\") " pod="openshift-ingress/payload-processing-e2e-mt-7d5f4de7-cf6c94fdc-zvq97" Oct 31 00:49:55.712781 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:55.712708 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fcdf\" (UniqueName: \"kubernetes.io/projected/cb02bae7-ad38-4a84-b668-2871c1c581e9-kube-api-access-7fcdf\") pod \"payload-processing-e2e-mt-7d5f4de7-cf6c94fdc-zvq97\" (UID: \"cb02bae7-ad38-4a84-b668-2871c1c581e9\") " pod="openshift-ingress/payload-processing-e2e-mt-7d5f4de7-cf6c94fdc-zvq97" Oct 31 00:49:55.718275 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:55.716134 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18761079-b183-45ca-9b61-cd9717b0195a" path="/var/lib/kubelet/pods/18761079-b183-45ca-9b61-cd9717b0195a/volumes" Oct 31 00:49:55.799139 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:55.799047 5222 generic.go:356] "Generic (PLEG): container finished" podID="9b5e141c-81c8-497a-aa91-c9e30c14fe30" containerID="11a7dff0b5e0f347d74ad8a8ef7e72fb5229cc58de19ac5416cc2ad439bf177f" exitCode=0 Oct 31 00:49:55.799139 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:55.799119 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-982ae3-k6nrb" event={"ID":"9b5e141c-81c8-497a-aa91-c9e30c14fe30","Type":"ContainerDied","Data":"11a7dff0b5e0f347d74ad8a8ef7e72fb5229cc58de19ac5416cc2ad439bf177f"} Oct 31 00:49:55.821834 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:55.821802 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7d5f4de7-cf6c94fdc-zvq97" Oct 31 00:49:55.912721 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:55.912674 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7d5f4de7-ffd6698f9-2r7sb" Oct 31 00:49:56.198342 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:49:56.198311 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb02bae7_ad38_4a84_b668_2871c1c581e9.slice/crio-8ca82d22b5b4b85076e4a96d7130992656d373dac069b875b66492399212444e WatchSource:0}: Error finding container 8ca82d22b5b4b85076e4a96d7130992656d373dac069b875b66492399212444e: Status 404 returned error can't find the container with id 8ca82d22b5b4b85076e4a96d7130992656d373dac069b875b66492399212444e Oct 31 00:49:56.202356 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:56.202335 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7d5f4de7-cf6c94fdc-zvq97"] Oct 31 00:49:56.300938 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:49:56.300909 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64f10e01_81b0_4bff_a597_19b2a0106639.slice/crio-5ed0a82869793a182cd000fb65487024a06e9c2be3baa3af0ac4518e31562747 WatchSource:0}: Error finding container 5ed0a82869793a182cd000fb65487024a06e9c2be3baa3af0ac4518e31562747: Status 404 returned error can't find the container with id 5ed0a82869793a182cd000fb65487024a06e9c2be3baa3af0ac4518e31562747 Oct 31 00:49:56.304643 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:56.304539 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7d5f4de7-ffd6698f9-2r7sb"] Oct 31 00:49:56.808982 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:56.808942 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7d5f4de7-ffd6698f9-2r7sb" event={"ID":"64f10e01-81b0-4bff-a597-19b2a0106639","Type":"ContainerStarted","Data":"79e79d70ad957d72fd45d3ddb364ea2e937e7e0e09a3831dabfb40a822b0dcb0"} Oct 31 00:49:56.809195 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:56.808999 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7d5f4de7-ffd6698f9-2r7sb" event={"ID":"64f10e01-81b0-4bff-a597-19b2a0106639","Type":"ContainerStarted","Data":"5ed0a82869793a182cd000fb65487024a06e9c2be3baa3af0ac4518e31562747"} Oct 31 00:49:56.809195 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:56.809084 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-7d5f4de7-ffd6698f9-2r7sb" Oct 31 00:49:56.810783 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:56.810761 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7d5f4de7-cf6c94fdc-zvq97" event={"ID":"cb02bae7-ad38-4a84-b668-2871c1c581e9","Type":"ContainerStarted","Data":"3a3243157e7aef86a98679813352d0987039c4a72e713550f006bccd33dabefc"} Oct 31 00:49:56.810919 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:56.810797 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7d5f4de7-cf6c94fdc-zvq97" event={"ID":"cb02bae7-ad38-4a84-b668-2871c1c581e9","Type":"ContainerStarted","Data":"8ca82d22b5b4b85076e4a96d7130992656d373dac069b875b66492399212444e"} Oct 31 00:49:56.810919 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:56.810849 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-7d5f4de7-cf6c94fdc-zvq97" Oct 31 00:49:56.931899 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:56.931825 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-7d5f4de7-ffd6698f9-2r7sb" podStartSLOduration=1.931810727 podStartE2EDuration="1.931810727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:49:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:49:56.927563457 +0000 UTC m=+2089.398733208" watchObservedRunningTime="2026-10-31 00:49:56.931810727 +0000 UTC m=+2089.402980478" Oct 31 00:49:57.001494 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:57.001468 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-982ae3-k6nrb" Oct 31 00:49:57.067460 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:57.066461 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-7d5f4de7-cf6c94fdc-zvq97" podStartSLOduration=2.0664364060000002 podStartE2EDuration="2.066436406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:49:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:49:57.055505674 +0000 UTC m=+2089.526675424" watchObservedRunningTime="2026-10-31 00:49:57.066436406 +0000 UTC m=+2089.537606157" Oct 31 00:49:57.083889 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:57.083855 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b5e141c-81c8-497a-aa91-c9e30c14fe30-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b5e141c-81c8-497a-aa91-c9e30c14fe30-maas-api-openshift-ca\") pod \"9b5e141c-81c8-497a-aa91-c9e30c14fe30\" (UID: \"9b5e141c-81c8-497a-aa91-c9e30c14fe30\") " Oct 31 00:49:57.084087 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:57.083912 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b5e141c-81c8-497a-aa91-c9e30c14fe30-kube-api-access-mlwj2\" (UniqueName: \"kubernetes.io/projected/9b5e141c-81c8-497a-aa91-c9e30c14fe30-kube-api-access-mlwj2\") pod \"9b5e141c-81c8-497a-aa91-c9e30c14fe30\" (UID: \"9b5e141c-81c8-497a-aa91-c9e30c14fe30\") " Oct 31 00:49:57.084253 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:57.084235 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b5e141c-81c8-497a-aa91-c9e30c14fe30-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b5e141c-81c8-497a-aa91-c9e30c14fe30-maas-api-xks-ca\") pod \"9b5e141c-81c8-497a-aa91-c9e30c14fe30\" (UID: \"9b5e141c-81c8-497a-aa91-c9e30c14fe30\") " Oct 31 00:49:57.085232 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:57.085199 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b5e141c-81c8-497a-aa91-c9e30c14fe30-maas-api-openshift-ca" pod "9b5e141c-81c8-497a-aa91-c9e30c14fe30" (UID: "9b5e141c-81c8-497a-aa91-c9e30c14fe30"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:49:57.087132 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:57.087101 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b5e141c-81c8-497a-aa91-c9e30c14fe30-maas-api-xks-ca" pod "9b5e141c-81c8-497a-aa91-c9e30c14fe30" (UID: "9b5e141c-81c8-497a-aa91-c9e30c14fe30"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:49:57.087590 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:57.087563 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b5e141c-81c8-497a-aa91-c9e30c14fe30-kube-api-access-mlwj2" pod "9b5e141c-81c8-497a-aa91-c9e30c14fe30" (UID: "9b5e141c-81c8-497a-aa91-c9e30c14fe30"). InnerVolumeSpecName "kube-api-access-mlwj2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:49:57.198394 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:57.193923 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b5e141c-81c8-497a-aa91-c9e30c14fe30-maas-api-xks-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:57.198394 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:57.193975 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b5e141c-81c8-497a-aa91-c9e30c14fe30-maas-api-openshift-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:57.198394 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:57.193991 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlwj2\" (UniqueName: \"kubernetes.io/projected/9b5e141c-81c8-497a-aa91-c9e30c14fe30-kube-api-access-mlwj2\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:57.829574 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:57.829530 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-982ae3-k6nrb" event={"ID":"9b5e141c-81c8-497a-aa91-c9e30c14fe30","Type":"ContainerDied","Data":"ea3e1960a6b9930f772c2abd1c362ac000e84fdad1f8fc56f934ded713ece7c6"} Oct 31 00:49:57.830059 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:57.829575 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-982ae3-k6nrb" Oct 31 00:49:57.830059 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:49:57.829622 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ea3e1960a6b9930f772c2abd1c362ac000e84fdad1f8fc56f934ded713ece7c6" Oct 31 00:50:02.768632 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:02.768598 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4ab8e5-79cc98db87-r95fp" Oct 31 00:50:02.769147 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:02.768676 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-4ab8e5-b77879c55-qptzw" Oct 31 00:50:05.917424 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:50:05.915794 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 00:50:07.833320 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:07.833285 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-7d5f4de7-ffd6698f9-2r7sb" Oct 31 00:50:07.833799 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:07.833399 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-7d5f4de7-cf6c94fdc-zvq97" Oct 31 00:50:08.007663 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:08.007635 5222 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h9hsf_0f8c3c03-a1fc-4376-98c6-feeeefa08068/ovn-acl-logging/0.log" Oct 31 00:50:08.014506 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:08.014480 5222 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h9hsf_0f8c3c03-a1fc-4376-98c6-feeeefa08068/ovn-acl-logging/0.log" Oct 31 00:50:09.402212 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:09.402180 5222 scope.go:122] "RemoveContainer" containerID="862386579a94025313386c22d5180c95b20944fd4836a10df96002e8f86ffa8d" Oct 31 00:50:09.852781 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:09.852734 5222 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-982ae3/distinct-w1-982ae3-kserve-84b6f97b84-4hhxk" podUID="c4ffad70-a133-4d21-aa05-c90cf8d5b6e9" containerName="main" containerID="cri-o://7c42913245dd294852346c67d91e6f36c6149d2540daa7580575bb8a00481aa7" gracePeriod=45 Oct 31 00:50:09.866935 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:09.866878 5222 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-982ae3/premium-w1-982ae3-kserve-5c8755f445-qm9bh" podUID="005a2416-8e4f-4a72-8f1a-169a25e0f18c" containerName="main" containerID="cri-o://9d8c49ba4c2fc723b0ffb62c09c7787751c22a1f30c6596085639af71f1a5246" gracePeriod=45 Oct 31 00:50:10.174712 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:10.174685 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-982ae3/distinct-w1-982ae3-kserve-84b6f97b84-4hhxk" Oct 31 00:50:10.178618 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:10.178596 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-982ae3/premium-w1-982ae3-kserve-5c8755f445-qm9bh" Oct 31 00:50:10.276245 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:10.276214 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4ffad70-a133-4d21-aa05-c90cf8d5b6e9-home\" (UniqueName: \"kubernetes.io/empty-dir/c4ffad70-a133-4d21-aa05-c90cf8d5b6e9-home\") pod \"c4ffad70-a133-4d21-aa05-c90cf8d5b6e9\" (UID: \"c4ffad70-a133-4d21-aa05-c90cf8d5b6e9\") " Oct 31 00:50:10.276450 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:10.276262 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4ffad70-a133-4d21-aa05-c90cf8d5b6e9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4ffad70-a133-4d21-aa05-c90cf8d5b6e9-dshm\") pod \"c4ffad70-a133-4d21-aa05-c90cf8d5b6e9\" (UID: \"c4ffad70-a133-4d21-aa05-c90cf8d5b6e9\") " Oct 31 00:50:10.276450 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:10.276348 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/005a2416-8e4f-4a72-8f1a-169a25e0f18c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/005a2416-8e4f-4a72-8f1a-169a25e0f18c-model-cache\") pod \"005a2416-8e4f-4a72-8f1a-169a25e0f18c\" (UID: \"005a2416-8e4f-4a72-8f1a-169a25e0f18c\") " Oct 31 00:50:10.276545 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:10.276462 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/005a2416-8e4f-4a72-8f1a-169a25e0f18c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/005a2416-8e4f-4a72-8f1a-169a25e0f18c-dshm\") pod \"005a2416-8e4f-4a72-8f1a-169a25e0f18c\" (UID: \"005a2416-8e4f-4a72-8f1a-169a25e0f18c\") " Oct 31 00:50:10.276581 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:10.276565 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c4ffad70-a133-4d21-aa05-c90cf8d5b6e9-tls-certs\" (UniqueName: \"kubernetes.io/secret/c4ffad70-a133-4d21-aa05-c90cf8d5b6e9-tls-certs\") pod \"c4ffad70-a133-4d21-aa05-c90cf8d5b6e9\" (UID: \"c4ffad70-a133-4d21-aa05-c90cf8d5b6e9\") " Oct 31 00:50:10.276640 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:10.276621 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/005a2416-8e4f-4a72-8f1a-169a25e0f18c-tls-certs\" (UniqueName: \"kubernetes.io/secret/005a2416-8e4f-4a72-8f1a-169a25e0f18c-tls-certs\") pod \"005a2416-8e4f-4a72-8f1a-169a25e0f18c\" (UID: \"005a2416-8e4f-4a72-8f1a-169a25e0f18c\") " Oct 31 00:50:10.276675 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:10.276656 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/005a2416-8e4f-4a72-8f1a-169a25e0f18c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/005a2416-8e4f-4a72-8f1a-169a25e0f18c-tmp-dir\") pod \"005a2416-8e4f-4a72-8f1a-169a25e0f18c\" (UID: \"005a2416-8e4f-4a72-8f1a-169a25e0f18c\") " Oct 31 00:50:10.276722 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:10.276709 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4ffad70-a133-4d21-aa05-c90cf8d5b6e9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4ffad70-a133-4d21-aa05-c90cf8d5b6e9-model-cache\") pod \"c4ffad70-a133-4d21-aa05-c90cf8d5b6e9\" (UID: \"c4ffad70-a133-4d21-aa05-c90cf8d5b6e9\") " Oct 31 00:50:10.276765 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:10.276746 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/005a2416-8e4f-4a72-8f1a-169a25e0f18c-kube-api-access-2ltmk\" (UniqueName: \"kubernetes.io/projected/005a2416-8e4f-4a72-8f1a-169a25e0f18c-kube-api-access-2ltmk\") pod \"005a2416-8e4f-4a72-8f1a-169a25e0f18c\" (UID: \"005a2416-8e4f-4a72-8f1a-169a25e0f18c\") " Oct 31 00:50:10.276818 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:10.276775 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4ffad70-a133-4d21-aa05-c90cf8d5b6e9-kube-api-access-mcgqj\" (UniqueName: \"kubernetes.io/projected/c4ffad70-a133-4d21-aa05-c90cf8d5b6e9-kube-api-access-mcgqj\") pod \"c4ffad70-a133-4d21-aa05-c90cf8d5b6e9\" (UID: \"c4ffad70-a133-4d21-aa05-c90cf8d5b6e9\") " Oct 31 00:50:10.276874 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:10.276838 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/005a2416-8e4f-4a72-8f1a-169a25e0f18c-home\" (UniqueName: \"kubernetes.io/empty-dir/005a2416-8e4f-4a72-8f1a-169a25e0f18c-home\") pod \"005a2416-8e4f-4a72-8f1a-169a25e0f18c\" (UID: \"005a2416-8e4f-4a72-8f1a-169a25e0f18c\") " Oct 31 00:50:10.276934 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:10.276887 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4ffad70-a133-4d21-aa05-c90cf8d5b6e9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4ffad70-a133-4d21-aa05-c90cf8d5b6e9-tmp-dir\") pod \"c4ffad70-a133-4d21-aa05-c90cf8d5b6e9\" (UID: \"c4ffad70-a133-4d21-aa05-c90cf8d5b6e9\") " Oct 31 00:50:10.277069 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:10.277046 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/005a2416-8e4f-4a72-8f1a-169a25e0f18c-tmp-dir" pod "005a2416-8e4f-4a72-8f1a-169a25e0f18c" (UID: "005a2416-8e4f-4a72-8f1a-169a25e0f18c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:50:10.277629 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:10.277603 5222 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/005a2416-8e4f-4a72-8f1a-169a25e0f18c-tmp-dir\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:10.277809 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:10.277786 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4ffad70-a133-4d21-aa05-c90cf8d5b6e9-tmp-dir" pod "c4ffad70-a133-4d21-aa05-c90cf8d5b6e9" (UID: "c4ffad70-a133-4d21-aa05-c90cf8d5b6e9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:50:10.277938 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:10.277913 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/005a2416-8e4f-4a72-8f1a-169a25e0f18c-model-cache" pod "005a2416-8e4f-4a72-8f1a-169a25e0f18c" (UID: "005a2416-8e4f-4a72-8f1a-169a25e0f18c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:50:10.278138 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:10.278117 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4ffad70-a133-4d21-aa05-c90cf8d5b6e9-model-cache" pod "c4ffad70-a133-4d21-aa05-c90cf8d5b6e9" (UID: "c4ffad70-a133-4d21-aa05-c90cf8d5b6e9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:50:10.278589 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:10.278564 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/005a2416-8e4f-4a72-8f1a-169a25e0f18c-home" pod "005a2416-8e4f-4a72-8f1a-169a25e0f18c" (UID: "005a2416-8e4f-4a72-8f1a-169a25e0f18c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:50:10.278689 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:10.278657 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4ffad70-a133-4d21-aa05-c90cf8d5b6e9-home" pod "c4ffad70-a133-4d21-aa05-c90cf8d5b6e9" (UID: "c4ffad70-a133-4d21-aa05-c90cf8d5b6e9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:50:10.279917 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:10.279872 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4ffad70-a133-4d21-aa05-c90cf8d5b6e9-tls-certs" pod "c4ffad70-a133-4d21-aa05-c90cf8d5b6e9" (UID: "c4ffad70-a133-4d21-aa05-c90cf8d5b6e9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:50:10.280688 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:10.280661 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/005a2416-8e4f-4a72-8f1a-169a25e0f18c-tls-certs" pod "005a2416-8e4f-4a72-8f1a-169a25e0f18c" (UID: "005a2416-8e4f-4a72-8f1a-169a25e0f18c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:50:10.280818 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:10.280792 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4ffad70-a133-4d21-aa05-c90cf8d5b6e9-kube-api-access-mcgqj" pod "c4ffad70-a133-4d21-aa05-c90cf8d5b6e9" (UID: "c4ffad70-a133-4d21-aa05-c90cf8d5b6e9"). InnerVolumeSpecName "kube-api-access-mcgqj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:50:10.281041 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:10.281015 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/005a2416-8e4f-4a72-8f1a-169a25e0f18c-dshm" pod "005a2416-8e4f-4a72-8f1a-169a25e0f18c" (UID: "005a2416-8e4f-4a72-8f1a-169a25e0f18c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:50:10.281870 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:10.281844 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4ffad70-a133-4d21-aa05-c90cf8d5b6e9-dshm" pod "c4ffad70-a133-4d21-aa05-c90cf8d5b6e9" (UID: "c4ffad70-a133-4d21-aa05-c90cf8d5b6e9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:50:10.282528 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:10.282506 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/005a2416-8e4f-4a72-8f1a-169a25e0f18c-kube-api-access-2ltmk" pod "005a2416-8e4f-4a72-8f1a-169a25e0f18c" (UID: "005a2416-8e4f-4a72-8f1a-169a25e0f18c"). InnerVolumeSpecName "kube-api-access-2ltmk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:50:10.378956 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:10.378861 5222 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4ffad70-a133-4d21-aa05-c90cf8d5b6e9-model-cache\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:10.378956 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:10.378893 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ltmk\" (UniqueName: \"kubernetes.io/projected/005a2416-8e4f-4a72-8f1a-169a25e0f18c-kube-api-access-2ltmk\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:10.378956 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:10.378905 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcgqj\" (UniqueName: \"kubernetes.io/projected/c4ffad70-a133-4d21-aa05-c90cf8d5b6e9-kube-api-access-mcgqj\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:10.378956 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:10.378914 5222 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/005a2416-8e4f-4a72-8f1a-169a25e0f18c-home\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:10.378956 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:10.378923 5222 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4ffad70-a133-4d21-aa05-c90cf8d5b6e9-tmp-dir\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:10.378956 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:10.378931 5222 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4ffad70-a133-4d21-aa05-c90cf8d5b6e9-home\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:10.378956 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:10.378938 5222 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4ffad70-a133-4d21-aa05-c90cf8d5b6e9-dshm\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:10.378956 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:10.378945 5222 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/005a2416-8e4f-4a72-8f1a-169a25e0f18c-model-cache\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:10.378956 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:10.378953 5222 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/005a2416-8e4f-4a72-8f1a-169a25e0f18c-dshm\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:10.378956 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:10.378963 5222 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4ffad70-a133-4d21-aa05-c90cf8d5b6e9-tls-certs\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:10.379656 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:10.378971 5222 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/005a2416-8e4f-4a72-8f1a-169a25e0f18c-tls-certs\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:10.972866 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:10.972823 5222 generic.go:356] "Generic (PLEG): container finished" podID="c4ffad70-a133-4d21-aa05-c90cf8d5b6e9" containerID="7c42913245dd294852346c67d91e6f36c6149d2540daa7580575bb8a00481aa7" exitCode=0 Oct 31 00:50:10.973339 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:10.972891 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-982ae3/distinct-w1-982ae3-kserve-84b6f97b84-4hhxk" Oct 31 00:50:10.973339 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:10.972904 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-982ae3/distinct-w1-982ae3-kserve-84b6f97b84-4hhxk" event={"ID":"c4ffad70-a133-4d21-aa05-c90cf8d5b6e9","Type":"ContainerDied","Data":"7c42913245dd294852346c67d91e6f36c6149d2540daa7580575bb8a00481aa7"} Oct 31 00:50:10.973339 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:10.972950 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-982ae3/distinct-w1-982ae3-kserve-84b6f97b84-4hhxk" event={"ID":"c4ffad70-a133-4d21-aa05-c90cf8d5b6e9","Type":"ContainerDied","Data":"6201dbb72c8650fdbe32974a5986f22d741104431323a0bf389c3e580c44111a"} Oct 31 00:50:10.973339 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:10.972968 5222 scope.go:122] "RemoveContainer" containerID="7c42913245dd294852346c67d91e6f36c6149d2540daa7580575bb8a00481aa7" Oct 31 00:50:10.974858 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:10.974833 5222 generic.go:356] "Generic (PLEG): container finished" podID="005a2416-8e4f-4a72-8f1a-169a25e0f18c" containerID="9d8c49ba4c2fc723b0ffb62c09c7787751c22a1f30c6596085639af71f1a5246" exitCode=0 Oct 31 00:50:10.974956 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:10.974901 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-982ae3/premium-w1-982ae3-kserve-5c8755f445-qm9bh" Oct 31 00:50:10.974956 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:10.974912 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-982ae3/premium-w1-982ae3-kserve-5c8755f445-qm9bh" event={"ID":"005a2416-8e4f-4a72-8f1a-169a25e0f18c","Type":"ContainerDied","Data":"9d8c49ba4c2fc723b0ffb62c09c7787751c22a1f30c6596085639af71f1a5246"} Oct 31 00:50:10.974956 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:10.974939 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-982ae3/premium-w1-982ae3-kserve-5c8755f445-qm9bh" event={"ID":"005a2416-8e4f-4a72-8f1a-169a25e0f18c","Type":"ContainerDied","Data":"c9da8f5009e90c30caa2490f14884ec1ee2911228ff8c903ae49fe0bf6349bd5"} Oct 31 00:50:10.986495 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:10.986475 5222 scope.go:122] "RemoveContainer" containerID="7c42913245dd294852346c67d91e6f36c6149d2540daa7580575bb8a00481aa7" Oct 31 00:50:10.986892 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:50:10.986871 5222 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c42913245dd294852346c67d91e6f36c6149d2540daa7580575bb8a00481aa7\": container with ID starting with 7c42913245dd294852346c67d91e6f36c6149d2540daa7580575bb8a00481aa7 not found: ID does not exist" containerID="7c42913245dd294852346c67d91e6f36c6149d2540daa7580575bb8a00481aa7" Oct 31 00:50:10.986989 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:10.986901 5222 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c42913245dd294852346c67d91e6f36c6149d2540daa7580575bb8a00481aa7"} err="failed to get container status \"7c42913245dd294852346c67d91e6f36c6149d2540daa7580575bb8a00481aa7\": rpc error: code = NotFound desc = could not find container \"7c42913245dd294852346c67d91e6f36c6149d2540daa7580575bb8a00481aa7\": container with ID starting with 7c42913245dd294852346c67d91e6f36c6149d2540daa7580575bb8a00481aa7 not found: ID does not exist" Oct 31 00:50:10.986989 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:10.986922 5222 scope.go:122] "RemoveContainer" containerID="9d8c49ba4c2fc723b0ffb62c09c7787751c22a1f30c6596085639af71f1a5246" Oct 31 00:50:10.997248 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:10.997223 5222 scope.go:122] "RemoveContainer" containerID="9d8c49ba4c2fc723b0ffb62c09c7787751c22a1f30c6596085639af71f1a5246" Oct 31 00:50:10.997690 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:50:10.997667 5222 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d8c49ba4c2fc723b0ffb62c09c7787751c22a1f30c6596085639af71f1a5246\": container with ID starting with 9d8c49ba4c2fc723b0ffb62c09c7787751c22a1f30c6596085639af71f1a5246 not found: ID does not exist" containerID="9d8c49ba4c2fc723b0ffb62c09c7787751c22a1f30c6596085639af71f1a5246" Oct 31 00:50:10.997776 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:10.997700 5222 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d8c49ba4c2fc723b0ffb62c09c7787751c22a1f30c6596085639af71f1a5246"} err="failed to get container status \"9d8c49ba4c2fc723b0ffb62c09c7787751c22a1f30c6596085639af71f1a5246\": rpc error: code = NotFound desc = could not find container \"9d8c49ba4c2fc723b0ffb62c09c7787751c22a1f30c6596085639af71f1a5246\": container with ID starting with 9d8c49ba4c2fc723b0ffb62c09c7787751c22a1f30c6596085639af71f1a5246 not found: ID does not exist" Oct 31 00:50:11.143645 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:11.143560 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-982ae3/premium-w1-982ae3-kserve-5c8755f445-qm9bh"] Oct 31 00:50:11.194287 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:11.194254 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-982ae3/premium-w1-982ae3-kserve-5c8755f445-qm9bh"] Oct 31 00:50:11.261804 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:11.261768 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-982ae3/distinct-w1-982ae3-kserve-84b6f97b84-4hhxk"] Oct 31 00:50:11.266591 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:11.266562 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-982ae3/distinct-w1-982ae3-kserve-84b6f97b84-4hhxk"] Oct 31 00:50:11.658065 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:11.658028 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="005a2416-8e4f-4a72-8f1a-169a25e0f18c" path="/var/lib/kubelet/pods/005a2416-8e4f-4a72-8f1a-169a25e0f18c/volumes" Oct 31 00:50:11.658558 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:11.658543 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4ffad70-a133-4d21-aa05-c90cf8d5b6e9" path="/var/lib/kubelet/pods/c4ffad70-a133-4d21-aa05-c90cf8d5b6e9/volumes" Oct 31 00:50:19.415117 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:19.415077 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-gxxl8"] Oct 31 00:50:19.415800 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:19.415782 5222 cpu_manager.go:414] "containerMap: removing container" podUID="9b5e141c-81c8-497a-aa91-c9e30c14fe30" containerName="revoke-keys" Oct 31 00:50:19.415843 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:19.415802 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b5e141c-81c8-497a-aa91-c9e30c14fe30" containerName="revoke-keys" Oct 31 00:50:19.415843 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:19.415815 5222 cpu_manager.go:414] "containerMap: removing container" podUID="c4ffad70-a133-4d21-aa05-c90cf8d5b6e9" containerName="main" Oct 31 00:50:19.415843 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:19.415820 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4ffad70-a133-4d21-aa05-c90cf8d5b6e9" containerName="main" Oct 31 00:50:19.415948 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:19.415845 5222 cpu_manager.go:414] "containerMap: removing container" podUID="005a2416-8e4f-4a72-8f1a-169a25e0f18c" containerName="main" Oct 31 00:50:19.415948 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:19.415850 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="005a2416-8e4f-4a72-8f1a-169a25e0f18c" containerName="main" Oct 31 00:50:19.416061 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:19.415965 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b5e141c-81c8-497a-aa91-c9e30c14fe30" containerName="revoke-keys" Oct 31 00:50:19.416061 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:19.415975 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4ffad70-a133-4d21-aa05-c90cf8d5b6e9" containerName="main" Oct 31 00:50:19.416061 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:19.415984 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="005a2416-8e4f-4a72-8f1a-169a25e0f18c" containerName="main" Oct 31 00:50:19.421009 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:19.420986 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-gxxl8" Oct 31 00:50:19.474931 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:19.474903 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-982ae3-openshift-default-dockercfg-cwtrb\"" Oct 31 00:50:19.593231 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:19.593190 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-gxxl8"] Oct 31 00:50:19.608708 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:19.608663 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86d77382-1376-49f5-abb0-165df5b8cd9f-istio-data\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-gxxl8\" (UID: \"86d77382-1376-49f5-abb0-165df5b8cd9f\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-gxxl8" Oct 31 00:50:19.608893 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:19.608809 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86d77382-1376-49f5-abb0-165df5b8cd9f-istio-token\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-gxxl8\" (UID: \"86d77382-1376-49f5-abb0-165df5b8cd9f\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-gxxl8" Oct 31 00:50:19.608893 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:19.608858 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86d77382-1376-49f5-abb0-165df5b8cd9f-workload-certs\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-gxxl8\" (UID: \"86d77382-1376-49f5-abb0-165df5b8cd9f\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-gxxl8" Oct 31 00:50:19.609035 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:19.608941 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86d77382-1376-49f5-abb0-165df5b8cd9f-workload-socket\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-gxxl8\" (UID: \"86d77382-1376-49f5-abb0-165df5b8cd9f\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-gxxl8" Oct 31 00:50:19.609035 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:19.609018 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86d77382-1376-49f5-abb0-165df5b8cd9f-istiod-ca-cert\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-gxxl8\" (UID: \"86d77382-1376-49f5-abb0-165df5b8cd9f\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-gxxl8" Oct 31 00:50:19.609141 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:19.609082 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86d77382-1376-49f5-abb0-165df5b8cd9f-istio-podinfo\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-gxxl8\" (UID: \"86d77382-1376-49f5-abb0-165df5b8cd9f\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-gxxl8" Oct 31 00:50:19.609141 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:19.609112 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgmr8\" (UniqueName: \"kubernetes.io/projected/86d77382-1376-49f5-abb0-165df5b8cd9f-kube-api-access-sgmr8\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-gxxl8\" (UID: \"86d77382-1376-49f5-abb0-165df5b8cd9f\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-gxxl8" Oct 31 00:50:19.609245 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:19.609175 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86d77382-1376-49f5-abb0-165df5b8cd9f-credential-socket\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-gxxl8\" (UID: \"86d77382-1376-49f5-abb0-165df5b8cd9f\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-gxxl8" Oct 31 00:50:19.609245 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:19.609197 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86d77382-1376-49f5-abb0-165df5b8cd9f-istio-envoy\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-gxxl8\" (UID: \"86d77382-1376-49f5-abb0-165df5b8cd9f\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-gxxl8" Oct 31 00:50:19.710122 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:19.710022 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86d77382-1376-49f5-abb0-165df5b8cd9f-credential-socket\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-gxxl8\" (UID: \"86d77382-1376-49f5-abb0-165df5b8cd9f\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-gxxl8" Oct 31 00:50:19.710122 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:19.710067 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86d77382-1376-49f5-abb0-165df5b8cd9f-istio-envoy\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-gxxl8\" (UID: \"86d77382-1376-49f5-abb0-165df5b8cd9f\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-gxxl8" Oct 31 00:50:19.710425 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:19.710187 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86d77382-1376-49f5-abb0-165df5b8cd9f-istio-data\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-gxxl8\" (UID: \"86d77382-1376-49f5-abb0-165df5b8cd9f\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-gxxl8" Oct 31 00:50:19.710425 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:19.710245 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86d77382-1376-49f5-abb0-165df5b8cd9f-istio-token\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-gxxl8\" (UID: \"86d77382-1376-49f5-abb0-165df5b8cd9f\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-gxxl8" Oct 31 00:50:19.710425 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:19.710276 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86d77382-1376-49f5-abb0-165df5b8cd9f-workload-certs\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-gxxl8\" (UID: \"86d77382-1376-49f5-abb0-165df5b8cd9f\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-gxxl8" Oct 31 00:50:19.710425 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:19.710345 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86d77382-1376-49f5-abb0-165df5b8cd9f-workload-socket\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-gxxl8\" (UID: \"86d77382-1376-49f5-abb0-165df5b8cd9f\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-gxxl8" Oct 31 00:50:19.710673 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:19.710438 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86d77382-1376-49f5-abb0-165df5b8cd9f-istiod-ca-cert\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-gxxl8\" (UID: \"86d77382-1376-49f5-abb0-165df5b8cd9f\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-gxxl8" Oct 31 00:50:19.710673 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:19.710501 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86d77382-1376-49f5-abb0-165df5b8cd9f-credential-socket\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-gxxl8\" (UID: \"86d77382-1376-49f5-abb0-165df5b8cd9f\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-gxxl8" Oct 31 00:50:19.710673 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:19.710516 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86d77382-1376-49f5-abb0-165df5b8cd9f-istio-podinfo\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-gxxl8\" (UID: \"86d77382-1376-49f5-abb0-165df5b8cd9f\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-gxxl8" Oct 31 00:50:19.710673 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:19.710545 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgmr8\" (UniqueName: \"kubernetes.io/projected/86d77382-1376-49f5-abb0-165df5b8cd9f-kube-api-access-sgmr8\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-gxxl8\" (UID: \"86d77382-1376-49f5-abb0-165df5b8cd9f\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-gxxl8" Oct 31 00:50:19.710673 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:19.710565 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86d77382-1376-49f5-abb0-165df5b8cd9f-istio-data\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-gxxl8\" (UID: \"86d77382-1376-49f5-abb0-165df5b8cd9f\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-gxxl8" Oct 31 00:50:19.710673 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:19.710663 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86d77382-1376-49f5-abb0-165df5b8cd9f-workload-certs\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-gxxl8\" (UID: \"86d77382-1376-49f5-abb0-165df5b8cd9f\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-gxxl8" Oct 31 00:50:19.710989 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:19.710871 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86d77382-1376-49f5-abb0-165df5b8cd9f-workload-socket\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-gxxl8\" (UID: \"86d77382-1376-49f5-abb0-165df5b8cd9f\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-gxxl8" Oct 31 00:50:19.711197 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:19.711174 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86d77382-1376-49f5-abb0-165df5b8cd9f-istiod-ca-cert\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-gxxl8\" (UID: \"86d77382-1376-49f5-abb0-165df5b8cd9f\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-gxxl8" Oct 31 00:50:19.712722 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:19.712697 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86d77382-1376-49f5-abb0-165df5b8cd9f-istio-envoy\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-gxxl8\" (UID: \"86d77382-1376-49f5-abb0-165df5b8cd9f\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-gxxl8" Oct 31 00:50:19.713164 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:19.713140 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86d77382-1376-49f5-abb0-165df5b8cd9f-istio-podinfo\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-gxxl8\" (UID: \"86d77382-1376-49f5-abb0-165df5b8cd9f\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-gxxl8" Oct 31 00:50:19.773119 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:19.773086 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86d77382-1376-49f5-abb0-165df5b8cd9f-istio-token\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-gxxl8\" (UID: \"86d77382-1376-49f5-abb0-165df5b8cd9f\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-gxxl8" Oct 31 00:50:19.773281 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:19.773224 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgmr8\" (UniqueName: \"kubernetes.io/projected/86d77382-1376-49f5-abb0-165df5b8cd9f-kube-api-access-sgmr8\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-gxxl8\" (UID: \"86d77382-1376-49f5-abb0-165df5b8cd9f\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-gxxl8" Oct 31 00:50:20.036617 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:20.036582 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-gxxl8" Oct 31 00:50:20.474211 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:50:20.474182 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86d77382_1376_49f5_abb0_165df5b8cd9f.slice/crio-120df8b9096b816a76274294a3f61e5f137354df13ea20b519cf46f4a2ee2d5a WatchSource:0}: Error finding container 120df8b9096b816a76274294a3f61e5f137354df13ea20b519cf46f4a2ee2d5a: Status 404 returned error can't find the container with id 120df8b9096b816a76274294a3f61e5f137354df13ea20b519cf46f4a2ee2d5a Oct 31 00:50:20.477203 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:20.477152 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:50:20.477325 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:20.477243 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:50:20.477325 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:20.477283 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:50:20.477945 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:20.477838 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-gxxl8"] Oct 31 00:50:21.056362 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:21.056328 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-gxxl8" event={"ID":"86d77382-1376-49f5-abb0-165df5b8cd9f","Type":"ContainerStarted","Data":"a0f60e1e18927ccbd4a68231441f63ac069243facce24bc79fbc72072276d281"} Oct 31 00:50:21.056585 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:21.056394 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-gxxl8" event={"ID":"86d77382-1376-49f5-abb0-165df5b8cd9f","Type":"ContainerStarted","Data":"120df8b9096b816a76274294a3f61e5f137354df13ea20b519cf46f4a2ee2d5a"} Oct 31 00:50:21.121828 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:21.121796 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-79tk8"] Oct 31 00:50:21.182953 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:21.182896 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-gxxl8" podStartSLOduration=2.182881746 podStartE2EDuration="2.182881746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:50:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:50:21.179171978 +0000 UTC m=+2113.650341738" watchObservedRunningTime="2026-10-31 00:50:21.182881746 +0000 UTC m=+2113.654051495" Oct 31 00:50:22.036826 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:22.036791 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-gxxl8" Oct 31 00:50:22.054418 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:22.053524 5222 patch_prober.go:29] interesting pod/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-gxxl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body= Oct 31 00:50:22.054418 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:22.053595 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-gxxl8" podUID="86d77382-1376-49f5-abb0-165df5b8cd9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" Oct 31 00:50:23.047397 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:23.037749 5222 patch_prober.go:29] interesting pod/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-gxxl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body= Oct 31 00:50:23.047397 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:23.037830 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-gxxl8" podUID="86d77382-1376-49f5-abb0-165df5b8cd9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" Oct 31 00:50:24.051397 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:24.037649 5222 patch_prober.go:29] interesting pod/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-gxxl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body= Oct 31 00:50:24.051397 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:24.037730 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-gxxl8" podUID="86d77382-1376-49f5-abb0-165df5b8cd9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" Oct 31 00:50:25.062862 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:25.062813 5222 patch_prober.go:29] interesting pod/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-gxxl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body= Oct 31 00:50:25.063415 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:25.062899 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-gxxl8" podUID="86d77382-1376-49f5-abb0-165df5b8cd9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" Oct 31 00:50:26.059397 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:26.049778 5222 patch_prober.go:29] interesting pod/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-gxxl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body= Oct 31 00:50:26.059397 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:26.049870 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-gxxl8" podUID="86d77382-1376-49f5-abb0-165df5b8cd9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" Oct 31 00:50:27.037335 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:27.037302 5222 patch_prober.go:29] interesting pod/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-gxxl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body= Oct 31 00:50:27.037764 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:27.037360 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-gxxl8" podUID="86d77382-1376-49f5-abb0-165df5b8cd9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" Oct 31 00:50:28.042659 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:28.042614 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-gxxl8" Oct 31 00:50:28.043161 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:28.043097 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-gxxl8" Oct 31 00:50:28.044005 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:28.043984 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-gxxl8" Oct 31 00:50:30.945199 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:30.945160 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-f32ec2-vbs5g"] Oct 31 00:50:30.996160 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:30.996124 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-f32ec2-vbs5g"] Oct 31 00:50:31.622029 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:31.621992 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5b4db8-c2v24"] Oct 31 00:50:31.626632 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:31.626597 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5b4db8-c2v24" Oct 31 00:50:31.668008 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:31.667969 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a119a257-ffd3-4573-ac30-d8ebee73908a" path="/var/lib/kubelet/pods/a119a257-ffd3-4573-ac30-d8ebee73908a/volumes" Oct 31 00:50:31.680965 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:31.680921 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5b4db8-c2v24"] Oct 31 00:50:31.707955 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:31.707908 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7c13393-b6f2-4212-8c35-4564200ab329-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-5b4db8-c2v24\" (UID: \"f7c13393-b6f2-4212-8c35-4564200ab329\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5b4db8-c2v24" Oct 31 00:50:31.708166 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:31.708032 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2k5js\" (UniqueName: \"kubernetes.io/projected/f7c13393-b6f2-4212-8c35-4564200ab329-kube-api-access-2k5js\") pod \"maas-api-revoke-keys-e2e-api-a-5b4db8-c2v24\" (UID: \"f7c13393-b6f2-4212-8c35-4564200ab329\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5b4db8-c2v24" Oct 31 00:50:31.708280 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:31.708261 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7c13393-b6f2-4212-8c35-4564200ab329-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-5b4db8-c2v24\" (UID: \"f7c13393-b6f2-4212-8c35-4564200ab329\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5b4db8-c2v24" Oct 31 00:50:31.726873 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:31.726839 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:50:31.726873 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:31.726840 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:50:31.727080 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:31.726896 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:50:31.734863 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:31.734830 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfz9z\"" Oct 31 00:50:31.810058 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:31.810022 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7c13393-b6f2-4212-8c35-4564200ab329-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-5b4db8-c2v24\" (UID: \"f7c13393-b6f2-4212-8c35-4564200ab329\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5b4db8-c2v24" Oct 31 00:50:31.810204 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:31.810124 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7c13393-b6f2-4212-8c35-4564200ab329-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-5b4db8-c2v24\" (UID: \"f7c13393-b6f2-4212-8c35-4564200ab329\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5b4db8-c2v24" Oct 31 00:50:31.810204 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:31.810164 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2k5js\" (UniqueName: \"kubernetes.io/projected/f7c13393-b6f2-4212-8c35-4564200ab329-kube-api-access-2k5js\") pod \"maas-api-revoke-keys-e2e-api-a-5b4db8-c2v24\" (UID: \"f7c13393-b6f2-4212-8c35-4564200ab329\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5b4db8-c2v24" Oct 31 00:50:31.810841 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:31.810817 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7c13393-b6f2-4212-8c35-4564200ab329-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-5b4db8-c2v24\" (UID: \"f7c13393-b6f2-4212-8c35-4564200ab329\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5b4db8-c2v24" Oct 31 00:50:31.812667 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:31.812645 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7c13393-b6f2-4212-8c35-4564200ab329-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-5b4db8-c2v24\" (UID: \"f7c13393-b6f2-4212-8c35-4564200ab329\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5b4db8-c2v24" Oct 31 00:50:31.873181 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:31.873106 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2k5js\" (UniqueName: \"kubernetes.io/projected/f7c13393-b6f2-4212-8c35-4564200ab329-kube-api-access-2k5js\") pod \"maas-api-revoke-keys-e2e-api-a-5b4db8-c2v24\" (UID: \"f7c13393-b6f2-4212-8c35-4564200ab329\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5b4db8-c2v24" Oct 31 00:50:31.943108 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:31.943070 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5b4db8-c2v24" Oct 31 00:50:32.420129 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:50:32.420099 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7c13393_b6f2_4212_8c35_4564200ab329.slice/crio-18326ba6def387f49aa65b7f7ec071807a0d439ad98f0bfdfec3d56d4cd9e085 WatchSource:0}: Error finding container 18326ba6def387f49aa65b7f7ec071807a0d439ad98f0bfdfec3d56d4cd9e085: Status 404 returned error can't find the container with id 18326ba6def387f49aa65b7f7ec071807a0d439ad98f0bfdfec3d56d4cd9e085 Oct 31 00:50:32.423612 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:32.423589 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5b4db8-c2v24"] Oct 31 00:50:33.155228 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:33.155182 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5b4db8-c2v24" event={"ID":"f7c13393-b6f2-4212-8c35-4564200ab329","Type":"ContainerStarted","Data":"53a6874622882ef8b0f3692d4fafd3f8c48effb925ff98d449556b45fa93f1b9"} Oct 31 00:50:33.155433 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:33.155254 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5b4db8-c2v24" event={"ID":"f7c13393-b6f2-4212-8c35-4564200ab329","Type":"ContainerStarted","Data":"18326ba6def387f49aa65b7f7ec071807a0d439ad98f0bfdfec3d56d4cd9e085"} Oct 31 00:50:33.292429 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:33.292343 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5b4db8-c2v24" podStartSLOduration=2.292325782 podStartE2EDuration="2.292325782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:50:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:50:33.292121531 +0000 UTC m=+2125.763291284" watchObservedRunningTime="2026-10-31 00:50:33.292325782 +0000 UTC m=+2125.763495529" Oct 31 00:50:34.165216 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:34.165180 5222 generic.go:356] "Generic (PLEG): container finished" podID="f7c13393-b6f2-4212-8c35-4564200ab329" containerID="53a6874622882ef8b0f3692d4fafd3f8c48effb925ff98d449556b45fa93f1b9" exitCode=0 Oct 31 00:50:34.165719 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:34.165265 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5b4db8-c2v24" event={"ID":"f7c13393-b6f2-4212-8c35-4564200ab329","Type":"ContainerDied","Data":"53a6874622882ef8b0f3692d4fafd3f8c48effb925ff98d449556b45fa93f1b9"} Oct 31 00:50:35.336742 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:35.336714 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5b4db8-c2v24" Oct 31 00:50:35.372828 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:35.372797 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7c13393-b6f2-4212-8c35-4564200ab329-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7c13393-b6f2-4212-8c35-4564200ab329-maas-api-openshift-ca\") pod \"f7c13393-b6f2-4212-8c35-4564200ab329\" (UID: \"f7c13393-b6f2-4212-8c35-4564200ab329\") " Oct 31 00:50:35.372828 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:35.372834 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7c13393-b6f2-4212-8c35-4564200ab329-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7c13393-b6f2-4212-8c35-4564200ab329-maas-api-xks-ca\") pod \"f7c13393-b6f2-4212-8c35-4564200ab329\" (UID: \"f7c13393-b6f2-4212-8c35-4564200ab329\") " Oct 31 00:50:35.373088 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:35.373069 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7c13393-b6f2-4212-8c35-4564200ab329-kube-api-access-2k5js\" (UniqueName: \"kubernetes.io/projected/f7c13393-b6f2-4212-8c35-4564200ab329-kube-api-access-2k5js\") pod \"f7c13393-b6f2-4212-8c35-4564200ab329\" (UID: \"f7c13393-b6f2-4212-8c35-4564200ab329\") " Oct 31 00:50:35.373239 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:35.373213 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7c13393-b6f2-4212-8c35-4564200ab329-maas-api-openshift-ca" pod "f7c13393-b6f2-4212-8c35-4564200ab329" (UID: "f7c13393-b6f2-4212-8c35-4564200ab329"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:50:35.373605 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:35.373588 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7c13393-b6f2-4212-8c35-4564200ab329-maas-api-openshift-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:35.375245 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:35.375213 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7c13393-b6f2-4212-8c35-4564200ab329-maas-api-xks-ca" pod "f7c13393-b6f2-4212-8c35-4564200ab329" (UID: "f7c13393-b6f2-4212-8c35-4564200ab329"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:50:35.375399 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:35.375364 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7c13393-b6f2-4212-8c35-4564200ab329-kube-api-access-2k5js" pod "f7c13393-b6f2-4212-8c35-4564200ab329" (UID: "f7c13393-b6f2-4212-8c35-4564200ab329"). InnerVolumeSpecName "kube-api-access-2k5js". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:50:35.475324 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:35.475239 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2k5js\" (UniqueName: \"kubernetes.io/projected/f7c13393-b6f2-4212-8c35-4564200ab329-kube-api-access-2k5js\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:35.475324 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:35.475268 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7c13393-b6f2-4212-8c35-4564200ab329-maas-api-xks-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:35.892083 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:50:35.892042 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 00:50:36.181003 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:36.180911 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5b4db8-c2v24" Oct 31 00:50:36.181003 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:36.180960 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5b4db8-c2v24" event={"ID":"f7c13393-b6f2-4212-8c35-4564200ab329","Type":"ContainerDied","Data":"18326ba6def387f49aa65b7f7ec071807a0d439ad98f0bfdfec3d56d4cd9e085"} Oct 31 00:50:36.181235 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:36.181059 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="18326ba6def387f49aa65b7f7ec071807a0d439ad98f0bfdfec3d56d4cd9e085" Oct 31 00:50:38.724511 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:38.724473 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e9c3c1-rpj5q"] Oct 31 00:50:38.725157 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:38.725144 5222 cpu_manager.go:414] "containerMap: removing container" podUID="f7c13393-b6f2-4212-8c35-4564200ab329" containerName="revoke-keys" Oct 31 00:50:38.725226 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:38.725158 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7c13393-b6f2-4212-8c35-4564200ab329" containerName="revoke-keys" Oct 31 00:50:38.725301 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:38.725292 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7c13393-b6f2-4212-8c35-4564200ab329" containerName="revoke-keys" Oct 31 00:50:38.728969 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:38.728950 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e9c3c1-rpj5q" Oct 31 00:50:38.781643 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:38.781609 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e9c3c1-rpj5q"] Oct 31 00:50:38.782676 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:38.782648 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:50:38.823844 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:38.823809 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a5bb0fe-0d5e-48da-bbee-839acf6a4b64-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-e9c3c1-rpj5q\" (UID: \"6a5bb0fe-0d5e-48da-bbee-839acf6a4b64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e9c3c1-rpj5q" Oct 31 00:50:38.824032 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:38.823932 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a5bb0fe-0d5e-48da-bbee-839acf6a4b64-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-e9c3c1-rpj5q\" (UID: \"6a5bb0fe-0d5e-48da-bbee-839acf6a4b64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e9c3c1-rpj5q" Oct 31 00:50:38.824032 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:38.824005 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scfkk\" (UniqueName: \"kubernetes.io/projected/6a5bb0fe-0d5e-48da-bbee-839acf6a4b64-kube-api-access-scfkk\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-e9c3c1-rpj5q\" (UID: \"6a5bb0fe-0d5e-48da-bbee-839acf6a4b64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e9c3c1-rpj5q" Oct 31 00:50:38.829044 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:38.829020 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:50:38.829214 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:38.829087 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:50:38.829335 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:38.829315 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfz9z\"" Oct 31 00:50:38.925802 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:38.925767 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a5bb0fe-0d5e-48da-bbee-839acf6a4b64-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-e9c3c1-rpj5q\" (UID: \"6a5bb0fe-0d5e-48da-bbee-839acf6a4b64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e9c3c1-rpj5q" Oct 31 00:50:38.926000 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:38.925840 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a5bb0fe-0d5e-48da-bbee-839acf6a4b64-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-e9c3c1-rpj5q\" (UID: \"6a5bb0fe-0d5e-48da-bbee-839acf6a4b64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e9c3c1-rpj5q" Oct 31 00:50:38.926000 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:38.925902 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scfkk\" (UniqueName: \"kubernetes.io/projected/6a5bb0fe-0d5e-48da-bbee-839acf6a4b64-kube-api-access-scfkk\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-e9c3c1-rpj5q\" (UID: \"6a5bb0fe-0d5e-48da-bbee-839acf6a4b64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e9c3c1-rpj5q" Oct 31 00:50:38.926566 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:38.926537 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a5bb0fe-0d5e-48da-bbee-839acf6a4b64-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-e9c3c1-rpj5q\" (UID: \"6a5bb0fe-0d5e-48da-bbee-839acf6a4b64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e9c3c1-rpj5q" Oct 31 00:50:38.928308 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:38.928288 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a5bb0fe-0d5e-48da-bbee-839acf6a4b64-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-e9c3c1-rpj5q\" (UID: \"6a5bb0fe-0d5e-48da-bbee-839acf6a4b64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e9c3c1-rpj5q" Oct 31 00:50:38.988724 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:38.988642 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scfkk\" (UniqueName: \"kubernetes.io/projected/6a5bb0fe-0d5e-48da-bbee-839acf6a4b64-kube-api-access-scfkk\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-e9c3c1-rpj5q\" (UID: \"6a5bb0fe-0d5e-48da-bbee-839acf6a4b64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e9c3c1-rpj5q" Oct 31 00:50:39.043428 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:39.043358 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e9c3c1-rpj5q" Oct 31 00:50:39.420402 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:50:39.420338 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a5bb0fe_0d5e_48da_bbee_839acf6a4b64.slice/crio-e3e702f7cf99f369555edffc9595189a3b880d82e81a5d6fbb5f114f6dfd42b1 WatchSource:0}: Error finding container e3e702f7cf99f369555edffc9595189a3b880d82e81a5d6fbb5f114f6dfd42b1: Status 404 returned error can't find the container with id e3e702f7cf99f369555edffc9595189a3b880d82e81a5d6fbb5f114f6dfd42b1 Oct 31 00:50:39.423732 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:39.423707 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e9c3c1-rpj5q"] Oct 31 00:50:40.211910 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:40.211878 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e9c3c1-rpj5q" event={"ID":"6a5bb0fe-0d5e-48da-bbee-839acf6a4b64","Type":"ContainerStarted","Data":"d1ab91eb128938b620d59f10568d2acf317af134a639088519edcaf215b4c721"} Oct 31 00:50:40.212470 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:40.211958 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e9c3c1-rpj5q" event={"ID":"6a5bb0fe-0d5e-48da-bbee-839acf6a4b64","Type":"ContainerStarted","Data":"e3e702f7cf99f369555edffc9595189a3b880d82e81a5d6fbb5f114f6dfd42b1"} Oct 31 00:50:40.338775 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:40.338648 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e9c3c1-rpj5q" podStartSLOduration=2.338632182 podStartE2EDuration="2.338632182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:50:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:50:40.336607628 +0000 UTC m=+2132.807777381" watchObservedRunningTime="2026-10-31 00:50:40.338632182 +0000 UTC m=+2132.809801925" Oct 31 00:50:41.221842 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:41.221810 5222 generic.go:356] "Generic (PLEG): container finished" podID="6a5bb0fe-0d5e-48da-bbee-839acf6a4b64" containerID="d1ab91eb128938b620d59f10568d2acf317af134a639088519edcaf215b4c721" exitCode=0 Oct 31 00:50:41.222331 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:41.221899 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e9c3c1-rpj5q" event={"ID":"6a5bb0fe-0d5e-48da-bbee-839acf6a4b64","Type":"ContainerDied","Data":"d1ab91eb128938b620d59f10568d2acf317af134a639088519edcaf215b4c721"} Oct 31 00:50:42.393835 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:42.393810 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e9c3c1-rpj5q" Oct 31 00:50:42.478754 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:42.478715 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a5bb0fe-0d5e-48da-bbee-839acf6a4b64-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a5bb0fe-0d5e-48da-bbee-839acf6a4b64-maas-api-xks-ca\") pod \"6a5bb0fe-0d5e-48da-bbee-839acf6a4b64\" (UID: \"6a5bb0fe-0d5e-48da-bbee-839acf6a4b64\") " Oct 31 00:50:42.478947 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:42.478769 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a5bb0fe-0d5e-48da-bbee-839acf6a4b64-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a5bb0fe-0d5e-48da-bbee-839acf6a4b64-maas-api-openshift-ca\") pod \"6a5bb0fe-0d5e-48da-bbee-839acf6a4b64\" (UID: \"6a5bb0fe-0d5e-48da-bbee-839acf6a4b64\") " Oct 31 00:50:42.479011 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:42.478955 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a5bb0fe-0d5e-48da-bbee-839acf6a4b64-kube-api-access-scfkk\" (UniqueName: \"kubernetes.io/projected/6a5bb0fe-0d5e-48da-bbee-839acf6a4b64-kube-api-access-scfkk\") pod \"6a5bb0fe-0d5e-48da-bbee-839acf6a4b64\" (UID: \"6a5bb0fe-0d5e-48da-bbee-839acf6a4b64\") " Oct 31 00:50:42.479144 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:42.479119 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a5bb0fe-0d5e-48da-bbee-839acf6a4b64-maas-api-openshift-ca" pod "6a5bb0fe-0d5e-48da-bbee-839acf6a4b64" (UID: "6a5bb0fe-0d5e-48da-bbee-839acf6a4b64"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:50:42.479544 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:42.479481 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a5bb0fe-0d5e-48da-bbee-839acf6a4b64-maas-api-openshift-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:42.481047 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:42.481012 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a5bb0fe-0d5e-48da-bbee-839acf6a4b64-maas-api-xks-ca" pod "6a5bb0fe-0d5e-48da-bbee-839acf6a4b64" (UID: "6a5bb0fe-0d5e-48da-bbee-839acf6a4b64"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:50:42.481155 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:42.481096 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a5bb0fe-0d5e-48da-bbee-839acf6a4b64-kube-api-access-scfkk" pod "6a5bb0fe-0d5e-48da-bbee-839acf6a4b64" (UID: "6a5bb0fe-0d5e-48da-bbee-839acf6a4b64"). InnerVolumeSpecName "kube-api-access-scfkk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:50:42.580467 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:42.580365 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scfkk\" (UniqueName: \"kubernetes.io/projected/6a5bb0fe-0d5e-48da-bbee-839acf6a4b64-kube-api-access-scfkk\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:42.580467 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:42.580467 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a5bb0fe-0d5e-48da-bbee-839acf6a4b64-maas-api-xks-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:43.108171 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:43.108133 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5b4db8-78dcbd7945-mhhfh"] Oct 31 00:50:43.210808 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:43.210772 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5b4db8-5c87d84775-wgg4h"] Oct 31 00:50:43.244813 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:43.244775 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e9c3c1-rpj5q" event={"ID":"6a5bb0fe-0d5e-48da-bbee-839acf6a4b64","Type":"ContainerDied","Data":"e3e702f7cf99f369555edffc9595189a3b880d82e81a5d6fbb5f114f6dfd42b1"} Oct 31 00:50:43.245019 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:43.244858 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e3e702f7cf99f369555edffc9595189a3b880d82e81a5d6fbb5f114f6dfd42b1" Oct 31 00:50:43.245082 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:43.245021 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e9c3c1-rpj5q" Oct 31 00:50:48.108883 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:48.108836 5222 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-5b4db8-78dcbd7945-mhhfh" podUID="b3b6cede-ec74-40cf-998e-5ff4238cd411" containerName="payload-processing" containerID="cri-o://0f5fefa9728e0e47b5db0c1f4a2f96e2488d3e6885a10521b85e6b3c76df33ee" gracePeriod=25 Oct 31 00:50:48.211459 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:48.211360 5222 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5b4db8-5c87d84775-wgg4h" podUID="5cc24a99-a137-4f53-aed6-6c914b1b9abb" containerName="payload-pre-processing" containerID="cri-o://02d3057ec60584680437ecc2d5bfa6c2ad3aad953fe048dccc8cfef4394f9f3e" gracePeriod=25 Oct 31 00:50:48.296944 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:48.293733 5222 generic.go:356] "Generic (PLEG): container finished" podID="b3b6cede-ec74-40cf-998e-5ff4238cd411" containerID="0f5fefa9728e0e47b5db0c1f4a2f96e2488d3e6885a10521b85e6b3c76df33ee" exitCode=0 Oct 31 00:50:48.296944 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:48.294113 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-5b4db8-78dcbd7945-mhhfh" event={"ID":"b3b6cede-ec74-40cf-998e-5ff4238cd411","Type":"ContainerDied","Data":"0f5fefa9728e0e47b5db0c1f4a2f96e2488d3e6885a10521b85e6b3c76df33ee"} Oct 31 00:50:48.397108 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:48.397084 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-5b4db8-78dcbd7945-mhhfh" Oct 31 00:50:48.477491 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:48.477461 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3b6cede-ec74-40cf-998e-5ff4238cd411-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3b6cede-ec74-40cf-998e-5ff4238cd411-plugins-config-volume\") pod \"b3b6cede-ec74-40cf-998e-5ff4238cd411\" (UID: \"b3b6cede-ec74-40cf-998e-5ff4238cd411\") " Oct 31 00:50:48.477724 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:48.477711 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3b6cede-ec74-40cf-998e-5ff4238cd411-kube-api-access-cjc4j\" (UniqueName: \"kubernetes.io/projected/b3b6cede-ec74-40cf-998e-5ff4238cd411-kube-api-access-cjc4j\") pod \"b3b6cede-ec74-40cf-998e-5ff4238cd411\" (UID: \"b3b6cede-ec74-40cf-998e-5ff4238cd411\") " Oct 31 00:50:48.477863 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:48.477836 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3b6cede-ec74-40cf-998e-5ff4238cd411-plugins-config-volume" pod "b3b6cede-ec74-40cf-998e-5ff4238cd411" (UID: "b3b6cede-ec74-40cf-998e-5ff4238cd411"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:50:48.478216 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:48.478190 5222 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3b6cede-ec74-40cf-998e-5ff4238cd411-plugins-config-volume\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:48.480228 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:48.480194 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3b6cede-ec74-40cf-998e-5ff4238cd411-kube-api-access-cjc4j" pod "b3b6cede-ec74-40cf-998e-5ff4238cd411" (UID: "b3b6cede-ec74-40cf-998e-5ff4238cd411"). InnerVolumeSpecName "kube-api-access-cjc4j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:50:48.488808 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:48.488779 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5b4db8-5c87d84775-wgg4h" Oct 31 00:50:48.578677 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:48.578643 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5cc24a99-a137-4f53-aed6-6c914b1b9abb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cc24a99-a137-4f53-aed6-6c914b1b9abb-plugins-config-volume\") pod \"5cc24a99-a137-4f53-aed6-6c914b1b9abb\" (UID: \"5cc24a99-a137-4f53-aed6-6c914b1b9abb\") " Oct 31 00:50:48.578975 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:48.578946 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cc24a99-a137-4f53-aed6-6c914b1b9abb-kube-api-access-nmzls\" (UniqueName: \"kubernetes.io/projected/5cc24a99-a137-4f53-aed6-6c914b1b9abb-kube-api-access-nmzls\") pod \"5cc24a99-a137-4f53-aed6-6c914b1b9abb\" (UID: \"5cc24a99-a137-4f53-aed6-6c914b1b9abb\") " Oct 31 00:50:48.579133 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:48.579093 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cc24a99-a137-4f53-aed6-6c914b1b9abb-plugins-config-volume" pod "5cc24a99-a137-4f53-aed6-6c914b1b9abb" (UID: "5cc24a99-a137-4f53-aed6-6c914b1b9abb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:50:48.579633 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:48.579612 5222 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cc24a99-a137-4f53-aed6-6c914b1b9abb-plugins-config-volume\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:48.579633 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:48.579636 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjc4j\" (UniqueName: \"kubernetes.io/projected/b3b6cede-ec74-40cf-998e-5ff4238cd411-kube-api-access-cjc4j\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:48.581650 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:48.581622 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cc24a99-a137-4f53-aed6-6c914b1b9abb-kube-api-access-nmzls" pod "5cc24a99-a137-4f53-aed6-6c914b1b9abb" (UID: "5cc24a99-a137-4f53-aed6-6c914b1b9abb"). InnerVolumeSpecName "kube-api-access-nmzls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:50:48.681580 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:48.681487 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmzls\" (UniqueName: \"kubernetes.io/projected/5cc24a99-a137-4f53-aed6-6c914b1b9abb-kube-api-access-nmzls\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:49.305526 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:49.305493 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-5b4db8-78dcbd7945-mhhfh" Oct 31 00:50:49.306033 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:49.305526 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-5b4db8-78dcbd7945-mhhfh" event={"ID":"b3b6cede-ec74-40cf-998e-5ff4238cd411","Type":"ContainerDied","Data":"5ddad855d11fdc896d3f147aa9fcada7c44c0770c7cf8fa113a3d5fd05ebc6ef"} Oct 31 00:50:49.306033 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:49.305604 5222 scope.go:122] "RemoveContainer" containerID="0f5fefa9728e0e47b5db0c1f4a2f96e2488d3e6885a10521b85e6b3c76df33ee" Oct 31 00:50:49.307119 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:49.307093 5222 generic.go:356] "Generic (PLEG): container finished" podID="5cc24a99-a137-4f53-aed6-6c914b1b9abb" containerID="02d3057ec60584680437ecc2d5bfa6c2ad3aad953fe048dccc8cfef4394f9f3e" exitCode=0 Oct 31 00:50:49.307225 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:49.307183 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5b4db8-5c87d84775-wgg4h" Oct 31 00:50:49.307225 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:49.307200 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5b4db8-5c87d84775-wgg4h" event={"ID":"5cc24a99-a137-4f53-aed6-6c914b1b9abb","Type":"ContainerDied","Data":"02d3057ec60584680437ecc2d5bfa6c2ad3aad953fe048dccc8cfef4394f9f3e"} Oct 31 00:50:49.307344 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:49.307259 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5b4db8-5c87d84775-wgg4h" event={"ID":"5cc24a99-a137-4f53-aed6-6c914b1b9abb","Type":"ContainerDied","Data":"088e4e21f0a93408d3df33863dd0023c9bcad66e2723cecf41fc04a2aeab4929"} Oct 31 00:50:49.318306 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:49.318288 5222 scope.go:122] "RemoveContainer" containerID="02d3057ec60584680437ecc2d5bfa6c2ad3aad953fe048dccc8cfef4394f9f3e" Oct 31 00:50:49.328547 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:49.328524 5222 scope.go:122] "RemoveContainer" containerID="02d3057ec60584680437ecc2d5bfa6c2ad3aad953fe048dccc8cfef4394f9f3e" Oct 31 00:50:49.328861 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:50:49.328834 5222 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02d3057ec60584680437ecc2d5bfa6c2ad3aad953fe048dccc8cfef4394f9f3e\": container with ID starting with 02d3057ec60584680437ecc2d5bfa6c2ad3aad953fe048dccc8cfef4394f9f3e not found: ID does not exist" containerID="02d3057ec60584680437ecc2d5bfa6c2ad3aad953fe048dccc8cfef4394f9f3e" Oct 31 00:50:49.328934 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:49.328875 5222 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02d3057ec60584680437ecc2d5bfa6c2ad3aad953fe048dccc8cfef4394f9f3e"} err="failed to get container status \"02d3057ec60584680437ecc2d5bfa6c2ad3aad953fe048dccc8cfef4394f9f3e\": rpc error: code = NotFound desc = could not find container \"02d3057ec60584680437ecc2d5bfa6c2ad3aad953fe048dccc8cfef4394f9f3e\": container with ID starting with 02d3057ec60584680437ecc2d5bfa6c2ad3aad953fe048dccc8cfef4394f9f3e not found: ID does not exist" Oct 31 00:50:49.475502 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:49.475414 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5b4db8-78dcbd7945-mhhfh"] Oct 31 00:50:49.526444 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:49.526404 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5b4db8-78dcbd7945-mhhfh"] Oct 31 00:50:49.591675 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:49.591639 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5b4db8-5c87d84775-wgg4h"] Oct 31 00:50:49.595603 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:49.595574 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5b4db8-5c87d84775-wgg4h"] Oct 31 00:50:49.657926 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:49.657887 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5cc24a99-a137-4f53-aed6-6c914b1b9abb" path="/var/lib/kubelet/pods/5cc24a99-a137-4f53-aed6-6c914b1b9abb/volumes" Oct 31 00:50:49.658348 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:49.658334 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3b6cede-ec74-40cf-998e-5ff4238cd411" path="/var/lib/kubelet/pods/b3b6cede-ec74-40cf-998e-5ff4238cd411/volumes" Oct 31 00:50:50.262719 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:50.261978 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e9c3c1-57685bbfff-ddgqw"] Oct 31 00:50:53.863018 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:53.862979 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-e030e0-j5qhc"] Oct 31 00:50:53.913864 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:53.913834 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-e030e0-j5qhc"] Oct 31 00:50:55.262697 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:55.262642 5222 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e9c3c1-57685bbfff-ddgqw" podUID="fadfc07d-4c35-4936-a0ae-aded9677b62b" containerName="payload-pre-processing" containerID="cri-o://e30312bfbd59764574bf4f8c5c225bc97370fbc41af9f6e73209c8cb8919dd40" gracePeriod=25 Oct 31 00:50:55.474341 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:55.474291 5222 generic.go:356] "Generic (PLEG): container finished" podID="fadfc07d-4c35-4936-a0ae-aded9677b62b" containerID="e30312bfbd59764574bf4f8c5c225bc97370fbc41af9f6e73209c8cb8919dd40" exitCode=0 Oct 31 00:50:55.474792 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:55.474403 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e9c3c1-57685bbfff-ddgqw" event={"ID":"fadfc07d-4c35-4936-a0ae-aded9677b62b","Type":"ContainerDied","Data":"e30312bfbd59764574bf4f8c5c225bc97370fbc41af9f6e73209c8cb8919dd40"} Oct 31 00:50:55.547215 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:55.547181 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e9c3c1-57685bbfff-ddgqw" Oct 31 00:50:55.655672 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:55.655642 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fadfc07d-4c35-4936-a0ae-aded9677b62b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fadfc07d-4c35-4936-a0ae-aded9677b62b-plugins-config-volume\") pod \"fadfc07d-4c35-4936-a0ae-aded9677b62b\" (UID: \"fadfc07d-4c35-4936-a0ae-aded9677b62b\") " Oct 31 00:50:55.655934 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:55.655920 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fadfc07d-4c35-4936-a0ae-aded9677b62b-kube-api-access-5ltvw\" (UniqueName: \"kubernetes.io/projected/fadfc07d-4c35-4936-a0ae-aded9677b62b-kube-api-access-5ltvw\") pod \"fadfc07d-4c35-4936-a0ae-aded9677b62b\" (UID: \"fadfc07d-4c35-4936-a0ae-aded9677b62b\") " Oct 31 00:50:55.656031 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:55.656005 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fadfc07d-4c35-4936-a0ae-aded9677b62b-plugins-config-volume" pod "fadfc07d-4c35-4936-a0ae-aded9677b62b" (UID: "fadfc07d-4c35-4936-a0ae-aded9677b62b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:50:55.656465 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:55.656445 5222 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fadfc07d-4c35-4936-a0ae-aded9677b62b-plugins-config-volume\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:55.657331 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:55.657305 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f0919d7-e8d7-4f73-88ac-278d2203d1d4" path="/var/lib/kubelet/pods/9f0919d7-e8d7-4f73-88ac-278d2203d1d4/volumes" Oct 31 00:50:55.658201 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:55.658180 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fadfc07d-4c35-4936-a0ae-aded9677b62b-kube-api-access-5ltvw" pod "fadfc07d-4c35-4936-a0ae-aded9677b62b" (UID: "fadfc07d-4c35-4936-a0ae-aded9677b62b"). InnerVolumeSpecName "kube-api-access-5ltvw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:50:55.757869 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:55.757826 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ltvw\" (UniqueName: \"kubernetes.io/projected/fadfc07d-4c35-4936-a0ae-aded9677b62b-kube-api-access-5ltvw\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:56.483473 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:56.483442 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e9c3c1-57685bbfff-ddgqw" Oct 31 00:50:56.484102 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:56.483435 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e9c3c1-57685bbfff-ddgqw" event={"ID":"fadfc07d-4c35-4936-a0ae-aded9677b62b","Type":"ContainerDied","Data":"8fe13fc858628313fd91e823b19cbf0226334ff8be69524aaf74b648446d396d"} Oct 31 00:50:56.484102 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:56.483601 5222 scope.go:122] "RemoveContainer" containerID="e30312bfbd59764574bf4f8c5c225bc97370fbc41af9f6e73209c8cb8919dd40" Oct 31 00:50:56.652543 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:56.652510 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e9c3c1-57685bbfff-ddgqw"] Oct 31 00:50:56.703839 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:56.703810 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e9c3c1-57685bbfff-ddgqw"] Oct 31 00:50:57.657569 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:50:57.657532 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fadfc07d-4c35-4936-a0ae-aded9677b62b" path="/var/lib/kubelet/pods/fadfc07d-4c35-4936-a0ae-aded9677b62b/volumes" Oct 31 00:51:00.864059 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:00.864021 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-e030e0-openshift-default-5c976d9f4f-lt8jf"] Oct 31 00:51:00.864628 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:00.864227 5222 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-e030e0-openshift-default-5c976d9f4f-lt8jf" podUID="fcada5bd-0764-4b0c-883a-d5d1867ca38e" containerName="istio-proxy" containerID="cri-o://aafe3fa517c262a8e1d2a3e86409496f65430f74f47ff02f5a54ff397117fe20" gracePeriod=30 Oct 31 00:51:01.741785 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:01.741750 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-2594a9-s2225"] Oct 31 00:51:01.742532 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:01.742509 5222 cpu_manager.go:414] "containerMap: removing container" podUID="b3b6cede-ec74-40cf-998e-5ff4238cd411" containerName="payload-processing" Oct 31 00:51:01.742532 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:01.742531 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3b6cede-ec74-40cf-998e-5ff4238cd411" containerName="payload-processing" Oct 31 00:51:01.742737 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:01.742559 5222 cpu_manager.go:414] "containerMap: removing container" podUID="6a5bb0fe-0d5e-48da-bbee-839acf6a4b64" containerName="revoke-keys" Oct 31 00:51:01.742737 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:01.742566 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a5bb0fe-0d5e-48da-bbee-839acf6a4b64" containerName="revoke-keys" Oct 31 00:51:01.742737 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:01.742594 5222 cpu_manager.go:414] "containerMap: removing container" podUID="5cc24a99-a137-4f53-aed6-6c914b1b9abb" containerName="payload-pre-processing" Oct 31 00:51:01.742737 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:01.742603 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cc24a99-a137-4f53-aed6-6c914b1b9abb" containerName="payload-pre-processing" Oct 31 00:51:01.742737 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:01.742644 5222 cpu_manager.go:414] "containerMap: removing container" podUID="fadfc07d-4c35-4936-a0ae-aded9677b62b" containerName="payload-pre-processing" Oct 31 00:51:01.742737 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:01.742653 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fadfc07d-4c35-4936-a0ae-aded9677b62b" containerName="payload-pre-processing" Oct 31 00:51:01.743036 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:01.742830 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="fadfc07d-4c35-4936-a0ae-aded9677b62b" containerName="payload-pre-processing" Oct 31 00:51:01.743036 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:01.742851 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3b6cede-ec74-40cf-998e-5ff4238cd411" containerName="payload-processing" Oct 31 00:51:01.743036 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:01.742864 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a5bb0fe-0d5e-48da-bbee-839acf6a4b64" containerName="revoke-keys" Oct 31 00:51:01.743036 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:01.742877 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cc24a99-a137-4f53-aed6-6c914b1b9abb" containerName="payload-pre-processing" Oct 31 00:51:01.746306 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:01.746274 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-2594a9-s2225" Oct 31 00:51:01.799624 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:01.799576 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-2594a9-s2225"] Oct 31 00:51:01.799797 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:01.799674 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:51:01.846960 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:01.846924 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfz9z\"" Oct 31 00:51:01.846960 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:01.846963 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:51:01.847198 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:01.846999 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:51:01.853689 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:01.853663 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f25858c-4c7a-41dc-915b-72accf53c267-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-2594a9-s2225\" (UID: \"6f25858c-4c7a-41dc-915b-72accf53c267\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-2594a9-s2225" Oct 31 00:51:01.853804 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:01.853731 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f25858c-4c7a-41dc-915b-72accf53c267-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-2594a9-s2225\" (UID: \"6f25858c-4c7a-41dc-915b-72accf53c267\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-2594a9-s2225" Oct 31 00:51:01.853926 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:01.853904 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnbxm\" (UniqueName: \"kubernetes.io/projected/6f25858c-4c7a-41dc-915b-72accf53c267-kube-api-access-lnbxm\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-2594a9-s2225\" (UID: \"6f25858c-4c7a-41dc-915b-72accf53c267\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-2594a9-s2225" Oct 31 00:51:01.955561 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:01.955522 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnbxm\" (UniqueName: \"kubernetes.io/projected/6f25858c-4c7a-41dc-915b-72accf53c267-kube-api-access-lnbxm\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-2594a9-s2225\" (UID: \"6f25858c-4c7a-41dc-915b-72accf53c267\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-2594a9-s2225" Oct 31 00:51:01.956011 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:01.955653 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f25858c-4c7a-41dc-915b-72accf53c267-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-2594a9-s2225\" (UID: \"6f25858c-4c7a-41dc-915b-72accf53c267\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-2594a9-s2225" Oct 31 00:51:01.956011 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:01.955704 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f25858c-4c7a-41dc-915b-72accf53c267-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-2594a9-s2225\" (UID: \"6f25858c-4c7a-41dc-915b-72accf53c267\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-2594a9-s2225" Oct 31 00:51:01.956487 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:01.956463 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f25858c-4c7a-41dc-915b-72accf53c267-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-2594a9-s2225\" (UID: \"6f25858c-4c7a-41dc-915b-72accf53c267\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-2594a9-s2225" Oct 31 00:51:01.958181 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:01.958161 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f25858c-4c7a-41dc-915b-72accf53c267-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-2594a9-s2225\" (UID: \"6f25858c-4c7a-41dc-915b-72accf53c267\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-2594a9-s2225" Oct 31 00:51:02.018121 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:02.018089 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnbxm\" (UniqueName: \"kubernetes.io/projected/6f25858c-4c7a-41dc-915b-72accf53c267-kube-api-access-lnbxm\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-2594a9-s2225\" (UID: \"6f25858c-4c7a-41dc-915b-72accf53c267\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-2594a9-s2225" Oct 31 00:51:02.063169 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:02.063127 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-2594a9-s2225" Oct 31 00:51:02.441688 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:51:02.441655 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f25858c_4c7a_41dc_915b_72accf53c267.slice/crio-234dbaf32d8707003217f76b5e40a09cd3523c2e6da29760308800dc15aebb0d WatchSource:0}: Error finding container 234dbaf32d8707003217f76b5e40a09cd3523c2e6da29760308800dc15aebb0d: Status 404 returned error can't find the container with id 234dbaf32d8707003217f76b5e40a09cd3523c2e6da29760308800dc15aebb0d Oct 31 00:51:02.445851 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:02.445825 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-2594a9-s2225"] Oct 31 00:51:02.530863 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:02.530829 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-2594a9-s2225" event={"ID":"6f25858c-4c7a-41dc-915b-72accf53c267","Type":"ContainerStarted","Data":"234dbaf32d8707003217f76b5e40a09cd3523c2e6da29760308800dc15aebb0d"} Oct 31 00:51:03.540400 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:03.540324 5222 generic.go:356] "Generic (PLEG): container finished" podID="6f25858c-4c7a-41dc-915b-72accf53c267" containerID="0fe56ba038af95dfab8d305b69a236e62d86a479a246be2bcbd21818a410f28a" exitCode=0 Oct 31 00:51:03.540826 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:03.540429 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-2594a9-s2225" event={"ID":"6f25858c-4c7a-41dc-915b-72accf53c267","Type":"ContainerDied","Data":"0fe56ba038af95dfab8d305b69a236e62d86a479a246be2bcbd21818a410f28a"} Oct 31 00:51:04.461362 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:04.461327 5222 patch_prober.go:29] interesting pod/e2e-shared-a-w6-e030e0-openshift-default-5c976d9f4f-lt8jf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:51:04.461362 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:04.461400 5222 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w6-e030e0-openshift-default-5c976d9f4f-lt8jf" podUID="fcada5bd-0764-4b0c-883a-d5d1867ca38e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:51:04.721836 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:04.721772 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-2594a9-s2225" Oct 31 00:51:04.796297 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:04.796257 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f25858c-4c7a-41dc-915b-72accf53c267-kube-api-access-lnbxm\" (UniqueName: \"kubernetes.io/projected/6f25858c-4c7a-41dc-915b-72accf53c267-kube-api-access-lnbxm\") pod \"6f25858c-4c7a-41dc-915b-72accf53c267\" (UID: \"6f25858c-4c7a-41dc-915b-72accf53c267\") " Oct 31 00:51:04.796565 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:04.796355 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f25858c-4c7a-41dc-915b-72accf53c267-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f25858c-4c7a-41dc-915b-72accf53c267-maas-api-openshift-ca\") pod \"6f25858c-4c7a-41dc-915b-72accf53c267\" (UID: \"6f25858c-4c7a-41dc-915b-72accf53c267\") " Oct 31 00:51:04.796565 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:04.796499 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f25858c-4c7a-41dc-915b-72accf53c267-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f25858c-4c7a-41dc-915b-72accf53c267-maas-api-xks-ca\") pod \"6f25858c-4c7a-41dc-915b-72accf53c267\" (UID: \"6f25858c-4c7a-41dc-915b-72accf53c267\") " Oct 31 00:51:04.796789 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:04.796758 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f25858c-4c7a-41dc-915b-72accf53c267-maas-api-openshift-ca" pod "6f25858c-4c7a-41dc-915b-72accf53c267" (UID: "6f25858c-4c7a-41dc-915b-72accf53c267"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:51:04.797242 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:04.797214 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f25858c-4c7a-41dc-915b-72accf53c267-maas-api-openshift-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:51:04.798702 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:04.798678 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f25858c-4c7a-41dc-915b-72accf53c267-kube-api-access-lnbxm" pod "6f25858c-4c7a-41dc-915b-72accf53c267" (UID: "6f25858c-4c7a-41dc-915b-72accf53c267"). InnerVolumeSpecName "kube-api-access-lnbxm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:51:04.798812 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:04.798736 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f25858c-4c7a-41dc-915b-72accf53c267-maas-api-xks-ca" pod "6f25858c-4c7a-41dc-915b-72accf53c267" (UID: "6f25858c-4c7a-41dc-915b-72accf53c267"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:51:04.898238 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:04.898201 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnbxm\" (UniqueName: \"kubernetes.io/projected/6f25858c-4c7a-41dc-915b-72accf53c267-kube-api-access-lnbxm\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:51:04.898238 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:04.898234 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f25858c-4c7a-41dc-915b-72accf53c267-maas-api-xks-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:51:05.556797 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:05.556766 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-2594a9-s2225" Oct 31 00:51:05.556991 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:05.556762 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-2594a9-s2225" event={"ID":"6f25858c-4c7a-41dc-915b-72accf53c267","Type":"ContainerDied","Data":"234dbaf32d8707003217f76b5e40a09cd3523c2e6da29760308800dc15aebb0d"} Oct 31 00:51:05.556991 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:05.556907 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="234dbaf32d8707003217f76b5e40a09cd3523c2e6da29760308800dc15aebb0d" Oct 31 00:51:05.894034 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:51:05.894001 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 00:51:06.172744 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:06.172717 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-e030e0-openshift-default-5c976d9f4f-lt8jf" Oct 31 00:51:06.318215 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:06.318179 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcada5bd-0764-4b0c-883a-d5d1867ca38e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcada5bd-0764-4b0c-883a-d5d1867ca38e-workload-certs\") pod \"fcada5bd-0764-4b0c-883a-d5d1867ca38e\" (UID: \"fcada5bd-0764-4b0c-883a-d5d1867ca38e\") " Oct 31 00:51:06.318431 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:06.318283 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcada5bd-0764-4b0c-883a-d5d1867ca38e-kube-api-access-hqfcg\" (UniqueName: \"kubernetes.io/projected/fcada5bd-0764-4b0c-883a-d5d1867ca38e-kube-api-access-hqfcg\") pod \"fcada5bd-0764-4b0c-883a-d5d1867ca38e\" (UID: \"fcada5bd-0764-4b0c-883a-d5d1867ca38e\") " Oct 31 00:51:06.318431 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:06.318312 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fcada5bd-0764-4b0c-883a-d5d1867ca38e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcada5bd-0764-4b0c-883a-d5d1867ca38e-istio-podinfo\") pod \"fcada5bd-0764-4b0c-883a-d5d1867ca38e\" (UID: \"fcada5bd-0764-4b0c-883a-d5d1867ca38e\") " Oct 31 00:51:06.318431 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:06.318419 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcada5bd-0764-4b0c-883a-d5d1867ca38e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcada5bd-0764-4b0c-883a-d5d1867ca38e-workload-socket\") pod \"fcada5bd-0764-4b0c-883a-d5d1867ca38e\" (UID: \"fcada5bd-0764-4b0c-883a-d5d1867ca38e\") " Oct 31 00:51:06.318581 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:06.318514 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcada5bd-0764-4b0c-883a-d5d1867ca38e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcada5bd-0764-4b0c-883a-d5d1867ca38e-istio-data\") pod \"fcada5bd-0764-4b0c-883a-d5d1867ca38e\" (UID: \"fcada5bd-0764-4b0c-883a-d5d1867ca38e\") " Oct 31 00:51:06.318581 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:06.318524 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcada5bd-0764-4b0c-883a-d5d1867ca38e-workload-certs" pod "fcada5bd-0764-4b0c-883a-d5d1867ca38e" (UID: "fcada5bd-0764-4b0c-883a-d5d1867ca38e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:51:06.318581 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:06.318546 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcada5bd-0764-4b0c-883a-d5d1867ca38e-istio-token\" (UniqueName: \"kubernetes.io/projected/fcada5bd-0764-4b0c-883a-d5d1867ca38e-istio-token\") pod \"fcada5bd-0764-4b0c-883a-d5d1867ca38e\" (UID: \"fcada5bd-0764-4b0c-883a-d5d1867ca38e\") " Oct 31 00:51:06.318713 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:06.318635 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcada5bd-0764-4b0c-883a-d5d1867ca38e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcada5bd-0764-4b0c-883a-d5d1867ca38e-istiod-ca-cert\") pod \"fcada5bd-0764-4b0c-883a-d5d1867ca38e\" (UID: \"fcada5bd-0764-4b0c-883a-d5d1867ca38e\") " Oct 31 00:51:06.318713 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:06.318684 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcada5bd-0764-4b0c-883a-d5d1867ca38e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcada5bd-0764-4b0c-883a-d5d1867ca38e-istio-envoy\") pod \"fcada5bd-0764-4b0c-883a-d5d1867ca38e\" (UID: \"fcada5bd-0764-4b0c-883a-d5d1867ca38e\") " Oct 31 00:51:06.318812 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:06.318742 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcada5bd-0764-4b0c-883a-d5d1867ca38e-workload-socket" pod "fcada5bd-0764-4b0c-883a-d5d1867ca38e" (UID: "fcada5bd-0764-4b0c-883a-d5d1867ca38e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:51:06.318812 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:06.318760 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcada5bd-0764-4b0c-883a-d5d1867ca38e-istio-data" pod "fcada5bd-0764-4b0c-883a-d5d1867ca38e" (UID: "fcada5bd-0764-4b0c-883a-d5d1867ca38e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:51:06.318812 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:06.318790 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcada5bd-0764-4b0c-883a-d5d1867ca38e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcada5bd-0764-4b0c-883a-d5d1867ca38e-credential-socket\") pod \"fcada5bd-0764-4b0c-883a-d5d1867ca38e\" (UID: \"fcada5bd-0764-4b0c-883a-d5d1867ca38e\") " Oct 31 00:51:06.319132 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:06.319105 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcada5bd-0764-4b0c-883a-d5d1867ca38e-credential-socket" pod "fcada5bd-0764-4b0c-883a-d5d1867ca38e" (UID: "fcada5bd-0764-4b0c-883a-d5d1867ca38e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:51:06.319246 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:06.319133 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcada5bd-0764-4b0c-883a-d5d1867ca38e-istiod-ca-cert" pod "fcada5bd-0764-4b0c-883a-d5d1867ca38e" (UID: "fcada5bd-0764-4b0c-883a-d5d1867ca38e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:51:06.319654 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:06.319629 5222 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcada5bd-0764-4b0c-883a-d5d1867ca38e-credential-socket\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:51:06.319739 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:06.319660 5222 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcada5bd-0764-4b0c-883a-d5d1867ca38e-workload-certs\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:51:06.319739 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:06.319675 5222 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcada5bd-0764-4b0c-883a-d5d1867ca38e-workload-socket\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:51:06.319739 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:06.319690 5222 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcada5bd-0764-4b0c-883a-d5d1867ca38e-istio-data\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:51:06.319739 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:06.319705 5222 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcada5bd-0764-4b0c-883a-d5d1867ca38e-istiod-ca-cert\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:51:06.320868 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:06.320840 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fcada5bd-0764-4b0c-883a-d5d1867ca38e-istio-podinfo" pod "fcada5bd-0764-4b0c-883a-d5d1867ca38e" (UID: "fcada5bd-0764-4b0c-883a-d5d1867ca38e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:51:06.320977 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:06.320914 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcada5bd-0764-4b0c-883a-d5d1867ca38e-kube-api-access-hqfcg" pod "fcada5bd-0764-4b0c-883a-d5d1867ca38e" (UID: "fcada5bd-0764-4b0c-883a-d5d1867ca38e"). InnerVolumeSpecName "kube-api-access-hqfcg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:51:06.321061 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:06.321045 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcada5bd-0764-4b0c-883a-d5d1867ca38e-istio-token" pod "fcada5bd-0764-4b0c-883a-d5d1867ca38e" (UID: "fcada5bd-0764-4b0c-883a-d5d1867ca38e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:51:06.321195 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:06.321180 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcada5bd-0764-4b0c-883a-d5d1867ca38e-istio-envoy" pod "fcada5bd-0764-4b0c-883a-d5d1867ca38e" (UID: "fcada5bd-0764-4b0c-883a-d5d1867ca38e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:51:06.421800 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:06.421706 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqfcg\" (UniqueName: \"kubernetes.io/projected/fcada5bd-0764-4b0c-883a-d5d1867ca38e-kube-api-access-hqfcg\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:51:06.421800 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:06.421737 5222 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcada5bd-0764-4b0c-883a-d5d1867ca38e-istio-podinfo\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:51:06.421800 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:06.421748 5222 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcada5bd-0764-4b0c-883a-d5d1867ca38e-istio-token\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:51:06.421800 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:06.421758 5222 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcada5bd-0764-4b0c-883a-d5d1867ca38e-istio-envoy\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:51:06.583860 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:06.583828 5222 generic.go:356] "Generic (PLEG): container finished" podID="fcada5bd-0764-4b0c-883a-d5d1867ca38e" containerID="aafe3fa517c262a8e1d2a3e86409496f65430f74f47ff02f5a54ff397117fe20" exitCode=0 Oct 31 00:51:06.584075 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:06.583889 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-e030e0-openshift-default-5c976d9f4f-lt8jf" Oct 31 00:51:06.584075 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:06.583908 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-e030e0-openshift-default-5c976d9f4f-lt8jf" event={"ID":"fcada5bd-0764-4b0c-883a-d5d1867ca38e","Type":"ContainerDied","Data":"aafe3fa517c262a8e1d2a3e86409496f65430f74f47ff02f5a54ff397117fe20"} Oct 31 00:51:06.584075 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:06.583974 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-e030e0-openshift-default-5c976d9f4f-lt8jf" event={"ID":"fcada5bd-0764-4b0c-883a-d5d1867ca38e","Type":"ContainerDied","Data":"b1cedde882e131ac0930f2539ebf3d8236a7632ed11d5dbe3f4748fea2760274"} Oct 31 00:51:06.584075 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:06.583999 5222 scope.go:122] "RemoveContainer" containerID="aafe3fa517c262a8e1d2a3e86409496f65430f74f47ff02f5a54ff397117fe20" Oct 31 00:51:06.597214 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:06.597190 5222 scope.go:122] "RemoveContainer" containerID="aba32c5fa7816f3b70cd111f5250a6773ea14a12a382519cf4288f57f4ca8115" Oct 31 00:51:06.607668 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:06.607644 5222 scope.go:122] "RemoveContainer" containerID="aafe3fa517c262a8e1d2a3e86409496f65430f74f47ff02f5a54ff397117fe20" Oct 31 00:51:06.607985 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:51:06.607966 5222 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aafe3fa517c262a8e1d2a3e86409496f65430f74f47ff02f5a54ff397117fe20\": container with ID starting with aafe3fa517c262a8e1d2a3e86409496f65430f74f47ff02f5a54ff397117fe20 not found: ID does not exist" containerID="aafe3fa517c262a8e1d2a3e86409496f65430f74f47ff02f5a54ff397117fe20" Oct 31 00:51:06.608030 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:06.607995 5222 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aafe3fa517c262a8e1d2a3e86409496f65430f74f47ff02f5a54ff397117fe20"} err="failed to get container status \"aafe3fa517c262a8e1d2a3e86409496f65430f74f47ff02f5a54ff397117fe20\": rpc error: code = NotFound desc = could not find container \"aafe3fa517c262a8e1d2a3e86409496f65430f74f47ff02f5a54ff397117fe20\": container with ID starting with aafe3fa517c262a8e1d2a3e86409496f65430f74f47ff02f5a54ff397117fe20 not found: ID does not exist" Oct 31 00:51:06.608030 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:06.608015 5222 scope.go:122] "RemoveContainer" containerID="aba32c5fa7816f3b70cd111f5250a6773ea14a12a382519cf4288f57f4ca8115" Oct 31 00:51:06.608276 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:51:06.608258 5222 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aba32c5fa7816f3b70cd111f5250a6773ea14a12a382519cf4288f57f4ca8115\": container with ID starting with aba32c5fa7816f3b70cd111f5250a6773ea14a12a382519cf4288f57f4ca8115 not found: ID does not exist" containerID="aba32c5fa7816f3b70cd111f5250a6773ea14a12a382519cf4288f57f4ca8115" Oct 31 00:51:06.608311 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:06.608285 5222 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aba32c5fa7816f3b70cd111f5250a6773ea14a12a382519cf4288f57f4ca8115"} err="failed to get container status \"aba32c5fa7816f3b70cd111f5250a6773ea14a12a382519cf4288f57f4ca8115\": rpc error: code = NotFound desc = could not find container \"aba32c5fa7816f3b70cd111f5250a6773ea14a12a382519cf4288f57f4ca8115\": container with ID starting with aba32c5fa7816f3b70cd111f5250a6773ea14a12a382519cf4288f57f4ca8115 not found: ID does not exist" Oct 31 00:51:06.755019 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:06.754939 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-e030e0-openshift-default-5c976d9f4f-lt8jf"] Oct 31 00:51:06.805775 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:06.805738 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-e030e0-openshift-default-5c976d9f4f-lt8jf"] Oct 31 00:51:07.657986 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:07.657953 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fcada5bd-0764-4b0c-883a-d5d1867ca38e" path="/var/lib/kubelet/pods/fcada5bd-0764-4b0c-883a-d5d1867ca38e/volumes" Oct 31 00:51:11.101212 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:11.101172 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-dcdbd0-4bv7s"] Oct 31 00:51:11.152441 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:11.152350 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-dcdbd0-4bv7s"] Oct 31 00:51:11.657758 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:11.657726 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c1b1ee2-7349-41c4-9fde-3bbf2aa4a8d8" path="/var/lib/kubelet/pods/5c1b1ee2-7349-41c4-9fde-3bbf2aa4a8d8/volumes" Oct 31 00:51:18.609833 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:18.609753 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-dcdbd0-openshift-default-86cd956cc9-p8h4z"] Oct 31 00:51:18.610232 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:18.609985 5222 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-dcdbd0-openshift-default-86cd956cc9-p8h4z" podUID="d0694975-c551-4a8a-b296-eaee47077b0c" containerName="istio-proxy" containerID="cri-o://e5dd597aea1c6554fa4de02548aad9b94a18448800059c5d26af7c4a84b57352" gracePeriod=30 Oct 31 00:51:20.009287 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:20.009242 5222 patch_prober.go:29] interesting pod/e2e-shared-b-w2-dcdbd0-openshift-default-86cd956cc9-p8h4z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:51:20.009720 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:20.009320 5222 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-dcdbd0-openshift-default-86cd956cc9-p8h4z" podUID="d0694975-c551-4a8a-b296-eaee47077b0c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:51:20.156204 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:20.156173 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e8ba06-7z5fl"] Oct 31 00:51:20.156873 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:20.156854 5222 cpu_manager.go:414] "containerMap: removing container" podUID="fcada5bd-0764-4b0c-883a-d5d1867ca38e" containerName="istio-proxy" Oct 31 00:51:20.156873 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:20.156870 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcada5bd-0764-4b0c-883a-d5d1867ca38e" containerName="istio-proxy" Oct 31 00:51:20.157070 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:20.156902 5222 cpu_manager.go:414] "containerMap: removing container" podUID="6f25858c-4c7a-41dc-915b-72accf53c267" containerName="revoke-keys" Oct 31 00:51:20.157070 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:20.156908 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f25858c-4c7a-41dc-915b-72accf53c267" containerName="revoke-keys" Oct 31 00:51:20.157070 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:20.157013 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcada5bd-0764-4b0c-883a-d5d1867ca38e" containerName="istio-proxy" Oct 31 00:51:20.157070 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:20.157029 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f25858c-4c7a-41dc-915b-72accf53c267" containerName="revoke-keys" Oct 31 00:51:20.162306 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:20.162281 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e8ba06-7z5fl" Oct 31 00:51:20.214542 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:20.214509 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e8ba06-7z5fl"] Oct 31 00:51:20.216180 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:20.216148 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:51:20.216392 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:20.216317 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfz9z\"" Oct 31 00:51:20.242880 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:20.242843 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0eb779be-9c69-429a-87b5-2707673ef946-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-e8ba06-7z5fl\" (UID: \"0eb779be-9c69-429a-87b5-2707673ef946\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e8ba06-7z5fl" Oct 31 00:51:20.243096 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:20.243073 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckmjb\" (UniqueName: \"kubernetes.io/projected/0eb779be-9c69-429a-87b5-2707673ef946-kube-api-access-ckmjb\") pod \"maas-api-revoke-keys-e2e-worker-w2-e8ba06-7z5fl\" (UID: \"0eb779be-9c69-429a-87b5-2707673ef946\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e8ba06-7z5fl" Oct 31 00:51:20.243341 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:20.243317 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0eb779be-9c69-429a-87b5-2707673ef946-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-e8ba06-7z5fl\" (UID: \"0eb779be-9c69-429a-87b5-2707673ef946\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e8ba06-7z5fl" Oct 31 00:51:20.261624 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:20.261532 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:51:20.261624 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:20.261616 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:51:20.344257 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:20.344228 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0eb779be-9c69-429a-87b5-2707673ef946-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-e8ba06-7z5fl\" (UID: \"0eb779be-9c69-429a-87b5-2707673ef946\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e8ba06-7z5fl" Oct 31 00:51:20.344461 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:20.344278 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0eb779be-9c69-429a-87b5-2707673ef946-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-e8ba06-7z5fl\" (UID: \"0eb779be-9c69-429a-87b5-2707673ef946\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e8ba06-7z5fl" Oct 31 00:51:20.344461 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:20.344439 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckmjb\" (UniqueName: \"kubernetes.io/projected/0eb779be-9c69-429a-87b5-2707673ef946-kube-api-access-ckmjb\") pod \"maas-api-revoke-keys-e2e-worker-w2-e8ba06-7z5fl\" (UID: \"0eb779be-9c69-429a-87b5-2707673ef946\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e8ba06-7z5fl" Oct 31 00:51:20.344926 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:20.344906 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0eb779be-9c69-429a-87b5-2707673ef946-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-e8ba06-7z5fl\" (UID: \"0eb779be-9c69-429a-87b5-2707673ef946\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e8ba06-7z5fl" Oct 31 00:51:20.346908 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:20.346889 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0eb779be-9c69-429a-87b5-2707673ef946-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-e8ba06-7z5fl\" (UID: \"0eb779be-9c69-429a-87b5-2707673ef946\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e8ba06-7z5fl" Oct 31 00:51:20.407559 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:20.407530 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckmjb\" (UniqueName: \"kubernetes.io/projected/0eb779be-9c69-429a-87b5-2707673ef946-kube-api-access-ckmjb\") pod \"maas-api-revoke-keys-e2e-worker-w2-e8ba06-7z5fl\" (UID: \"0eb779be-9c69-429a-87b5-2707673ef946\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e8ba06-7z5fl" Oct 31 00:51:20.476057 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:20.476020 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e8ba06-7z5fl" Oct 31 00:51:20.858915 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:51:20.858880 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0eb779be_9c69_429a_87b5_2707673ef946.slice/crio-99c29fd347f197f236e332b232418b55fa0a7eb29dd28420bfb6def4183497f9 WatchSource:0}: Error finding container 99c29fd347f197f236e332b232418b55fa0a7eb29dd28420bfb6def4183497f9: Status 404 returned error can't find the container with id 99c29fd347f197f236e332b232418b55fa0a7eb29dd28420bfb6def4183497f9 Oct 31 00:51:20.861446 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:20.861420 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e8ba06-7z5fl"] Oct 31 00:51:20.867772 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:20.867748 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73e94f2f-vnx94"] Oct 31 00:51:20.868756 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:20.868736 5222 cpu_manager.go:414] "containerMap: removing container" podUID="fcada5bd-0764-4b0c-883a-d5d1867ca38e" containerName="istio-proxy" Oct 31 00:51:20.868808 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:20.868761 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcada5bd-0764-4b0c-883a-d5d1867ca38e" containerName="istio-proxy" Oct 31 00:51:20.869023 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:20.868994 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcada5bd-0764-4b0c-883a-d5d1867ca38e" containerName="istio-proxy" Oct 31 00:51:20.874601 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:20.874577 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73e94f2f-vnx94" Oct 31 00:51:20.954702 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:20.954658 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ddbb670-bb0c-45b6-ac6f-be091e71e08f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-73e94f2f-vnx94\" (UID: \"8ddbb670-bb0c-45b6-ac6f-be091e71e08f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73e94f2f-vnx94" Oct 31 00:51:20.954864 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:20.954714 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5z5xs\" (UniqueName: \"kubernetes.io/projected/8ddbb670-bb0c-45b6-ac6f-be091e71e08f-kube-api-access-5z5xs\") pod \"maas-api-revoke-sub-73e94f2f-vnx94\" (UID: \"8ddbb670-bb0c-45b6-ac6f-be091e71e08f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73e94f2f-vnx94" Oct 31 00:51:20.954926 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:20.954906 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ddbb670-bb0c-45b6-ac6f-be091e71e08f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-73e94f2f-vnx94\" (UID: \"8ddbb670-bb0c-45b6-ac6f-be091e71e08f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73e94f2f-vnx94" Oct 31 00:51:20.959897 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:20.959869 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73e94f2f-vnx94"] Oct 31 00:51:20.976880 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:20.976844 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72cb1008-qp222"] Oct 31 00:51:20.981234 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:20.981210 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72cb1008-qp222" Oct 31 00:51:21.036855 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:21.036812 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72cb1008-qp222"] Oct 31 00:51:21.057035 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:21.056987 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d113faa4-aa17-4a24-bf7c-bfe3e01d9ee0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-72cb1008-qp222\" (UID: \"d113faa4-aa17-4a24-bf7c-bfe3e01d9ee0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72cb1008-qp222" Oct 31 00:51:21.057254 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:21.057169 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ddbb670-bb0c-45b6-ac6f-be091e71e08f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-73e94f2f-vnx94\" (UID: \"8ddbb670-bb0c-45b6-ac6f-be091e71e08f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73e94f2f-vnx94" Oct 31 00:51:21.057254 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:21.057247 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5z5xs\" (UniqueName: \"kubernetes.io/projected/8ddbb670-bb0c-45b6-ac6f-be091e71e08f-kube-api-access-5z5xs\") pod \"maas-api-revoke-sub-73e94f2f-vnx94\" (UID: \"8ddbb670-bb0c-45b6-ac6f-be091e71e08f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73e94f2f-vnx94" Oct 31 00:51:21.057441 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:21.057324 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ddbb670-bb0c-45b6-ac6f-be091e71e08f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-73e94f2f-vnx94\" (UID: \"8ddbb670-bb0c-45b6-ac6f-be091e71e08f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73e94f2f-vnx94" Oct 31 00:51:21.057441 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:21.057392 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2l5q\" (UniqueName: \"kubernetes.io/projected/d113faa4-aa17-4a24-bf7c-bfe3e01d9ee0-kube-api-access-s2l5q\") pod \"maas-api-revoke-sub-72cb1008-qp222\" (UID: \"d113faa4-aa17-4a24-bf7c-bfe3e01d9ee0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72cb1008-qp222" Oct 31 00:51:21.057572 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:21.057472 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d113faa4-aa17-4a24-bf7c-bfe3e01d9ee0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-72cb1008-qp222\" (UID: \"d113faa4-aa17-4a24-bf7c-bfe3e01d9ee0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72cb1008-qp222" Oct 31 00:51:21.058345 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:21.058319 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ddbb670-bb0c-45b6-ac6f-be091e71e08f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-73e94f2f-vnx94\" (UID: \"8ddbb670-bb0c-45b6-ac6f-be091e71e08f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73e94f2f-vnx94" Oct 31 00:51:21.060083 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:21.060055 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ddbb670-bb0c-45b6-ac6f-be091e71e08f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-73e94f2f-vnx94\" (UID: \"8ddbb670-bb0c-45b6-ac6f-be091e71e08f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73e94f2f-vnx94" Oct 31 00:51:21.120195 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:21.120114 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5z5xs\" (UniqueName: \"kubernetes.io/projected/8ddbb670-bb0c-45b6-ac6f-be091e71e08f-kube-api-access-5z5xs\") pod \"maas-api-revoke-sub-73e94f2f-vnx94\" (UID: \"8ddbb670-bb0c-45b6-ac6f-be091e71e08f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73e94f2f-vnx94" Oct 31 00:51:21.158429 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:21.158397 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d113faa4-aa17-4a24-bf7c-bfe3e01d9ee0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-72cb1008-qp222\" (UID: \"d113faa4-aa17-4a24-bf7c-bfe3e01d9ee0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72cb1008-qp222" Oct 31 00:51:21.158594 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:21.158533 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d113faa4-aa17-4a24-bf7c-bfe3e01d9ee0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-72cb1008-qp222\" (UID: \"d113faa4-aa17-4a24-bf7c-bfe3e01d9ee0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72cb1008-qp222" Oct 31 00:51:21.158673 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:21.158662 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2l5q\" (UniqueName: \"kubernetes.io/projected/d113faa4-aa17-4a24-bf7c-bfe3e01d9ee0-kube-api-access-s2l5q\") pod \"maas-api-revoke-sub-72cb1008-qp222\" (UID: \"d113faa4-aa17-4a24-bf7c-bfe3e01d9ee0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72cb1008-qp222" Oct 31 00:51:21.159143 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:21.159112 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d113faa4-aa17-4a24-bf7c-bfe3e01d9ee0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-72cb1008-qp222\" (UID: \"d113faa4-aa17-4a24-bf7c-bfe3e01d9ee0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72cb1008-qp222" Oct 31 00:51:21.161046 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:21.161027 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d113faa4-aa17-4a24-bf7c-bfe3e01d9ee0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-72cb1008-qp222\" (UID: \"d113faa4-aa17-4a24-bf7c-bfe3e01d9ee0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72cb1008-qp222" Oct 31 00:51:21.200698 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:21.200664 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73e94f2f-vnx94" Oct 31 00:51:21.221261 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:21.221227 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2l5q\" (UniqueName: \"kubernetes.io/projected/d113faa4-aa17-4a24-bf7c-bfe3e01d9ee0-kube-api-access-s2l5q\") pod \"maas-api-revoke-sub-72cb1008-qp222\" (UID: \"d113faa4-aa17-4a24-bf7c-bfe3e01d9ee0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72cb1008-qp222" Oct 31 00:51:21.296989 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:21.296719 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72cb1008-qp222" Oct 31 00:51:21.628545 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:51:21.628501 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ddbb670_bb0c_45b6_ac6f_be091e71e08f.slice/crio-d6e9f539ea0d52207e815e2906edaf3b70838984c8d63a61db708dea286eceb4 WatchSource:0}: Error finding container d6e9f539ea0d52207e815e2906edaf3b70838984c8d63a61db708dea286eceb4: Status 404 returned error can't find the container with id d6e9f539ea0d52207e815e2906edaf3b70838984c8d63a61db708dea286eceb4 Oct 31 00:51:21.632781 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:21.632754 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73e94f2f-vnx94"] Oct 31 00:51:21.713032 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:51:21.712995 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd113faa4_aa17_4a24_bf7c_bfe3e01d9ee0.slice/crio-aa70875bdd83a1bea4c3275b79fe1d35099ec8ea0f74d88fc611ed71cc2ce788 WatchSource:0}: Error finding container aa70875bdd83a1bea4c3275b79fe1d35099ec8ea0f74d88fc611ed71cc2ce788: Status 404 returned error can't find the container with id aa70875bdd83a1bea4c3275b79fe1d35099ec8ea0f74d88fc611ed71cc2ce788 Oct 31 00:51:21.717279 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:21.717249 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72cb1008-qp222"] Oct 31 00:51:21.794797 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:21.794758 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72cb1008-qp222" event={"ID":"d113faa4-aa17-4a24-bf7c-bfe3e01d9ee0","Type":"ContainerStarted","Data":"aa70875bdd83a1bea4c3275b79fe1d35099ec8ea0f74d88fc611ed71cc2ce788"} Oct 31 00:51:21.796247 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:21.796213 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73e94f2f-vnx94" event={"ID":"8ddbb670-bb0c-45b6-ac6f-be091e71e08f","Type":"ContainerStarted","Data":"d6e9f539ea0d52207e815e2906edaf3b70838984c8d63a61db708dea286eceb4"} Oct 31 00:51:21.797887 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:21.797860 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e8ba06-7z5fl" event={"ID":"0eb779be-9c69-429a-87b5-2707673ef946","Type":"ContainerStarted","Data":"7485db9836ea874ebdab72d00635eb84682a4926b031c8e572201471d1ee085a"} Oct 31 00:51:21.798000 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:21.797904 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e8ba06-7z5fl" event={"ID":"0eb779be-9c69-429a-87b5-2707673ef946","Type":"ContainerStarted","Data":"99c29fd347f197f236e332b232418b55fa0a7eb29dd28420bfb6def4183497f9"} Oct 31 00:51:21.917004 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:21.916904 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e8ba06-7z5fl" podStartSLOduration=1.916888303 podStartE2EDuration="1.916888303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:51:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:51:21.915430268 +0000 UTC m=+2174.386600018" watchObservedRunningTime="2026-10-31 00:51:21.916888303 +0000 UTC m=+2174.388058082" Oct 31 00:51:22.808659 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:22.808617 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72cb1008-qp222" event={"ID":"d113faa4-aa17-4a24-bf7c-bfe3e01d9ee0","Type":"ContainerStarted","Data":"778f9e0a89a273cdbacdd720189692f13d142a178a8874b6fd6e6fc2106bc63e"} Oct 31 00:51:22.810649 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:22.810604 5222 generic.go:356] "Generic (PLEG): container finished" podID="8ddbb670-bb0c-45b6-ac6f-be091e71e08f" containerID="884319de0e2dfa6a978a236b9ce62d0a0b0cf94856a63f687ab766a5563fc1d6" exitCode=0 Oct 31 00:51:22.811196 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:22.810690 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73e94f2f-vnx94" event={"ID":"8ddbb670-bb0c-45b6-ac6f-be091e71e08f","Type":"ContainerDied","Data":"884319de0e2dfa6a978a236b9ce62d0a0b0cf94856a63f687ab766a5563fc1d6"} Oct 31 00:51:22.812470 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:22.812443 5222 generic.go:356] "Generic (PLEG): container finished" podID="0eb779be-9c69-429a-87b5-2707673ef946" containerID="7485db9836ea874ebdab72d00635eb84682a4926b031c8e572201471d1ee085a" exitCode=0 Oct 31 00:51:22.812594 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:22.812521 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e8ba06-7z5fl" event={"ID":"0eb779be-9c69-429a-87b5-2707673ef946","Type":"ContainerDied","Data":"7485db9836ea874ebdab72d00635eb84682a4926b031c8e572201471d1ee085a"} Oct 31 00:51:22.929967 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:22.929858 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72cb1008-qp222" podStartSLOduration=2.9298395839999998 podStartE2EDuration="2.929839584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:51:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:51:22.92790734 +0000 UTC m=+2175.399077182" watchObservedRunningTime="2026-10-31 00:51:22.929839584 +0000 UTC m=+2175.401009336" Oct 31 00:51:23.824324 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:23.824292 5222 generic.go:356] "Generic (PLEG): container finished" podID="d0694975-c551-4a8a-b296-eaee47077b0c" containerID="e5dd597aea1c6554fa4de02548aad9b94a18448800059c5d26af7c4a84b57352" exitCode=0 Oct 31 00:51:23.824783 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:23.824362 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-dcdbd0-openshift-default-86cd956cc9-p8h4z" event={"ID":"d0694975-c551-4a8a-b296-eaee47077b0c","Type":"ContainerDied","Data":"e5dd597aea1c6554fa4de02548aad9b94a18448800059c5d26af7c4a84b57352"} Oct 31 00:51:23.824783 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:23.824465 5222 scope.go:122] "RemoveContainer" containerID="ce3d7e380246894abac01542c0fdcd9b01ae85322ca675c9d5d7ac62e7e8f8ce" Oct 31 00:51:23.826714 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:23.826501 5222 generic.go:356] "Generic (PLEG): container finished" podID="d113faa4-aa17-4a24-bf7c-bfe3e01d9ee0" containerID="778f9e0a89a273cdbacdd720189692f13d142a178a8874b6fd6e6fc2106bc63e" exitCode=0 Oct 31 00:51:23.826714 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:23.826536 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72cb1008-qp222" event={"ID":"d113faa4-aa17-4a24-bf7c-bfe3e01d9ee0","Type":"ContainerDied","Data":"778f9e0a89a273cdbacdd720189692f13d142a178a8874b6fd6e6fc2106bc63e"} Oct 31 00:51:23.913796 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:23.913466 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-dcdbd0-openshift-default-86cd956cc9-p8h4z" Oct 31 00:51:24.010647 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:24.010614 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0694975-c551-4a8a-b296-eaee47077b0c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0694975-c551-4a8a-b296-eaee47077b0c-workload-socket\") pod \"d0694975-c551-4a8a-b296-eaee47077b0c\" (UID: \"d0694975-c551-4a8a-b296-eaee47077b0c\") " Oct 31 00:51:24.010792 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:24.010705 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0694975-c551-4a8a-b296-eaee47077b0c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0694975-c551-4a8a-b296-eaee47077b0c-istio-data\") pod \"d0694975-c551-4a8a-b296-eaee47077b0c\" (UID: \"d0694975-c551-4a8a-b296-eaee47077b0c\") " Oct 31 00:51:24.010792 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:24.010748 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0694975-c551-4a8a-b296-eaee47077b0c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0694975-c551-4a8a-b296-eaee47077b0c-istiod-ca-cert\") pod \"d0694975-c551-4a8a-b296-eaee47077b0c\" (UID: \"d0694975-c551-4a8a-b296-eaee47077b0c\") " Oct 31 00:51:24.010792 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:24.010782 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d0694975-c551-4a8a-b296-eaee47077b0c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0694975-c551-4a8a-b296-eaee47077b0c-istio-podinfo\") pod \"d0694975-c551-4a8a-b296-eaee47077b0c\" (UID: \"d0694975-c551-4a8a-b296-eaee47077b0c\") " Oct 31 00:51:24.010996 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:24.010884 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0694975-c551-4a8a-b296-eaee47077b0c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0694975-c551-4a8a-b296-eaee47077b0c-istio-envoy\") pod \"d0694975-c551-4a8a-b296-eaee47077b0c\" (UID: \"d0694975-c551-4a8a-b296-eaee47077b0c\") " Oct 31 00:51:24.010996 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:24.010914 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0694975-c551-4a8a-b296-eaee47077b0c-istio-token\" (UniqueName: \"kubernetes.io/projected/d0694975-c551-4a8a-b296-eaee47077b0c-istio-token\") pod \"d0694975-c551-4a8a-b296-eaee47077b0c\" (UID: \"d0694975-c551-4a8a-b296-eaee47077b0c\") " Oct 31 00:51:24.010996 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:24.010908 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0694975-c551-4a8a-b296-eaee47077b0c-workload-socket" pod "d0694975-c551-4a8a-b296-eaee47077b0c" (UID: "d0694975-c551-4a8a-b296-eaee47077b0c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:51:24.010996 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:24.010977 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0694975-c551-4a8a-b296-eaee47077b0c-istio-data" pod "d0694975-c551-4a8a-b296-eaee47077b0c" (UID: "d0694975-c551-4a8a-b296-eaee47077b0c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:51:24.010996 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:24.010988 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0694975-c551-4a8a-b296-eaee47077b0c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0694975-c551-4a8a-b296-eaee47077b0c-credential-socket\") pod \"d0694975-c551-4a8a-b296-eaee47077b0c\" (UID: \"d0694975-c551-4a8a-b296-eaee47077b0c\") " Oct 31 00:51:24.011499 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:24.011093 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0694975-c551-4a8a-b296-eaee47077b0c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0694975-c551-4a8a-b296-eaee47077b0c-workload-certs\") pod \"d0694975-c551-4a8a-b296-eaee47077b0c\" (UID: \"d0694975-c551-4a8a-b296-eaee47077b0c\") " Oct 31 00:51:24.011499 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:24.011145 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0694975-c551-4a8a-b296-eaee47077b0c-kube-api-access-d66gr\" (UniqueName: \"kubernetes.io/projected/d0694975-c551-4a8a-b296-eaee47077b0c-kube-api-access-d66gr\") pod \"d0694975-c551-4a8a-b296-eaee47077b0c\" (UID: \"d0694975-c551-4a8a-b296-eaee47077b0c\") " Oct 31 00:51:24.011499 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:24.011301 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0694975-c551-4a8a-b296-eaee47077b0c-istiod-ca-cert" pod "d0694975-c551-4a8a-b296-eaee47077b0c" (UID: "d0694975-c551-4a8a-b296-eaee47077b0c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:51:24.011875 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:24.011814 5222 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0694975-c551-4a8a-b296-eaee47077b0c-istio-data\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:51:24.011875 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:24.011843 5222 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0694975-c551-4a8a-b296-eaee47077b0c-istiod-ca-cert\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:51:24.011875 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:24.011858 5222 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0694975-c551-4a8a-b296-eaee47077b0c-workload-socket\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:51:24.012239 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:24.012203 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0694975-c551-4a8a-b296-eaee47077b0c-credential-socket" pod "d0694975-c551-4a8a-b296-eaee47077b0c" (UID: "d0694975-c551-4a8a-b296-eaee47077b0c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:51:24.012328 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:24.012306 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0694975-c551-4a8a-b296-eaee47077b0c-workload-certs" pod "d0694975-c551-4a8a-b296-eaee47077b0c" (UID: "d0694975-c551-4a8a-b296-eaee47077b0c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:51:24.014199 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:24.014165 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0694975-c551-4a8a-b296-eaee47077b0c-istio-token" pod "d0694975-c551-4a8a-b296-eaee47077b0c" (UID: "d0694975-c551-4a8a-b296-eaee47077b0c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:51:24.014298 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:24.014236 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d0694975-c551-4a8a-b296-eaee47077b0c-istio-podinfo" pod "d0694975-c551-4a8a-b296-eaee47077b0c" (UID: "d0694975-c551-4a8a-b296-eaee47077b0c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:51:24.015173 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:24.015152 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0694975-c551-4a8a-b296-eaee47077b0c-istio-envoy" pod "d0694975-c551-4a8a-b296-eaee47077b0c" (UID: "d0694975-c551-4a8a-b296-eaee47077b0c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:51:24.015302 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:24.015286 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0694975-c551-4a8a-b296-eaee47077b0c-kube-api-access-d66gr" pod "d0694975-c551-4a8a-b296-eaee47077b0c" (UID: "d0694975-c551-4a8a-b296-eaee47077b0c"). InnerVolumeSpecName "kube-api-access-d66gr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:51:24.042282 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:24.042254 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73e94f2f-vnx94" Oct 31 00:51:24.045129 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:24.045108 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e8ba06-7z5fl" Oct 31 00:51:24.113114 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:24.113069 5222 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0694975-c551-4a8a-b296-eaee47077b0c-istio-podinfo\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:51:24.113114 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:24.113107 5222 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0694975-c551-4a8a-b296-eaee47077b0c-istio-envoy\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:51:24.113342 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:24.113123 5222 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0694975-c551-4a8a-b296-eaee47077b0c-istio-token\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:51:24.113342 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:24.113136 5222 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0694975-c551-4a8a-b296-eaee47077b0c-credential-socket\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:51:24.113342 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:24.113150 5222 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0694975-c551-4a8a-b296-eaee47077b0c-workload-certs\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:51:24.113342 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:24.113162 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d66gr\" (UniqueName: \"kubernetes.io/projected/d0694975-c551-4a8a-b296-eaee47077b0c-kube-api-access-d66gr\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:51:24.213955 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:24.213850 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0eb779be-9c69-429a-87b5-2707673ef946-kube-api-access-ckmjb\" (UniqueName: \"kubernetes.io/projected/0eb779be-9c69-429a-87b5-2707673ef946-kube-api-access-ckmjb\") pod \"0eb779be-9c69-429a-87b5-2707673ef946\" (UID: \"0eb779be-9c69-429a-87b5-2707673ef946\") " Oct 31 00:51:24.214136 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:24.214032 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0eb779be-9c69-429a-87b5-2707673ef946-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0eb779be-9c69-429a-87b5-2707673ef946-maas-api-xks-ca\") pod \"0eb779be-9c69-429a-87b5-2707673ef946\" (UID: \"0eb779be-9c69-429a-87b5-2707673ef946\") " Oct 31 00:51:24.214136 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:24.214067 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ddbb670-bb0c-45b6-ac6f-be091e71e08f-kube-api-access-5z5xs\" (UniqueName: \"kubernetes.io/projected/8ddbb670-bb0c-45b6-ac6f-be091e71e08f-kube-api-access-5z5xs\") pod \"8ddbb670-bb0c-45b6-ac6f-be091e71e08f\" (UID: \"8ddbb670-bb0c-45b6-ac6f-be091e71e08f\") " Oct 31 00:51:24.214136 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:24.214110 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ddbb670-bb0c-45b6-ac6f-be091e71e08f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ddbb670-bb0c-45b6-ac6f-be091e71e08f-maas-api-openshift-ca\") pod \"8ddbb670-bb0c-45b6-ac6f-be091e71e08f\" (UID: \"8ddbb670-bb0c-45b6-ac6f-be091e71e08f\") " Oct 31 00:51:24.214322 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:24.214177 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ddbb670-bb0c-45b6-ac6f-be091e71e08f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ddbb670-bb0c-45b6-ac6f-be091e71e08f-maas-api-xks-ca\") pod \"8ddbb670-bb0c-45b6-ac6f-be091e71e08f\" (UID: \"8ddbb670-bb0c-45b6-ac6f-be091e71e08f\") " Oct 31 00:51:24.214619 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:24.214588 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ddbb670-bb0c-45b6-ac6f-be091e71e08f-maas-api-openshift-ca" pod "8ddbb670-bb0c-45b6-ac6f-be091e71e08f" (UID: "8ddbb670-bb0c-45b6-ac6f-be091e71e08f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:51:24.214780 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:24.214762 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0eb779be-9c69-429a-87b5-2707673ef946-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0eb779be-9c69-429a-87b5-2707673ef946-maas-api-openshift-ca\") pod \"0eb779be-9c69-429a-87b5-2707673ef946\" (UID: \"0eb779be-9c69-429a-87b5-2707673ef946\") " Oct 31 00:51:24.217098 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:24.215502 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0eb779be-9c69-429a-87b5-2707673ef946-maas-api-openshift-ca" pod "0eb779be-9c69-429a-87b5-2707673ef946" (UID: "0eb779be-9c69-429a-87b5-2707673ef946"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:51:24.217098 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:24.215757 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ddbb670-bb0c-45b6-ac6f-be091e71e08f-maas-api-openshift-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:51:24.217279 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:24.217094 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0eb779be-9c69-429a-87b5-2707673ef946-kube-api-access-ckmjb" pod "0eb779be-9c69-429a-87b5-2707673ef946" (UID: "0eb779be-9c69-429a-87b5-2707673ef946"). InnerVolumeSpecName "kube-api-access-ckmjb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:51:24.217279 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:24.217176 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0eb779be-9c69-429a-87b5-2707673ef946-maas-api-xks-ca" pod "0eb779be-9c69-429a-87b5-2707673ef946" (UID: "0eb779be-9c69-429a-87b5-2707673ef946"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:51:24.217524 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:24.215780 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0eb779be-9c69-429a-87b5-2707673ef946-maas-api-openshift-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:51:24.217524 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:24.217468 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ddbb670-bb0c-45b6-ac6f-be091e71e08f-maas-api-xks-ca" pod "8ddbb670-bb0c-45b6-ac6f-be091e71e08f" (UID: "8ddbb670-bb0c-45b6-ac6f-be091e71e08f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:51:24.217618 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:24.217526 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ddbb670-bb0c-45b6-ac6f-be091e71e08f-kube-api-access-5z5xs" pod "8ddbb670-bb0c-45b6-ac6f-be091e71e08f" (UID: "8ddbb670-bb0c-45b6-ac6f-be091e71e08f"). InnerVolumeSpecName "kube-api-access-5z5xs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:51:24.318551 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:24.318518 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckmjb\" (UniqueName: \"kubernetes.io/projected/0eb779be-9c69-429a-87b5-2707673ef946-kube-api-access-ckmjb\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:51:24.318551 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:24.318550 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0eb779be-9c69-429a-87b5-2707673ef946-maas-api-xks-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:51:24.318551 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:24.318559 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5z5xs\" (UniqueName: \"kubernetes.io/projected/8ddbb670-bb0c-45b6-ac6f-be091e71e08f-kube-api-access-5z5xs\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:51:24.318800 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:24.318569 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ddbb670-bb0c-45b6-ac6f-be091e71e08f-maas-api-xks-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:51:24.835642 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:24.835609 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-dcdbd0-openshift-default-86cd956cc9-p8h4z" Oct 31 00:51:24.836156 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:24.835641 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-dcdbd0-openshift-default-86cd956cc9-p8h4z" event={"ID":"d0694975-c551-4a8a-b296-eaee47077b0c","Type":"ContainerDied","Data":"a53d563fe9f049b4392e680e5c3f8bd711e499b10fa2fa0742df1776658c6497"} Oct 31 00:51:24.836156 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:24.835711 5222 scope.go:122] "RemoveContainer" containerID="e5dd597aea1c6554fa4de02548aad9b94a18448800059c5d26af7c4a84b57352" Oct 31 00:51:24.837279 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:24.837251 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73e94f2f-vnx94" Oct 31 00:51:24.837422 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:24.837249 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73e94f2f-vnx94" event={"ID":"8ddbb670-bb0c-45b6-ac6f-be091e71e08f","Type":"ContainerDied","Data":"d6e9f539ea0d52207e815e2906edaf3b70838984c8d63a61db708dea286eceb4"} Oct 31 00:51:24.837422 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:24.837418 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d6e9f539ea0d52207e815e2906edaf3b70838984c8d63a61db708dea286eceb4" Oct 31 00:51:24.839135 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:24.839107 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e8ba06-7z5fl" Oct 31 00:51:24.839263 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:24.839240 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e8ba06-7z5fl" event={"ID":"0eb779be-9c69-429a-87b5-2707673ef946","Type":"ContainerDied","Data":"99c29fd347f197f236e332b232418b55fa0a7eb29dd28420bfb6def4183497f9"} Oct 31 00:51:24.839333 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:24.839288 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="99c29fd347f197f236e332b232418b55fa0a7eb29dd28420bfb6def4183497f9" Oct 31 00:51:25.007742 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:25.007705 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72cb1008-qp222" Oct 31 00:51:25.028304 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:25.028271 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d113faa4-aa17-4a24-bf7c-bfe3e01d9ee0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d113faa4-aa17-4a24-bf7c-bfe3e01d9ee0-maas-api-xks-ca\") pod \"d113faa4-aa17-4a24-bf7c-bfe3e01d9ee0\" (UID: \"d113faa4-aa17-4a24-bf7c-bfe3e01d9ee0\") " Oct 31 00:51:25.028554 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:25.028429 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d113faa4-aa17-4a24-bf7c-bfe3e01d9ee0-kube-api-access-s2l5q\" (UniqueName: \"kubernetes.io/projected/d113faa4-aa17-4a24-bf7c-bfe3e01d9ee0-kube-api-access-s2l5q\") pod \"d113faa4-aa17-4a24-bf7c-bfe3e01d9ee0\" (UID: \"d113faa4-aa17-4a24-bf7c-bfe3e01d9ee0\") " Oct 31 00:51:25.028554 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:25.028466 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d113faa4-aa17-4a24-bf7c-bfe3e01d9ee0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d113faa4-aa17-4a24-bf7c-bfe3e01d9ee0-maas-api-openshift-ca\") pod \"d113faa4-aa17-4a24-bf7c-bfe3e01d9ee0\" (UID: \"d113faa4-aa17-4a24-bf7c-bfe3e01d9ee0\") " Oct 31 00:51:25.028879 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:25.028847 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d113faa4-aa17-4a24-bf7c-bfe3e01d9ee0-maas-api-openshift-ca" pod "d113faa4-aa17-4a24-bf7c-bfe3e01d9ee0" (UID: "d113faa4-aa17-4a24-bf7c-bfe3e01d9ee0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:51:25.030758 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:25.030725 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d113faa4-aa17-4a24-bf7c-bfe3e01d9ee0-maas-api-xks-ca" pod "d113faa4-aa17-4a24-bf7c-bfe3e01d9ee0" (UID: "d113faa4-aa17-4a24-bf7c-bfe3e01d9ee0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:51:25.030879 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:25.030826 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d113faa4-aa17-4a24-bf7c-bfe3e01d9ee0-kube-api-access-s2l5q" pod "d113faa4-aa17-4a24-bf7c-bfe3e01d9ee0" (UID: "d113faa4-aa17-4a24-bf7c-bfe3e01d9ee0"). InnerVolumeSpecName "kube-api-access-s2l5q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:51:25.065296 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:25.065262 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-dcdbd0-openshift-default-86cd956cc9-p8h4z"] Oct 31 00:51:25.070981 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:25.070946 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-dcdbd0-openshift-default-86cd956cc9-p8h4z"] Oct 31 00:51:25.130356 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:25.130324 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d113faa4-aa17-4a24-bf7c-bfe3e01d9ee0-maas-api-xks-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:51:25.130356 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:25.130352 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2l5q\" (UniqueName: \"kubernetes.io/projected/d113faa4-aa17-4a24-bf7c-bfe3e01d9ee0-kube-api-access-s2l5q\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:51:25.130356 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:25.130365 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d113faa4-aa17-4a24-bf7c-bfe3e01d9ee0-maas-api-openshift-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:51:25.658415 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:25.658358 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0694975-c551-4a8a-b296-eaee47077b0c" path="/var/lib/kubelet/pods/d0694975-c551-4a8a-b296-eaee47077b0c/volumes" Oct 31 00:51:25.848942 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:25.848911 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72cb1008-qp222" Oct 31 00:51:25.848942 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:25.848929 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72cb1008-qp222" event={"ID":"d113faa4-aa17-4a24-bf7c-bfe3e01d9ee0","Type":"ContainerDied","Data":"aa70875bdd83a1bea4c3275b79fe1d35099ec8ea0f74d88fc611ed71cc2ce788"} Oct 31 00:51:25.849448 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:25.849003 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aa70875bdd83a1bea4c3275b79fe1d35099ec8ea0f74d88fc611ed71cc2ce788" Oct 31 00:51:30.167466 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:30.167431 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-79tk8"] Oct 31 00:51:31.794503 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:31.794463 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-e8ba06-bb9fb667f-x6fzk"] Oct 31 00:51:31.864312 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:31.864269 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-e8ba06-7b548bc5f-m9r6m"] Oct 31 00:51:35.891248 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:51:35.891210 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 00:51:36.795121 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:36.795035 5222 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-e8ba06-bb9fb667f-x6fzk" podUID="207c55f2-0d5c-497c-b021-c6ba4939370b" containerName="payload-processing" containerID="cri-o://924123f35d24276665ed606867ca065a9f35f091fa65355a50a823c42cb1d316" gracePeriod=25 Oct 31 00:51:36.864841 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:36.864801 5222 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e8ba06-7b548bc5f-m9r6m" podUID="06704c9d-24a7-4e63-bdd3-b7ee85866fbf" containerName="payload-pre-processing" containerID="cri-o://c7ad93d2942a5d855c70743f3b243a5ccbfe1d8a5b8e0972bed3dee955d231f6" gracePeriod=25 Oct 31 00:51:37.859057 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:37.859009 5222 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e8ba06-7b548bc5f-m9r6m" podUID="06704c9d-24a7-4e63-bdd3-b7ee85866fbf" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:51:37.859508 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:37.859078 5222 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-e8ba06-bb9fb667f-x6fzk" podUID="207c55f2-0d5c-497c-b021-c6ba4939370b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:51:41.022138 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:41.022109 5222 generic.go:356] "Generic (PLEG): container finished" podID="c46a2765-b445-4cd6-9b3a-1aef0a68b895" containerID="ffd419175db9ab219d8aa6bb9ebd361568b3e73b48ccb04f0f9d7ff01734c8bd" exitCode=0 Oct 31 00:51:41.022608 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:41.022182 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4ab8e5-openshift-default-6c89ffbf45-mtckf" event={"ID":"c46a2765-b445-4cd6-9b3a-1aef0a68b895","Type":"ContainerDied","Data":"ffd419175db9ab219d8aa6bb9ebd361568b3e73b48ccb04f0f9d7ff01734c8bd"} Oct 31 00:51:41.022701 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:41.022673 5222 scope.go:122] "RemoveContainer" containerID="ffd419175db9ab219d8aa6bb9ebd361568b3e73b48ccb04f0f9d7ff01734c8bd" Oct 31 00:51:41.024314 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:41.024286 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:51:41.024388 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:41.024358 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:51:41.024438 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:41.024406 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:51:42.031707 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:42.031674 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4ab8e5-openshift-default-6c89ffbf45-mtckf" event={"ID":"c46a2765-b445-4cd6-9b3a-1aef0a68b895","Type":"ContainerStarted","Data":"b6ff4242e6b0992cd4acc1bd25bb6984ff8beed4ae1f8953d333d3f0d7f1f5ac"} Oct 31 00:51:42.513612 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:42.513511 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-4ab8e5-openshift-default-6c89ffbf45-mtckf" Oct 31 00:51:42.579733 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:42.579701 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-4ab8e5-openshift-default-6c89ffbf45-mtckf" Oct 31 00:51:42.580746 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:42.580718 5222 patch_prober.go:29] interesting pod/e2e-inf-a-4ab8e5-openshift-default-6c89ffbf45-mtckf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Oct 31 00:51:42.580873 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:42.580780 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4ab8e5-openshift-default-6c89ffbf45-mtckf" podUID="c46a2765-b445-4cd6-9b3a-1aef0a68b895" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Oct 31 00:51:43.579320 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:43.579281 5222 patch_prober.go:29] interesting pod/e2e-inf-a-4ab8e5-openshift-default-6c89ffbf45-mtckf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Oct 31 00:51:43.579766 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:43.579350 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4ab8e5-openshift-default-6c89ffbf45-mtckf" podUID="c46a2765-b445-4cd6-9b3a-1aef0a68b895" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Oct 31 00:51:44.579792 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:44.579752 5222 patch_prober.go:29] interesting pod/e2e-inf-a-4ab8e5-openshift-default-6c89ffbf45-mtckf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Oct 31 00:51:44.580207 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:44.579812 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4ab8e5-openshift-default-6c89ffbf45-mtckf" podUID="c46a2765-b445-4cd6-9b3a-1aef0a68b895" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Oct 31 00:51:45.579285 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:45.579248 5222 patch_prober.go:29] interesting pod/e2e-inf-a-4ab8e5-openshift-default-6c89ffbf45-mtckf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Oct 31 00:51:45.579517 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:45.579300 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4ab8e5-openshift-default-6c89ffbf45-mtckf" podUID="c46a2765-b445-4cd6-9b3a-1aef0a68b895" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Oct 31 00:51:46.580057 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:46.580020 5222 patch_prober.go:29] interesting pod/e2e-inf-a-4ab8e5-openshift-default-6c89ffbf45-mtckf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Oct 31 00:51:46.580519 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:46.580080 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4ab8e5-openshift-default-6c89ffbf45-mtckf" podUID="c46a2765-b445-4cd6-9b3a-1aef0a68b895" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Oct 31 00:51:47.644016 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:47.643982 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-4ab8e5-openshift-default-6c89ffbf45-mtckf" Oct 31 00:51:47.645300 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:47.645279 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-4ab8e5-openshift-default-6c89ffbf45-mtckf" Oct 31 00:51:47.859185 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:47.859148 5222 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-e8ba06-bb9fb667f-x6fzk" podUID="207c55f2-0d5c-497c-b021-c6ba4939370b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:51:47.859420 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:47.859156 5222 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e8ba06-7b548bc5f-m9r6m" podUID="06704c9d-24a7-4e63-bdd3-b7ee85866fbf" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:51:53.563329 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:53.563293 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-2rtnt"] Oct 31 00:51:53.563954 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:53.563939 5222 cpu_manager.go:414] "containerMap: removing container" podUID="d0694975-c551-4a8a-b296-eaee47077b0c" containerName="istio-proxy" Oct 31 00:51:53.564002 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:53.563956 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0694975-c551-4a8a-b296-eaee47077b0c" containerName="istio-proxy" Oct 31 00:51:53.564002 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:53.563969 5222 cpu_manager.go:414] "containerMap: removing container" podUID="d113faa4-aa17-4a24-bf7c-bfe3e01d9ee0" containerName="revoke-keys" Oct 31 00:51:53.564002 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:53.563974 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d113faa4-aa17-4a24-bf7c-bfe3e01d9ee0" containerName="revoke-keys" Oct 31 00:51:53.564002 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:53.563990 5222 cpu_manager.go:414] "containerMap: removing container" podUID="8ddbb670-bb0c-45b6-ac6f-be091e71e08f" containerName="revoke-keys" Oct 31 00:51:53.564002 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:53.563995 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ddbb670-bb0c-45b6-ac6f-be091e71e08f" containerName="revoke-keys" Oct 31 00:51:53.564163 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:53.564005 5222 cpu_manager.go:414] "containerMap: removing container" podUID="0eb779be-9c69-429a-87b5-2707673ef946" containerName="revoke-keys" Oct 31 00:51:53.564163 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:53.564010 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0eb779be-9c69-429a-87b5-2707673ef946" containerName="revoke-keys" Oct 31 00:51:53.564163 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:53.564115 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="0eb779be-9c69-429a-87b5-2707673ef946" containerName="revoke-keys" Oct 31 00:51:53.564163 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:53.564129 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0694975-c551-4a8a-b296-eaee47077b0c" containerName="istio-proxy" Oct 31 00:51:53.564163 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:53.564137 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0694975-c551-4a8a-b296-eaee47077b0c" containerName="istio-proxy" Oct 31 00:51:53.564163 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:53.564143 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="d113faa4-aa17-4a24-bf7c-bfe3e01d9ee0" containerName="revoke-keys" Oct 31 00:51:53.564163 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:53.564151 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ddbb670-bb0c-45b6-ac6f-be091e71e08f" containerName="revoke-keys" Oct 31 00:51:53.567026 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:53.567001 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-2rtnt" Oct 31 00:51:53.621104 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:53.621063 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-7d5f4de7-openshift-default-dockercfg-q5jrh\"" Oct 31 00:51:53.667046 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:53.666996 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69c94b24-49c5-49b4-825b-50778d04669a-workload-socket\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-2rtnt\" (UID: \"69c94b24-49c5-49b4-825b-50778d04669a\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-2rtnt" Oct 31 00:51:53.667246 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:53.667060 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69c94b24-49c5-49b4-825b-50778d04669a-credential-socket\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-2rtnt\" (UID: \"69c94b24-49c5-49b4-825b-50778d04669a\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-2rtnt" Oct 31 00:51:53.667246 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:53.667099 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69c94b24-49c5-49b4-825b-50778d04669a-workload-certs\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-2rtnt\" (UID: \"69c94b24-49c5-49b4-825b-50778d04669a\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-2rtnt" Oct 31 00:51:53.667414 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:53.667244 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pl4vw\" (UniqueName: \"kubernetes.io/projected/69c94b24-49c5-49b4-825b-50778d04669a-kube-api-access-pl4vw\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-2rtnt\" (UID: \"69c94b24-49c5-49b4-825b-50778d04669a\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-2rtnt" Oct 31 00:51:53.667414 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:53.667295 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69c94b24-49c5-49b4-825b-50778d04669a-istiod-ca-cert\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-2rtnt\" (UID: \"69c94b24-49c5-49b4-825b-50778d04669a\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-2rtnt" Oct 31 00:51:53.667590 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:53.667560 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69c94b24-49c5-49b4-825b-50778d04669a-istio-data\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-2rtnt\" (UID: \"69c94b24-49c5-49b4-825b-50778d04669a\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-2rtnt" Oct 31 00:51:53.667713 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:53.667635 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69c94b24-49c5-49b4-825b-50778d04669a-istio-podinfo\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-2rtnt\" (UID: \"69c94b24-49c5-49b4-825b-50778d04669a\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-2rtnt" Oct 31 00:51:53.667713 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:53.667709 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69c94b24-49c5-49b4-825b-50778d04669a-istio-token\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-2rtnt\" (UID: \"69c94b24-49c5-49b4-825b-50778d04669a\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-2rtnt" Oct 31 00:51:53.667854 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:53.667774 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69c94b24-49c5-49b4-825b-50778d04669a-istio-envoy\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-2rtnt\" (UID: \"69c94b24-49c5-49b4-825b-50778d04669a\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-2rtnt" Oct 31 00:51:53.678316 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:53.678278 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-2rtnt"] Oct 31 00:51:53.768780 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:53.768744 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69c94b24-49c5-49b4-825b-50778d04669a-workload-socket\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-2rtnt\" (UID: \"69c94b24-49c5-49b4-825b-50778d04669a\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-2rtnt" Oct 31 00:51:53.768974 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:53.768798 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69c94b24-49c5-49b4-825b-50778d04669a-credential-socket\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-2rtnt\" (UID: \"69c94b24-49c5-49b4-825b-50778d04669a\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-2rtnt" Oct 31 00:51:53.768974 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:53.768917 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69c94b24-49c5-49b4-825b-50778d04669a-workload-certs\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-2rtnt\" (UID: \"69c94b24-49c5-49b4-825b-50778d04669a\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-2rtnt" Oct 31 00:51:53.768974 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:53.768971 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pl4vw\" (UniqueName: \"kubernetes.io/projected/69c94b24-49c5-49b4-825b-50778d04669a-kube-api-access-pl4vw\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-2rtnt\" (UID: \"69c94b24-49c5-49b4-825b-50778d04669a\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-2rtnt" Oct 31 00:51:53.769172 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:53.769085 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69c94b24-49c5-49b4-825b-50778d04669a-istiod-ca-cert\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-2rtnt\" (UID: \"69c94b24-49c5-49b4-825b-50778d04669a\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-2rtnt" Oct 31 00:51:53.769172 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:53.769129 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69c94b24-49c5-49b4-825b-50778d04669a-credential-socket\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-2rtnt\" (UID: \"69c94b24-49c5-49b4-825b-50778d04669a\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-2rtnt" Oct 31 00:51:53.769291 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:53.769184 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69c94b24-49c5-49b4-825b-50778d04669a-workload-socket\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-2rtnt\" (UID: \"69c94b24-49c5-49b4-825b-50778d04669a\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-2rtnt" Oct 31 00:51:53.769291 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:53.769257 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69c94b24-49c5-49b4-825b-50778d04669a-istio-data\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-2rtnt\" (UID: \"69c94b24-49c5-49b4-825b-50778d04669a\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-2rtnt" Oct 31 00:51:53.769291 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:53.769274 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69c94b24-49c5-49b4-825b-50778d04669a-workload-certs\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-2rtnt\" (UID: \"69c94b24-49c5-49b4-825b-50778d04669a\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-2rtnt" Oct 31 00:51:53.769532 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:53.769304 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69c94b24-49c5-49b4-825b-50778d04669a-istio-podinfo\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-2rtnt\" (UID: \"69c94b24-49c5-49b4-825b-50778d04669a\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-2rtnt" Oct 31 00:51:53.769532 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:53.769421 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69c94b24-49c5-49b4-825b-50778d04669a-istio-token\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-2rtnt\" (UID: \"69c94b24-49c5-49b4-825b-50778d04669a\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-2rtnt" Oct 31 00:51:53.769532 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:53.769482 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69c94b24-49c5-49b4-825b-50778d04669a-istio-envoy\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-2rtnt\" (UID: \"69c94b24-49c5-49b4-825b-50778d04669a\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-2rtnt" Oct 31 00:51:53.769682 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:53.769618 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69c94b24-49c5-49b4-825b-50778d04669a-istio-data\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-2rtnt\" (UID: \"69c94b24-49c5-49b4-825b-50778d04669a\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-2rtnt" Oct 31 00:51:53.769800 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:53.769781 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69c94b24-49c5-49b4-825b-50778d04669a-istiod-ca-cert\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-2rtnt\" (UID: \"69c94b24-49c5-49b4-825b-50778d04669a\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-2rtnt" Oct 31 00:51:53.771873 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:53.771850 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69c94b24-49c5-49b4-825b-50778d04669a-istio-envoy\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-2rtnt\" (UID: \"69c94b24-49c5-49b4-825b-50778d04669a\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-2rtnt" Oct 31 00:51:53.772039 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:53.772020 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69c94b24-49c5-49b4-825b-50778d04669a-istio-podinfo\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-2rtnt\" (UID: \"69c94b24-49c5-49b4-825b-50778d04669a\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-2rtnt" Oct 31 00:51:53.831906 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:53.831827 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69c94b24-49c5-49b4-825b-50778d04669a-istio-token\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-2rtnt\" (UID: \"69c94b24-49c5-49b4-825b-50778d04669a\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-2rtnt" Oct 31 00:51:53.831906 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:53.831874 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pl4vw\" (UniqueName: \"kubernetes.io/projected/69c94b24-49c5-49b4-825b-50778d04669a-kube-api-access-pl4vw\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-2rtnt\" (UID: \"69c94b24-49c5-49b4-825b-50778d04669a\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-2rtnt" Oct 31 00:51:53.880755 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:53.880723 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-2rtnt" Oct 31 00:51:54.307389 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:51:54.307334 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69c94b24_49c5_49b4_825b_50778d04669a.slice/crio-5b8b1a33f821a2b3071308ea0879b7b2276c2fdfa073fea74fbfef6766a31885 WatchSource:0}: Error finding container 5b8b1a33f821a2b3071308ea0879b7b2276c2fdfa073fea74fbfef6766a31885: Status 404 returned error can't find the container with id 5b8b1a33f821a2b3071308ea0879b7b2276c2fdfa073fea74fbfef6766a31885 Oct 31 00:51:54.310593 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:54.310558 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:51:54.310686 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:54.310631 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:51:54.310686 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:54.310659 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:51:54.311466 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:54.311444 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-2rtnt"] Oct 31 00:51:55.193386 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:55.193325 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-2rtnt" event={"ID":"69c94b24-49c5-49b4-825b-50778d04669a","Type":"ContainerStarted","Data":"1759f6dde775c2a3e85842b471291eb25eeec97618130cd89261cc2a9abdf738"} Oct 31 00:51:55.193874 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:55.193411 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-2rtnt" event={"ID":"69c94b24-49c5-49b4-825b-50778d04669a","Type":"ContainerStarted","Data":"5b8b1a33f821a2b3071308ea0879b7b2276c2fdfa073fea74fbfef6766a31885"} Oct 31 00:51:55.380652 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:55.380584 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-2rtnt" podStartSLOduration=2.380570723 podStartE2EDuration="2.380570723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:51:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:51:55.379704378 +0000 UTC m=+2207.850874128" watchObservedRunningTime="2026-10-31 00:51:55.380570723 +0000 UTC m=+2207.851740471" Oct 31 00:51:55.881187 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:55.881154 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-2rtnt" Oct 31 00:51:55.883164 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:55.883142 5222 patch_prober.go:29] interesting pod/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-2rtnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" start-of-body= Oct 31 00:51:55.883245 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:55.883193 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-2rtnt" podUID="69c94b24-49c5-49b4-825b-50778d04669a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" Oct 31 00:51:56.881191 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:56.881151 5222 patch_prober.go:29] interesting pod/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-2rtnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" start-of-body= Oct 31 00:51:56.881629 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:56.881224 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-2rtnt" podUID="69c94b24-49c5-49b4-825b-50778d04669a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" Oct 31 00:51:57.219068 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:57.219036 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-e8ba06-bb9fb667f-x6fzk" Oct 31 00:51:57.225493 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:57.224223 5222 generic.go:356] "Generic (PLEG): container finished" podID="207c55f2-0d5c-497c-b021-c6ba4939370b" containerID="924123f35d24276665ed606867ca065a9f35f091fa65355a50a823c42cb1d316" exitCode=0 Oct 31 00:51:57.225867 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:57.225482 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-e8ba06-bb9fb667f-x6fzk" Oct 31 00:51:57.226203 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:57.225512 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-e8ba06-bb9fb667f-x6fzk" event={"ID":"207c55f2-0d5c-497c-b021-c6ba4939370b","Type":"ContainerDied","Data":"924123f35d24276665ed606867ca065a9f35f091fa65355a50a823c42cb1d316"} Oct 31 00:51:57.226305 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:57.226274 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-e8ba06-bb9fb667f-x6fzk" event={"ID":"207c55f2-0d5c-497c-b021-c6ba4939370b","Type":"ContainerDied","Data":"4c59f8ad7928bc750731457b3190269fb3bdfcb3c6e0e672bf1690a8c1cdc1e4"} Oct 31 00:51:57.226305 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:57.226302 5222 scope.go:122] "RemoveContainer" containerID="924123f35d24276665ed606867ca065a9f35f091fa65355a50a823c42cb1d316" Oct 31 00:51:57.229130 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:57.229106 5222 generic.go:356] "Generic (PLEG): container finished" podID="06704c9d-24a7-4e63-bdd3-b7ee85866fbf" containerID="c7ad93d2942a5d855c70743f3b243a5ccbfe1d8a5b8e0972bed3dee955d231f6" exitCode=0 Oct 31 00:51:57.229235 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:57.229210 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e8ba06-7b548bc5f-m9r6m" event={"ID":"06704c9d-24a7-4e63-bdd3-b7ee85866fbf","Type":"ContainerDied","Data":"c7ad93d2942a5d855c70743f3b243a5ccbfe1d8a5b8e0972bed3dee955d231f6"} Oct 31 00:51:57.265196 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:57.265175 5222 scope.go:122] "RemoveContainer" containerID="924123f35d24276665ed606867ca065a9f35f091fa65355a50a823c42cb1d316" Oct 31 00:51:57.265566 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:51:57.265535 5222 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"924123f35d24276665ed606867ca065a9f35f091fa65355a50a823c42cb1d316\": container with ID starting with 924123f35d24276665ed606867ca065a9f35f091fa65355a50a823c42cb1d316 not found: ID does not exist" containerID="924123f35d24276665ed606867ca065a9f35f091fa65355a50a823c42cb1d316" Oct 31 00:51:57.265863 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:57.265836 5222 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"924123f35d24276665ed606867ca065a9f35f091fa65355a50a823c42cb1d316"} err="failed to get container status \"924123f35d24276665ed606867ca065a9f35f091fa65355a50a823c42cb1d316\": rpc error: code = NotFound desc = could not find container \"924123f35d24276665ed606867ca065a9f35f091fa65355a50a823c42cb1d316\": container with ID starting with 924123f35d24276665ed606867ca065a9f35f091fa65355a50a823c42cb1d316 not found: ID does not exist" Oct 31 00:51:57.266112 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:57.266095 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e8ba06-7b548bc5f-m9r6m" Oct 31 00:51:57.320332 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:57.320294 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/207c55f2-0d5c-497c-b021-c6ba4939370b-kube-api-access-jx7sm\" (UniqueName: \"kubernetes.io/projected/207c55f2-0d5c-497c-b021-c6ba4939370b-kube-api-access-jx7sm\") pod \"207c55f2-0d5c-497c-b021-c6ba4939370b\" (UID: \"207c55f2-0d5c-497c-b021-c6ba4939370b\") " Oct 31 00:51:57.320566 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:57.320443 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06704c9d-24a7-4e63-bdd3-b7ee85866fbf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06704c9d-24a7-4e63-bdd3-b7ee85866fbf-plugins-config-volume\") pod \"06704c9d-24a7-4e63-bdd3-b7ee85866fbf\" (UID: \"06704c9d-24a7-4e63-bdd3-b7ee85866fbf\") " Oct 31 00:51:57.320566 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:57.320507 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06704c9d-24a7-4e63-bdd3-b7ee85866fbf-kube-api-access-tkckd\" (UniqueName: \"kubernetes.io/projected/06704c9d-24a7-4e63-bdd3-b7ee85866fbf-kube-api-access-tkckd\") pod \"06704c9d-24a7-4e63-bdd3-b7ee85866fbf\" (UID: \"06704c9d-24a7-4e63-bdd3-b7ee85866fbf\") " Oct 31 00:51:57.320701 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:57.320597 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/207c55f2-0d5c-497c-b021-c6ba4939370b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/207c55f2-0d5c-497c-b021-c6ba4939370b-plugins-config-volume\") pod \"207c55f2-0d5c-497c-b021-c6ba4939370b\" (UID: \"207c55f2-0d5c-497c-b021-c6ba4939370b\") " Oct 31 00:51:57.320879 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:57.320853 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06704c9d-24a7-4e63-bdd3-b7ee85866fbf-plugins-config-volume" pod "06704c9d-24a7-4e63-bdd3-b7ee85866fbf" (UID: "06704c9d-24a7-4e63-bdd3-b7ee85866fbf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:51:57.321024 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:57.320995 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/207c55f2-0d5c-497c-b021-c6ba4939370b-plugins-config-volume" pod "207c55f2-0d5c-497c-b021-c6ba4939370b" (UID: "207c55f2-0d5c-497c-b021-c6ba4939370b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:51:57.321322 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:57.321293 5222 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/207c55f2-0d5c-497c-b021-c6ba4939370b-plugins-config-volume\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:51:57.321322 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:57.321324 5222 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06704c9d-24a7-4e63-bdd3-b7ee85866fbf-plugins-config-volume\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:51:57.323891 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:57.322946 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/207c55f2-0d5c-497c-b021-c6ba4939370b-kube-api-access-jx7sm" pod "207c55f2-0d5c-497c-b021-c6ba4939370b" (UID: "207c55f2-0d5c-497c-b021-c6ba4939370b"). InnerVolumeSpecName "kube-api-access-jx7sm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:51:57.324579 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:57.324559 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06704c9d-24a7-4e63-bdd3-b7ee85866fbf-kube-api-access-tkckd" pod "06704c9d-24a7-4e63-bdd3-b7ee85866fbf" (UID: "06704c9d-24a7-4e63-bdd3-b7ee85866fbf"). InnerVolumeSpecName "kube-api-access-tkckd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:51:57.422512 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:57.422428 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jx7sm\" (UniqueName: \"kubernetes.io/projected/207c55f2-0d5c-497c-b021-c6ba4939370b-kube-api-access-jx7sm\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:51:57.422512 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:57.422470 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkckd\" (UniqueName: \"kubernetes.io/projected/06704c9d-24a7-4e63-bdd3-b7ee85866fbf-kube-api-access-tkckd\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:51:57.692448 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:57.692348 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-e8ba06-bb9fb667f-x6fzk"] Oct 31 00:51:57.692448 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:57.692406 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-e8ba06-bb9fb667f-x6fzk"] Oct 31 00:51:57.881981 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:57.881930 5222 patch_prober.go:29] interesting pod/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-2rtnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" start-of-body= Oct 31 00:51:57.882733 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:57.882676 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-2rtnt" podUID="69c94b24-49c5-49b4-825b-50778d04669a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" Oct 31 00:51:58.264812 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:58.264778 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e8ba06-7b548bc5f-m9r6m" Oct 31 00:51:58.265000 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:58.264813 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e8ba06-7b548bc5f-m9r6m" event={"ID":"06704c9d-24a7-4e63-bdd3-b7ee85866fbf","Type":"ContainerDied","Data":"753a75e65d00462b2fbf77c8b79f49a15d2b5fc6d3383f9d688dbc503ef2c511"} Oct 31 00:51:58.265000 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:58.264891 5222 scope.go:122] "RemoveContainer" containerID="c7ad93d2942a5d855c70743f3b243a5ccbfe1d8a5b8e0972bed3dee955d231f6" Oct 31 00:51:58.483565 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:58.483514 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-e8ba06-7b548bc5f-m9r6m"] Oct 31 00:51:58.488153 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:58.488125 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-e8ba06-7b548bc5f-m9r6m"] Oct 31 00:51:58.881176 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:58.881141 5222 patch_prober.go:29] interesting pod/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-2rtnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" start-of-body= Oct 31 00:51:58.881410 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:58.881195 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-2rtnt" podUID="69c94b24-49c5-49b4-825b-50778d04669a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" Oct 31 00:51:59.568782 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:59.568740 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e9c3c1-rpj5q"] Oct 31 00:51:59.612777 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:59.612741 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e9c3c1-rpj5q"] Oct 31 00:51:59.670655 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:59.670626 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06704c9d-24a7-4e63-bdd3-b7ee85866fbf" path="/var/lib/kubelet/pods/06704c9d-24a7-4e63-bdd3-b7ee85866fbf/volumes" Oct 31 00:51:59.671175 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:59.671160 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="207c55f2-0d5c-497c-b021-c6ba4939370b" path="/var/lib/kubelet/pods/207c55f2-0d5c-497c-b021-c6ba4939370b/volumes" Oct 31 00:51:59.671706 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:59.671691 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a5bb0fe-0d5e-48da-bbee-839acf6a4b64" path="/var/lib/kubelet/pods/6a5bb0fe-0d5e-48da-bbee-839acf6a4b64/volumes" Oct 31 00:51:59.882025 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:59.881940 5222 patch_prober.go:29] interesting pod/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-2rtnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" start-of-body= Oct 31 00:51:59.882025 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:51:59.881992 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-2rtnt" podUID="69c94b24-49c5-49b4-825b-50778d04669a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" Oct 31 00:52:00.959174 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:00.958250 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-2rtnt" Oct 31 00:52:00.960289 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:00.960144 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-2rtnt" Oct 31 00:52:00.960899 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:00.960876 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-2rtnt" Oct 31 00:52:05.891648 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:52:05.891613 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 00:52:08.689583 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:08.689549 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-mwd5g"] Oct 31 00:52:08.690488 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:08.690466 5222 cpu_manager.go:414] "containerMap: removing container" podUID="06704c9d-24a7-4e63-bdd3-b7ee85866fbf" containerName="payload-pre-processing" Oct 31 00:52:08.690488 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:08.690487 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06704c9d-24a7-4e63-bdd3-b7ee85866fbf" containerName="payload-pre-processing" Oct 31 00:52:08.690801 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:08.690505 5222 cpu_manager.go:414] "containerMap: removing container" podUID="d0694975-c551-4a8a-b296-eaee47077b0c" containerName="istio-proxy" Oct 31 00:52:08.690801 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:08.690513 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0694975-c551-4a8a-b296-eaee47077b0c" containerName="istio-proxy" Oct 31 00:52:08.690801 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:08.690553 5222 cpu_manager.go:414] "containerMap: removing container" podUID="207c55f2-0d5c-497c-b021-c6ba4939370b" containerName="payload-processing" Oct 31 00:52:08.690801 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:08.690561 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="207c55f2-0d5c-497c-b021-c6ba4939370b" containerName="payload-processing" Oct 31 00:52:08.690801 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:08.690686 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="06704c9d-24a7-4e63-bdd3-b7ee85866fbf" containerName="payload-pre-processing" Oct 31 00:52:08.690801 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:08.690703 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="207c55f2-0d5c-497c-b021-c6ba4939370b" containerName="payload-processing" Oct 31 00:52:08.696261 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:08.696241 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-mwd5g" Oct 31 00:52:08.739402 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:08.739355 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-hpj9w"] Oct 31 00:52:08.743613 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:08.743590 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-hpj9w" Oct 31 00:52:08.792814 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:08.792770 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-mwd5g"] Oct 31 00:52:08.838845 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:08.838807 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-hpj9w"] Oct 31 00:52:08.839646 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:08.839613 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb14bbdd-960a-457c-900c-7de064657e5e-istio-data\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-hpj9w\" (UID: \"cb14bbdd-960a-457c-900c-7de064657e5e\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-hpj9w" Oct 31 00:52:08.839823 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:08.839653 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e32ce001-80c9-4be1-8a5a-46de777f11be-istio-token\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-mwd5g\" (UID: \"e32ce001-80c9-4be1-8a5a-46de777f11be\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-mwd5g" Oct 31 00:52:08.839823 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:08.839688 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb14bbdd-960a-457c-900c-7de064657e5e-istio-token\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-hpj9w\" (UID: \"cb14bbdd-960a-457c-900c-7de064657e5e\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-hpj9w" Oct 31 00:52:08.839823 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:08.839783 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb14bbdd-960a-457c-900c-7de064657e5e-istio-podinfo\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-hpj9w\" (UID: \"cb14bbdd-960a-457c-900c-7de064657e5e\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-hpj9w" Oct 31 00:52:08.840002 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:08.839887 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e32ce001-80c9-4be1-8a5a-46de777f11be-workload-socket\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-mwd5g\" (UID: \"e32ce001-80c9-4be1-8a5a-46de777f11be\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-mwd5g" Oct 31 00:52:08.840080 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:08.840034 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e32ce001-80c9-4be1-8a5a-46de777f11be-credential-socket\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-mwd5g\" (UID: \"e32ce001-80c9-4be1-8a5a-46de777f11be\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-mwd5g" Oct 31 00:52:08.840145 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:08.840095 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb14bbdd-960a-457c-900c-7de064657e5e-credential-socket\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-hpj9w\" (UID: \"cb14bbdd-960a-457c-900c-7de064657e5e\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-hpj9w" Oct 31 00:52:08.840208 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:08.840191 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e32ce001-80c9-4be1-8a5a-46de777f11be-istiod-ca-cert\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-mwd5g\" (UID: \"e32ce001-80c9-4be1-8a5a-46de777f11be\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-mwd5g" Oct 31 00:52:08.840262 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:08.840236 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e32ce001-80c9-4be1-8a5a-46de777f11be-istio-envoy\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-mwd5g\" (UID: \"e32ce001-80c9-4be1-8a5a-46de777f11be\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-mwd5g" Oct 31 00:52:08.840324 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:08.840273 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb14bbdd-960a-457c-900c-7de064657e5e-istiod-ca-cert\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-hpj9w\" (UID: \"cb14bbdd-960a-457c-900c-7de064657e5e\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-hpj9w" Oct 31 00:52:08.840324 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:08.840312 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb14bbdd-960a-457c-900c-7de064657e5e-workload-certs\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-hpj9w\" (UID: \"cb14bbdd-960a-457c-900c-7de064657e5e\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-hpj9w" Oct 31 00:52:08.840477 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:08.840347 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j567t\" (UniqueName: \"kubernetes.io/projected/e32ce001-80c9-4be1-8a5a-46de777f11be-kube-api-access-j567t\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-mwd5g\" (UID: \"e32ce001-80c9-4be1-8a5a-46de777f11be\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-mwd5g" Oct 31 00:52:08.840477 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:08.840428 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb14bbdd-960a-457c-900c-7de064657e5e-istio-envoy\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-hpj9w\" (UID: \"cb14bbdd-960a-457c-900c-7de064657e5e\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-hpj9w" Oct 31 00:52:08.840931 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:08.840474 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb14bbdd-960a-457c-900c-7de064657e5e-workload-socket\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-hpj9w\" (UID: \"cb14bbdd-960a-457c-900c-7de064657e5e\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-hpj9w" Oct 31 00:52:08.840931 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:08.840515 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5gbs\" (UniqueName: \"kubernetes.io/projected/cb14bbdd-960a-457c-900c-7de064657e5e-kube-api-access-c5gbs\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-hpj9w\" (UID: \"cb14bbdd-960a-457c-900c-7de064657e5e\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-hpj9w" Oct 31 00:52:08.840931 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:08.840539 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e32ce001-80c9-4be1-8a5a-46de777f11be-istio-data\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-mwd5g\" (UID: \"e32ce001-80c9-4be1-8a5a-46de777f11be\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-mwd5g" Oct 31 00:52:08.840931 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:08.840640 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e32ce001-80c9-4be1-8a5a-46de777f11be-istio-podinfo\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-mwd5g\" (UID: \"e32ce001-80c9-4be1-8a5a-46de777f11be\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-mwd5g" Oct 31 00:52:08.840931 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:08.840695 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e32ce001-80c9-4be1-8a5a-46de777f11be-workload-certs\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-mwd5g\" (UID: \"e32ce001-80c9-4be1-8a5a-46de777f11be\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-mwd5g" Oct 31 00:52:08.942774 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:08.942625 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e32ce001-80c9-4be1-8a5a-46de777f11be-istiod-ca-cert\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-mwd5g\" (UID: \"e32ce001-80c9-4be1-8a5a-46de777f11be\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-mwd5g" Oct 31 00:52:08.942774 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:08.942693 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e32ce001-80c9-4be1-8a5a-46de777f11be-istio-envoy\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-mwd5g\" (UID: \"e32ce001-80c9-4be1-8a5a-46de777f11be\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-mwd5g" Oct 31 00:52:08.942774 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:08.942734 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb14bbdd-960a-457c-900c-7de064657e5e-istiod-ca-cert\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-hpj9w\" (UID: \"cb14bbdd-960a-457c-900c-7de064657e5e\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-hpj9w" Oct 31 00:52:08.942774 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:08.942775 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb14bbdd-960a-457c-900c-7de064657e5e-workload-certs\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-hpj9w\" (UID: \"cb14bbdd-960a-457c-900c-7de064657e5e\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-hpj9w" Oct 31 00:52:08.943176 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:08.942808 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j567t\" (UniqueName: \"kubernetes.io/projected/e32ce001-80c9-4be1-8a5a-46de777f11be-kube-api-access-j567t\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-mwd5g\" (UID: \"e32ce001-80c9-4be1-8a5a-46de777f11be\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-mwd5g" Oct 31 00:52:08.943176 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:08.942836 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb14bbdd-960a-457c-900c-7de064657e5e-istio-envoy\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-hpj9w\" (UID: \"cb14bbdd-960a-457c-900c-7de064657e5e\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-hpj9w" Oct 31 00:52:08.943176 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:08.942873 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb14bbdd-960a-457c-900c-7de064657e5e-workload-socket\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-hpj9w\" (UID: \"cb14bbdd-960a-457c-900c-7de064657e5e\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-hpj9w" Oct 31 00:52:08.943176 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:08.942909 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5gbs\" (UniqueName: \"kubernetes.io/projected/cb14bbdd-960a-457c-900c-7de064657e5e-kube-api-access-c5gbs\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-hpj9w\" (UID: \"cb14bbdd-960a-457c-900c-7de064657e5e\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-hpj9w" Oct 31 00:52:08.943176 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:08.942933 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e32ce001-80c9-4be1-8a5a-46de777f11be-istio-data\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-mwd5g\" (UID: \"e32ce001-80c9-4be1-8a5a-46de777f11be\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-mwd5g" Oct 31 00:52:08.943176 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:08.943043 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e32ce001-80c9-4be1-8a5a-46de777f11be-istio-podinfo\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-mwd5g\" (UID: \"e32ce001-80c9-4be1-8a5a-46de777f11be\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-mwd5g" Oct 31 00:52:08.943176 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:08.943105 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e32ce001-80c9-4be1-8a5a-46de777f11be-workload-certs\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-mwd5g\" (UID: \"e32ce001-80c9-4be1-8a5a-46de777f11be\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-mwd5g" Oct 31 00:52:08.943176 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:08.943131 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb14bbdd-960a-457c-900c-7de064657e5e-istio-data\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-hpj9w\" (UID: \"cb14bbdd-960a-457c-900c-7de064657e5e\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-hpj9w" Oct 31 00:52:08.943176 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:08.943153 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e32ce001-80c9-4be1-8a5a-46de777f11be-istio-token\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-mwd5g\" (UID: \"e32ce001-80c9-4be1-8a5a-46de777f11be\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-mwd5g" Oct 31 00:52:08.943176 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:08.943178 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb14bbdd-960a-457c-900c-7de064657e5e-istio-token\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-hpj9w\" (UID: \"cb14bbdd-960a-457c-900c-7de064657e5e\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-hpj9w" Oct 31 00:52:08.943760 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:08.943297 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb14bbdd-960a-457c-900c-7de064657e5e-istio-podinfo\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-hpj9w\" (UID: \"cb14bbdd-960a-457c-900c-7de064657e5e\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-hpj9w" Oct 31 00:52:08.943760 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:08.943362 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e32ce001-80c9-4be1-8a5a-46de777f11be-workload-socket\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-mwd5g\" (UID: \"e32ce001-80c9-4be1-8a5a-46de777f11be\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-mwd5g" Oct 31 00:52:08.943760 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:08.943455 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e32ce001-80c9-4be1-8a5a-46de777f11be-credential-socket\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-mwd5g\" (UID: \"e32ce001-80c9-4be1-8a5a-46de777f11be\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-mwd5g" Oct 31 00:52:08.943760 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:08.943482 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb14bbdd-960a-457c-900c-7de064657e5e-credential-socket\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-hpj9w\" (UID: \"cb14bbdd-960a-457c-900c-7de064657e5e\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-hpj9w" Oct 31 00:52:08.943760 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:08.943504 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e32ce001-80c9-4be1-8a5a-46de777f11be-istiod-ca-cert\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-mwd5g\" (UID: \"e32ce001-80c9-4be1-8a5a-46de777f11be\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-mwd5g" Oct 31 00:52:08.944038 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:08.943820 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb14bbdd-960a-457c-900c-7de064657e5e-credential-socket\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-hpj9w\" (UID: \"cb14bbdd-960a-457c-900c-7de064657e5e\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-hpj9w" Oct 31 00:52:08.944541 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:08.944513 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb14bbdd-960a-457c-900c-7de064657e5e-workload-socket\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-hpj9w\" (UID: \"cb14bbdd-960a-457c-900c-7de064657e5e\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-hpj9w" Oct 31 00:52:08.944713 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:08.944567 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb14bbdd-960a-457c-900c-7de064657e5e-workload-certs\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-hpj9w\" (UID: \"cb14bbdd-960a-457c-900c-7de064657e5e\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-hpj9w" Oct 31 00:52:08.945267 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:08.945224 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb14bbdd-960a-457c-900c-7de064657e5e-istiod-ca-cert\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-hpj9w\" (UID: \"cb14bbdd-960a-457c-900c-7de064657e5e\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-hpj9w" Oct 31 00:52:08.945452 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:08.945321 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb14bbdd-960a-457c-900c-7de064657e5e-istio-data\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-hpj9w\" (UID: \"cb14bbdd-960a-457c-900c-7de064657e5e\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-hpj9w" Oct 31 00:52:08.945666 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:08.945641 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e32ce001-80c9-4be1-8a5a-46de777f11be-workload-socket\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-mwd5g\" (UID: \"e32ce001-80c9-4be1-8a5a-46de777f11be\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-mwd5g" Oct 31 00:52:08.946136 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:08.946099 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e32ce001-80c9-4be1-8a5a-46de777f11be-workload-certs\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-mwd5g\" (UID: \"e32ce001-80c9-4be1-8a5a-46de777f11be\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-mwd5g" Oct 31 00:52:08.946434 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:08.946412 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e32ce001-80c9-4be1-8a5a-46de777f11be-istio-data\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-mwd5g\" (UID: \"e32ce001-80c9-4be1-8a5a-46de777f11be\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-mwd5g" Oct 31 00:52:08.946651 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:08.946629 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e32ce001-80c9-4be1-8a5a-46de777f11be-credential-socket\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-mwd5g\" (UID: \"e32ce001-80c9-4be1-8a5a-46de777f11be\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-mwd5g" Oct 31 00:52:08.947605 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:08.947584 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e32ce001-80c9-4be1-8a5a-46de777f11be-istio-podinfo\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-mwd5g\" (UID: \"e32ce001-80c9-4be1-8a5a-46de777f11be\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-mwd5g" Oct 31 00:52:08.952754 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:08.952729 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e32ce001-80c9-4be1-8a5a-46de777f11be-istio-envoy\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-mwd5g\" (UID: \"e32ce001-80c9-4be1-8a5a-46de777f11be\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-mwd5g" Oct 31 00:52:08.952858 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:08.952751 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb14bbdd-960a-457c-900c-7de064657e5e-istio-envoy\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-hpj9w\" (UID: \"cb14bbdd-960a-457c-900c-7de064657e5e\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-hpj9w" Oct 31 00:52:08.954835 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:08.954809 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb14bbdd-960a-457c-900c-7de064657e5e-istio-podinfo\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-hpj9w\" (UID: \"cb14bbdd-960a-457c-900c-7de064657e5e\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-hpj9w" Oct 31 00:52:09.015293 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:09.015251 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j567t\" (UniqueName: \"kubernetes.io/projected/e32ce001-80c9-4be1-8a5a-46de777f11be-kube-api-access-j567t\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-mwd5g\" (UID: \"e32ce001-80c9-4be1-8a5a-46de777f11be\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-mwd5g" Oct 31 00:52:09.015708 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:09.015682 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5gbs\" (UniqueName: \"kubernetes.io/projected/cb14bbdd-960a-457c-900c-7de064657e5e-kube-api-access-c5gbs\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-hpj9w\" (UID: \"cb14bbdd-960a-457c-900c-7de064657e5e\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-hpj9w" Oct 31 00:52:09.015814 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:09.015725 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb14bbdd-960a-457c-900c-7de064657e5e-istio-token\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-hpj9w\" (UID: \"cb14bbdd-960a-457c-900c-7de064657e5e\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-hpj9w" Oct 31 00:52:09.016231 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:09.016213 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e32ce001-80c9-4be1-8a5a-46de777f11be-istio-token\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-mwd5g\" (UID: \"e32ce001-80c9-4be1-8a5a-46de777f11be\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-mwd5g" Oct 31 00:52:09.057998 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:09.057963 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-hpj9w" Oct 31 00:52:09.314177 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:09.314133 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-mwd5g" Oct 31 00:52:09.493886 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:09.493845 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-hpj9w"] Oct 31 00:52:09.505089 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:09.503418 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:52:09.505089 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:09.503494 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:52:09.505089 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:09.503522 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:52:09.724903 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:09.724769 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:52:09.724903 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:09.724840 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:52:09.724903 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:09.724875 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:52:09.729790 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:09.729767 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-mwd5g"] Oct 31 00:52:10.490859 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:10.490489 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-hpj9w" event={"ID":"cb14bbdd-960a-457c-900c-7de064657e5e","Type":"ContainerStarted","Data":"218749fd9f675eb9807f080764790e06e6f3cc6f089833c55695c269c84bb890"} Oct 31 00:52:10.490859 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:10.490671 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-hpj9w" event={"ID":"cb14bbdd-960a-457c-900c-7de064657e5e","Type":"ContainerStarted","Data":"88a789530b09b9285299a5e93bc99a711c8733f7f2cbd9b247358a3add2a60f5"} Oct 31 00:52:10.492703 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:10.492671 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-mwd5g" event={"ID":"e32ce001-80c9-4be1-8a5a-46de777f11be","Type":"ContainerStarted","Data":"18004b16438c027ac33f5ecff4c92cc19fb4e0676a8890c022f2c889481e6540"} Oct 31 00:52:10.492879 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:10.492724 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-mwd5g" event={"ID":"e32ce001-80c9-4be1-8a5a-46de777f11be","Type":"ContainerStarted","Data":"ff84d7df07c31dcc480d926de626ee54bd59c4a24bdbde3c0d672dd3d8e52fc2"} Oct 31 00:52:10.629984 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:10.629176 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-hpj9w" podStartSLOduration=2.629159393 podStartE2EDuration="2.629159393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:52:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:52:10.627564085 +0000 UTC m=+2223.098733835" watchObservedRunningTime="2026-10-31 00:52:10.629159393 +0000 UTC m=+2223.100329142" Oct 31 00:52:10.780463 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:10.762312 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-mwd5g" podStartSLOduration=2.762292139 podStartE2EDuration="2.762292139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:52:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:52:10.759916554 +0000 UTC m=+2223.231086304" watchObservedRunningTime="2026-10-31 00:52:10.762292139 +0000 UTC m=+2223.233461888" Oct 31 00:52:11.069453 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:11.068802 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-hpj9w" Oct 31 00:52:11.077395 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:11.072979 5222 patch_prober.go:29] interesting pod/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-hpj9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body= Oct 31 00:52:11.077395 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:11.073056 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-hpj9w" podUID="cb14bbdd-960a-457c-900c-7de064657e5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" Oct 31 00:52:11.318421 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:11.315763 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-mwd5g" Oct 31 00:52:11.332492 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:11.324524 5222 patch_prober.go:29] interesting pod/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-mwd5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" start-of-body= Oct 31 00:52:11.332492 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:11.324604 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-mwd5g" podUID="e32ce001-80c9-4be1-8a5a-46de777f11be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" Oct 31 00:52:12.062522 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:12.062484 5222 patch_prober.go:29] interesting pod/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-hpj9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body= Oct 31 00:52:12.063171 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:12.063127 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-hpj9w" podUID="cb14bbdd-960a-457c-900c-7de064657e5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" Oct 31 00:52:12.185485 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:12.185437 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-2594a9-s2225"] Oct 31 00:52:12.191351 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:12.191298 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-2594a9-s2225"] Oct 31 00:52:12.314647 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:12.314551 5222 patch_prober.go:29] interesting pod/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-mwd5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" start-of-body= Oct 31 00:52:12.314647 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:12.314614 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-mwd5g" podUID="e32ce001-80c9-4be1-8a5a-46de777f11be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" Oct 31 00:52:13.072183 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:13.072143 5222 patch_prober.go:29] interesting pod/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-hpj9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body= Oct 31 00:52:13.072867 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:13.072833 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-hpj9w" podUID="cb14bbdd-960a-457c-900c-7de064657e5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" Oct 31 00:52:13.315400 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:13.315332 5222 patch_prober.go:29] interesting pod/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-mwd5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" start-of-body= Oct 31 00:52:13.315586 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:13.315406 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-mwd5g" podUID="e32ce001-80c9-4be1-8a5a-46de777f11be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" Oct 31 00:52:13.700750 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:13.700718 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f25858c-4c7a-41dc-915b-72accf53c267" path="/var/lib/kubelet/pods/6f25858c-4c7a-41dc-915b-72accf53c267/volumes" Oct 31 00:52:14.061394 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:14.060823 5222 patch_prober.go:29] interesting pod/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-hpj9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body= Oct 31 00:52:14.061394 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:14.060894 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-hpj9w" podUID="cb14bbdd-960a-457c-900c-7de064657e5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" Oct 31 00:52:14.314717 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:14.314630 5222 patch_prober.go:29] interesting pod/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-mwd5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" start-of-body= Oct 31 00:52:14.314717 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:14.314684 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-mwd5g" podUID="e32ce001-80c9-4be1-8a5a-46de777f11be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" Oct 31 00:52:15.059397 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:15.058852 5222 patch_prober.go:29] interesting pod/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-hpj9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body= Oct 31 00:52:15.059397 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:15.058921 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-hpj9w" podUID="cb14bbdd-960a-457c-900c-7de064657e5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" Oct 31 00:52:15.315357 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:15.315262 5222 patch_prober.go:29] interesting pod/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-mwd5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" start-of-body= Oct 31 00:52:15.315357 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:15.315320 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-mwd5g" podUID="e32ce001-80c9-4be1-8a5a-46de777f11be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" Oct 31 00:52:16.061832 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:16.061793 5222 patch_prober.go:29] interesting pod/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-hpj9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body= Oct 31 00:52:16.062103 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:16.062073 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-hpj9w" podUID="cb14bbdd-960a-457c-900c-7de064657e5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" Oct 31 00:52:16.314882 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:16.314776 5222 patch_prober.go:29] interesting pod/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-mwd5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" start-of-body= Oct 31 00:52:16.314882 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:16.314869 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-mwd5g" podUID="e32ce001-80c9-4be1-8a5a-46de777f11be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" Oct 31 00:52:16.386953 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:16.386905 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bdfc3f86-6vskf"] Oct 31 00:52:16.393777 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:16.393749 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdfc3f86-6vskf" Oct 31 00:52:16.421311 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:16.421279 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k86tx\" (UniqueName: \"kubernetes.io/projected/7f697756-a33a-4883-a075-a97f84e3d3c4-kube-api-access-k86tx\") pod \"maas-api-revoke-sub-bdfc3f86-6vskf\" (UID: \"7f697756-a33a-4883-a075-a97f84e3d3c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdfc3f86-6vskf" Oct 31 00:52:16.421531 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:16.421468 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f697756-a33a-4883-a075-a97f84e3d3c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bdfc3f86-6vskf\" (UID: \"7f697756-a33a-4883-a075-a97f84e3d3c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdfc3f86-6vskf" Oct 31 00:52:16.421624 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:16.421565 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f697756-a33a-4883-a075-a97f84e3d3c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bdfc3f86-6vskf\" (UID: \"7f697756-a33a-4883-a075-a97f84e3d3c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdfc3f86-6vskf" Oct 31 00:52:16.452635 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:16.452603 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:52:16.452997 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:16.452977 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:52:16.453172 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:16.452977 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfz9z\"" Oct 31 00:52:16.453172 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:16.453060 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:52:16.491431 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:16.475415 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bdfc3f86-6vskf"] Oct 31 00:52:16.548391 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:16.544479 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f697756-a33a-4883-a075-a97f84e3d3c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bdfc3f86-6vskf\" (UID: \"7f697756-a33a-4883-a075-a97f84e3d3c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdfc3f86-6vskf" Oct 31 00:52:16.548391 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:16.544797 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f697756-a33a-4883-a075-a97f84e3d3c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bdfc3f86-6vskf\" (UID: \"7f697756-a33a-4883-a075-a97f84e3d3c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdfc3f86-6vskf" Oct 31 00:52:16.548391 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:16.545448 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k86tx\" (UniqueName: \"kubernetes.io/projected/7f697756-a33a-4883-a075-a97f84e3d3c4-kube-api-access-k86tx\") pod \"maas-api-revoke-sub-bdfc3f86-6vskf\" (UID: \"7f697756-a33a-4883-a075-a97f84e3d3c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdfc3f86-6vskf" Oct 31 00:52:16.548391 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:16.547739 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f697756-a33a-4883-a075-a97f84e3d3c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bdfc3f86-6vskf\" (UID: \"7f697756-a33a-4883-a075-a97f84e3d3c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdfc3f86-6vskf" Oct 31 00:52:16.556394 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:16.554890 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f697756-a33a-4883-a075-a97f84e3d3c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bdfc3f86-6vskf\" (UID: \"7f697756-a33a-4883-a075-a97f84e3d3c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdfc3f86-6vskf" Oct 31 00:52:16.610752 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:16.610663 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k86tx\" (UniqueName: \"kubernetes.io/projected/7f697756-a33a-4883-a075-a97f84e3d3c4-kube-api-access-k86tx\") pod \"maas-api-revoke-sub-bdfc3f86-6vskf\" (UID: \"7f697756-a33a-4883-a075-a97f84e3d3c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdfc3f86-6vskf" Oct 31 00:52:16.710498 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:16.710174 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdfc3f86-6vskf" Oct 31 00:52:17.151511 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:17.151473 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-hpj9w" Oct 31 00:52:17.151937 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:17.151912 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-hpj9w" Oct 31 00:52:17.155492 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:17.155474 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-hpj9w" Oct 31 00:52:17.165728 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:52:17.165697 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f697756_a33a_4883_a075_a97f84e3d3c4.slice/crio-29bbd6d62eab287a77226fb2be1ae1067efd4db88987e283e3b10dd168ec3808 WatchSource:0}: Error finding container 29bbd6d62eab287a77226fb2be1ae1067efd4db88987e283e3b10dd168ec3808: Status 404 returned error can't find the container with id 29bbd6d62eab287a77226fb2be1ae1067efd4db88987e283e3b10dd168ec3808 Oct 31 00:52:17.169166 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:17.169137 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bdfc3f86-6vskf"] Oct 31 00:52:17.377747 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:17.377717 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-mwd5g" Oct 31 00:52:17.378486 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:17.378148 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-mwd5g" Oct 31 00:52:17.380405 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:17.380343 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-mwd5g" Oct 31 00:52:17.570790 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:17.570750 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdfc3f86-6vskf" event={"ID":"7f697756-a33a-4883-a075-a97f84e3d3c4","Type":"ContainerStarted","Data":"29bbd6d62eab287a77226fb2be1ae1067efd4db88987e283e3b10dd168ec3808"} Oct 31 00:52:18.580187 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:18.580149 5222 generic.go:356] "Generic (PLEG): container finished" podID="7f697756-a33a-4883-a075-a97f84e3d3c4" containerID="e34bc7b8fe27c5c92d02a809c201a47a18499a5cef9c7fceeae43a301adf5c17" exitCode=0 Oct 31 00:52:18.580649 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:18.580245 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdfc3f86-6vskf" event={"ID":"7f697756-a33a-4883-a075-a97f84e3d3c4","Type":"ContainerDied","Data":"e34bc7b8fe27c5c92d02a809c201a47a18499a5cef9c7fceeae43a301adf5c17"} Oct 31 00:52:19.178609 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:19.178575 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-2594a9-openshift-default-8694ddf5f7-v4jsh"] Oct 31 00:52:19.178844 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:19.178816 5222 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-2594a9-openshift-default-8694ddf5f7-v4jsh" podUID="4c57fa36-80bb-403d-97a3-b18601c386aa" containerName="istio-proxy" containerID="cri-o://564a07ec10feb3039d0b988b7be438f8800118f84e222f998af0fad078434a94" gracePeriod=30 Oct 31 00:52:19.762226 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:19.762201 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdfc3f86-6vskf" Oct 31 00:52:19.908893 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:19.908799 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f697756-a33a-4883-a075-a97f84e3d3c4-kube-api-access-k86tx\" (UniqueName: \"kubernetes.io/projected/7f697756-a33a-4883-a075-a97f84e3d3c4-kube-api-access-k86tx\") pod \"7f697756-a33a-4883-a075-a97f84e3d3c4\" (UID: \"7f697756-a33a-4883-a075-a97f84e3d3c4\") " Oct 31 00:52:19.909089 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:19.908900 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f697756-a33a-4883-a075-a97f84e3d3c4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f697756-a33a-4883-a075-a97f84e3d3c4-maas-api-xks-ca\") pod \"7f697756-a33a-4883-a075-a97f84e3d3c4\" (UID: \"7f697756-a33a-4883-a075-a97f84e3d3c4\") " Oct 31 00:52:19.909089 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:19.909006 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f697756-a33a-4883-a075-a97f84e3d3c4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f697756-a33a-4883-a075-a97f84e3d3c4-maas-api-openshift-ca\") pod \"7f697756-a33a-4883-a075-a97f84e3d3c4\" (UID: \"7f697756-a33a-4883-a075-a97f84e3d3c4\") " Oct 31 00:52:19.909642 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:19.909611 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f697756-a33a-4883-a075-a97f84e3d3c4-maas-api-openshift-ca" pod "7f697756-a33a-4883-a075-a97f84e3d3c4" (UID: "7f697756-a33a-4883-a075-a97f84e3d3c4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:52:19.909942 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:19.909833 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f697756-a33a-4883-a075-a97f84e3d3c4-maas-api-openshift-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:52:19.911484 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:19.911451 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f697756-a33a-4883-a075-a97f84e3d3c4-maas-api-xks-ca" pod "7f697756-a33a-4883-a075-a97f84e3d3c4" (UID: "7f697756-a33a-4883-a075-a97f84e3d3c4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:52:19.911786 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:19.911753 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f697756-a33a-4883-a075-a97f84e3d3c4-kube-api-access-k86tx" pod "7f697756-a33a-4883-a075-a97f84e3d3c4" (UID: "7f697756-a33a-4883-a075-a97f84e3d3c4"). InnerVolumeSpecName "kube-api-access-k86tx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:52:20.010975 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:20.010938 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k86tx\" (UniqueName: \"kubernetes.io/projected/7f697756-a33a-4883-a075-a97f84e3d3c4-kube-api-access-k86tx\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:52:20.010975 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:20.010973 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f697756-a33a-4883-a075-a97f84e3d3c4-maas-api-xks-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:52:20.596944 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:20.596911 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdfc3f86-6vskf" Oct 31 00:52:20.597191 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:20.596947 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdfc3f86-6vskf" event={"ID":"7f697756-a33a-4883-a075-a97f84e3d3c4","Type":"ContainerDied","Data":"29bbd6d62eab287a77226fb2be1ae1067efd4db88987e283e3b10dd168ec3808"} Oct 31 00:52:20.597191 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:20.596999 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="29bbd6d62eab287a77226fb2be1ae1067efd4db88987e283e3b10dd168ec3808" Oct 31 00:52:24.578492 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:24.578462 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-2594a9-openshift-default-8694ddf5f7-v4jsh" Oct 31 00:52:24.632072 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:24.632041 5222 generic.go:356] "Generic (PLEG): container finished" podID="4c57fa36-80bb-403d-97a3-b18601c386aa" containerID="564a07ec10feb3039d0b988b7be438f8800118f84e222f998af0fad078434a94" exitCode=0 Oct 31 00:52:24.632274 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:24.632074 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-2594a9-openshift-default-8694ddf5f7-v4jsh" event={"ID":"4c57fa36-80bb-403d-97a3-b18601c386aa","Type":"ContainerDied","Data":"564a07ec10feb3039d0b988b7be438f8800118f84e222f998af0fad078434a94"} Oct 31 00:52:24.632274 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:24.632101 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-2594a9-openshift-default-8694ddf5f7-v4jsh" Oct 31 00:52:24.632274 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:24.632148 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-2594a9-openshift-default-8694ddf5f7-v4jsh" event={"ID":"4c57fa36-80bb-403d-97a3-b18601c386aa","Type":"ContainerDied","Data":"35532464daf3da8c07d46b0578bd0188319988020a4100e92a79900468105d94"} Oct 31 00:52:24.632274 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:24.632173 5222 scope.go:122] "RemoveContainer" containerID="564a07ec10feb3039d0b988b7be438f8800118f84e222f998af0fad078434a94" Oct 31 00:52:24.643257 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:24.643236 5222 scope.go:122] "RemoveContainer" containerID="0d360cc605ccde706b1a6139edee34d8c211f6e0ed82e81d3ee0caa7de814e0c" Oct 31 00:52:24.654202 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:24.654179 5222 scope.go:122] "RemoveContainer" containerID="564a07ec10feb3039d0b988b7be438f8800118f84e222f998af0fad078434a94" Oct 31 00:52:24.654591 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:52:24.654571 5222 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"564a07ec10feb3039d0b988b7be438f8800118f84e222f998af0fad078434a94\": container with ID starting with 564a07ec10feb3039d0b988b7be438f8800118f84e222f998af0fad078434a94 not found: ID does not exist" containerID="564a07ec10feb3039d0b988b7be438f8800118f84e222f998af0fad078434a94" Oct 31 00:52:24.654672 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:24.654602 5222 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"564a07ec10feb3039d0b988b7be438f8800118f84e222f998af0fad078434a94"} err="failed to get container status \"564a07ec10feb3039d0b988b7be438f8800118f84e222f998af0fad078434a94\": rpc error: code = NotFound desc = could not find container \"564a07ec10feb3039d0b988b7be438f8800118f84e222f998af0fad078434a94\": container with ID starting with 564a07ec10feb3039d0b988b7be438f8800118f84e222f998af0fad078434a94 not found: ID does not exist" Oct 31 00:52:24.654672 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:24.654621 5222 scope.go:122] "RemoveContainer" containerID="0d360cc605ccde706b1a6139edee34d8c211f6e0ed82e81d3ee0caa7de814e0c" Oct 31 00:52:24.654890 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:52:24.654869 5222 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d360cc605ccde706b1a6139edee34d8c211f6e0ed82e81d3ee0caa7de814e0c\": container with ID starting with 0d360cc605ccde706b1a6139edee34d8c211f6e0ed82e81d3ee0caa7de814e0c not found: ID does not exist" containerID="0d360cc605ccde706b1a6139edee34d8c211f6e0ed82e81d3ee0caa7de814e0c" Oct 31 00:52:24.654944 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:24.654901 5222 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d360cc605ccde706b1a6139edee34d8c211f6e0ed82e81d3ee0caa7de814e0c"} err="failed to get container status \"0d360cc605ccde706b1a6139edee34d8c211f6e0ed82e81d3ee0caa7de814e0c\": rpc error: code = NotFound desc = could not find container \"0d360cc605ccde706b1a6139edee34d8c211f6e0ed82e81d3ee0caa7de814e0c\": container with ID starting with 0d360cc605ccde706b1a6139edee34d8c211f6e0ed82e81d3ee0caa7de814e0c not found: ID does not exist" Oct 31 00:52:24.655079 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:24.655060 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c57fa36-80bb-403d-97a3-b18601c386aa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c57fa36-80bb-403d-97a3-b18601c386aa-workload-certs\") pod \"4c57fa36-80bb-403d-97a3-b18601c386aa\" (UID: \"4c57fa36-80bb-403d-97a3-b18601c386aa\") " Oct 31 00:52:24.655265 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:24.655250 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4c57fa36-80bb-403d-97a3-b18601c386aa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c57fa36-80bb-403d-97a3-b18601c386aa-istio-podinfo\") pod \"4c57fa36-80bb-403d-97a3-b18601c386aa\" (UID: \"4c57fa36-80bb-403d-97a3-b18601c386aa\") " Oct 31 00:52:24.655304 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:24.655282 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c57fa36-80bb-403d-97a3-b18601c386aa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c57fa36-80bb-403d-97a3-b18601c386aa-workload-socket\") pod \"4c57fa36-80bb-403d-97a3-b18601c386aa\" (UID: \"4c57fa36-80bb-403d-97a3-b18601c386aa\") " Oct 31 00:52:24.655525 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:24.655331 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c57fa36-80bb-403d-97a3-b18601c386aa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c57fa36-80bb-403d-97a3-b18601c386aa-credential-socket\") pod \"4c57fa36-80bb-403d-97a3-b18601c386aa\" (UID: \"4c57fa36-80bb-403d-97a3-b18601c386aa\") " Oct 31 00:52:24.655525 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:24.655358 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c57fa36-80bb-403d-97a3-b18601c386aa-workload-certs" pod "4c57fa36-80bb-403d-97a3-b18601c386aa" (UID: "4c57fa36-80bb-403d-97a3-b18601c386aa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:52:24.655525 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:24.655391 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c57fa36-80bb-403d-97a3-b18601c386aa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c57fa36-80bb-403d-97a3-b18601c386aa-istio-envoy\") pod \"4c57fa36-80bb-403d-97a3-b18601c386aa\" (UID: \"4c57fa36-80bb-403d-97a3-b18601c386aa\") " Oct 31 00:52:24.655525 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:24.655447 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c57fa36-80bb-403d-97a3-b18601c386aa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c57fa36-80bb-403d-97a3-b18601c386aa-istio-data\") pod \"4c57fa36-80bb-403d-97a3-b18601c386aa\" (UID: \"4c57fa36-80bb-403d-97a3-b18601c386aa\") " Oct 31 00:52:24.655525 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:24.655466 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c57fa36-80bb-403d-97a3-b18601c386aa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c57fa36-80bb-403d-97a3-b18601c386aa-istiod-ca-cert\") pod \"4c57fa36-80bb-403d-97a3-b18601c386aa\" (UID: \"4c57fa36-80bb-403d-97a3-b18601c386aa\") " Oct 31 00:52:24.655804 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:24.655525 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c57fa36-80bb-403d-97a3-b18601c386aa-kube-api-access-87hgh\" (UniqueName: \"kubernetes.io/projected/4c57fa36-80bb-403d-97a3-b18601c386aa-kube-api-access-87hgh\") pod \"4c57fa36-80bb-403d-97a3-b18601c386aa\" (UID: \"4c57fa36-80bb-403d-97a3-b18601c386aa\") " Oct 31 00:52:24.655804 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:24.655544 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c57fa36-80bb-403d-97a3-b18601c386aa-istio-token\" (UniqueName: \"kubernetes.io/projected/4c57fa36-80bb-403d-97a3-b18601c386aa-istio-token\") pod \"4c57fa36-80bb-403d-97a3-b18601c386aa\" (UID: \"4c57fa36-80bb-403d-97a3-b18601c386aa\") " Oct 31 00:52:24.655804 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:24.655602 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c57fa36-80bb-403d-97a3-b18601c386aa-workload-socket" pod "4c57fa36-80bb-403d-97a3-b18601c386aa" (UID: "4c57fa36-80bb-403d-97a3-b18601c386aa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:52:24.655804 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:24.655733 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c57fa36-80bb-403d-97a3-b18601c386aa-istio-data" pod "4c57fa36-80bb-403d-97a3-b18601c386aa" (UID: "4c57fa36-80bb-403d-97a3-b18601c386aa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:52:24.656018 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:24.655903 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c57fa36-80bb-403d-97a3-b18601c386aa-credential-socket" pod "4c57fa36-80bb-403d-97a3-b18601c386aa" (UID: "4c57fa36-80bb-403d-97a3-b18601c386aa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:52:24.656105 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:24.656081 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c57fa36-80bb-403d-97a3-b18601c386aa-istiod-ca-cert" pod "4c57fa36-80bb-403d-97a3-b18601c386aa" (UID: "4c57fa36-80bb-403d-97a3-b18601c386aa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:52:24.656602 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:24.656581 5222 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c57fa36-80bb-403d-97a3-b18601c386aa-workload-certs\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:52:24.656701 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:24.656607 5222 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c57fa36-80bb-403d-97a3-b18601c386aa-workload-socket\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:52:24.656701 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:24.656625 5222 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c57fa36-80bb-403d-97a3-b18601c386aa-credential-socket\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:52:24.656701 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:24.656640 5222 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c57fa36-80bb-403d-97a3-b18601c386aa-istio-data\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:52:24.656701 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:24.656655 5222 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c57fa36-80bb-403d-97a3-b18601c386aa-istiod-ca-cert\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:52:24.658195 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:24.658169 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c57fa36-80bb-403d-97a3-b18601c386aa-kube-api-access-87hgh" pod "4c57fa36-80bb-403d-97a3-b18601c386aa" (UID: "4c57fa36-80bb-403d-97a3-b18601c386aa"). InnerVolumeSpecName "kube-api-access-87hgh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:52:24.658283 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:24.658211 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c57fa36-80bb-403d-97a3-b18601c386aa-istio-token" pod "4c57fa36-80bb-403d-97a3-b18601c386aa" (UID: "4c57fa36-80bb-403d-97a3-b18601c386aa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:52:24.658421 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:24.658396 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c57fa36-80bb-403d-97a3-b18601c386aa-istio-envoy" pod "4c57fa36-80bb-403d-97a3-b18601c386aa" (UID: "4c57fa36-80bb-403d-97a3-b18601c386aa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:52:24.658549 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:24.658473 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4c57fa36-80bb-403d-97a3-b18601c386aa-istio-podinfo" pod "4c57fa36-80bb-403d-97a3-b18601c386aa" (UID: "4c57fa36-80bb-403d-97a3-b18601c386aa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:52:24.757734 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:24.757696 5222 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c57fa36-80bb-403d-97a3-b18601c386aa-istio-podinfo\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:52:24.757734 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:24.757735 5222 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c57fa36-80bb-403d-97a3-b18601c386aa-istio-envoy\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:52:24.757972 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:24.757753 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87hgh\" (UniqueName: \"kubernetes.io/projected/4c57fa36-80bb-403d-97a3-b18601c386aa-kube-api-access-87hgh\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:52:24.757972 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:24.757769 5222 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c57fa36-80bb-403d-97a3-b18601c386aa-istio-token\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:52:25.064352 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:25.064314 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-2594a9-openshift-default-8694ddf5f7-v4jsh"] Oct 31 00:52:25.070210 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:25.070181 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-2594a9-openshift-default-8694ddf5f7-v4jsh"] Oct 31 00:52:25.656887 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:25.656855 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c57fa36-80bb-403d-97a3-b18601c386aa" path="/var/lib/kubelet/pods/4c57fa36-80bb-403d-97a3-b18601c386aa/volumes" Oct 31 00:52:29.480724 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:29.480685 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-982ae3-k6nrb"] Oct 31 00:52:29.532192 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:29.532160 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-982ae3-k6nrb"] Oct 31 00:52:29.537863 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:29.537827 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-c5pmx"] Oct 31 00:52:29.538120 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:29.538090 5222 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-c5pmx" podUID="35e93965-7511-4e3b-ae8e-af51574f79f1" containerName="istio-proxy" containerID="cri-o://8b9b985c6a48a3040e67e9a999fa9f63a05981fbf411c49a8ad0d0cfd25d2e2b" gracePeriod=30 Oct 31 00:52:29.658893 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:29.658856 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b5e141c-81c8-497a-aa91-c9e30c14fe30" path="/var/lib/kubelet/pods/9b5e141c-81c8-497a-aa91-c9e30c14fe30/volumes" Oct 31 00:52:30.553350 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:30.553316 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7d5f4de7-td9pl"] Oct 31 00:52:30.553970 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:30.553955 5222 cpu_manager.go:414] "containerMap: removing container" podUID="4c57fa36-80bb-403d-97a3-b18601c386aa" containerName="istio-proxy" Oct 31 00:52:30.554021 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:30.553971 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c57fa36-80bb-403d-97a3-b18601c386aa" containerName="istio-proxy" Oct 31 00:52:30.554021 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:30.553990 5222 cpu_manager.go:414] "containerMap: removing container" podUID="4c57fa36-80bb-403d-97a3-b18601c386aa" containerName="istio-proxy" Oct 31 00:52:30.554021 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:30.553995 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c57fa36-80bb-403d-97a3-b18601c386aa" containerName="istio-proxy" Oct 31 00:52:30.554021 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:30.554010 5222 cpu_manager.go:414] "containerMap: removing container" podUID="7f697756-a33a-4883-a075-a97f84e3d3c4" containerName="revoke-keys" Oct 31 00:52:30.554021 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:30.554015 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f697756-a33a-4883-a075-a97f84e3d3c4" containerName="revoke-keys" Oct 31 00:52:30.554195 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:30.554122 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c57fa36-80bb-403d-97a3-b18601c386aa" containerName="istio-proxy" Oct 31 00:52:30.554195 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:30.554134 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c57fa36-80bb-403d-97a3-b18601c386aa" containerName="istio-proxy" Oct 31 00:52:30.554195 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:30.554142 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f697756-a33a-4883-a075-a97f84e3d3c4" containerName="revoke-keys" Oct 31 00:52:30.557969 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:30.557948 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7d5f4de7-td9pl" Oct 31 00:52:30.613316 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:30.613283 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:52:30.613545 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:30.613418 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:52:30.620985 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:30.620941 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7d5f4de7-td9pl"] Oct 31 00:52:30.660699 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:30.660660 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:52:30.660699 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:30.660667 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfz9z\"" Oct 31 00:52:30.664472 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:30.664444 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26xcj\" (UniqueName: \"kubernetes.io/projected/cff84b10-3c83-47d1-903b-f0a95703ce96-kube-api-access-26xcj\") pod \"maas-api-revoke-keys-e2e-mt-7d5f4de7-td9pl\" (UID: \"cff84b10-3c83-47d1-903b-f0a95703ce96\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7d5f4de7-td9pl" Oct 31 00:52:30.664596 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:30.664577 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cff84b10-3c83-47d1-903b-f0a95703ce96-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-7d5f4de7-td9pl\" (UID: \"cff84b10-3c83-47d1-903b-f0a95703ce96\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7d5f4de7-td9pl" Oct 31 00:52:30.664702 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:30.664685 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cff84b10-3c83-47d1-903b-f0a95703ce96-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-7d5f4de7-td9pl\" (UID: \"cff84b10-3c83-47d1-903b-f0a95703ce96\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7d5f4de7-td9pl" Oct 31 00:52:30.765869 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:30.765829 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cff84b10-3c83-47d1-903b-f0a95703ce96-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-7d5f4de7-td9pl\" (UID: \"cff84b10-3c83-47d1-903b-f0a95703ce96\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7d5f4de7-td9pl" Oct 31 00:52:30.766100 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:30.765890 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cff84b10-3c83-47d1-903b-f0a95703ce96-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-7d5f4de7-td9pl\" (UID: \"cff84b10-3c83-47d1-903b-f0a95703ce96\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7d5f4de7-td9pl" Oct 31 00:52:30.766100 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:30.765994 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26xcj\" (UniqueName: \"kubernetes.io/projected/cff84b10-3c83-47d1-903b-f0a95703ce96-kube-api-access-26xcj\") pod \"maas-api-revoke-keys-e2e-mt-7d5f4de7-td9pl\" (UID: \"cff84b10-3c83-47d1-903b-f0a95703ce96\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7d5f4de7-td9pl" Oct 31 00:52:30.766737 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:30.766710 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cff84b10-3c83-47d1-903b-f0a95703ce96-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-7d5f4de7-td9pl\" (UID: \"cff84b10-3c83-47d1-903b-f0a95703ce96\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7d5f4de7-td9pl" Oct 31 00:52:30.768534 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:30.768510 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cff84b10-3c83-47d1-903b-f0a95703ce96-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-7d5f4de7-td9pl\" (UID: \"cff84b10-3c83-47d1-903b-f0a95703ce96\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7d5f4de7-td9pl" Oct 31 00:52:30.828752 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:30.828657 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26xcj\" (UniqueName: \"kubernetes.io/projected/cff84b10-3c83-47d1-903b-f0a95703ce96-kube-api-access-26xcj\") pod \"maas-api-revoke-keys-e2e-mt-7d5f4de7-td9pl\" (UID: \"cff84b10-3c83-47d1-903b-f0a95703ce96\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7d5f4de7-td9pl" Oct 31 00:52:30.872824 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:30.872769 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7d5f4de7-td9pl" Oct 31 00:52:31.250013 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:52:31.249981 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcff84b10_3c83_47d1_903b_f0a95703ce96.slice/crio-4030f90a917187d6ceac416c9611177545c29f3da4894e517a5d3e7a8a804630 WatchSource:0}: Error finding container 4030f90a917187d6ceac416c9611177545c29f3da4894e517a5d3e7a8a804630: Status 404 returned error can't find the container with id 4030f90a917187d6ceac416c9611177545c29f3da4894e517a5d3e7a8a804630 Oct 31 00:52:31.253185 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:31.253134 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7d5f4de7-td9pl"] Oct 31 00:52:31.687048 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:31.687012 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7d5f4de7-td9pl" event={"ID":"cff84b10-3c83-47d1-903b-f0a95703ce96","Type":"ContainerStarted","Data":"7918809807c83036acf97d6ec3bf7b7452a3bb45b9d8e9d828932c336a63d344"} Oct 31 00:52:31.687602 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:31.687074 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7d5f4de7-td9pl" event={"ID":"cff84b10-3c83-47d1-903b-f0a95703ce96","Type":"ContainerStarted","Data":"4030f90a917187d6ceac416c9611177545c29f3da4894e517a5d3e7a8a804630"} Oct 31 00:52:31.806263 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:31.806204 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7d5f4de7-td9pl" podStartSLOduration=1.806188055 podStartE2EDuration="1.806188055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:52:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:52:31.804518635 +0000 UTC m=+2244.275688386" watchObservedRunningTime="2026-10-31 00:52:31.806188055 +0000 UTC m=+2244.277357804" Oct 31 00:52:32.699187 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:32.699153 5222 generic.go:356] "Generic (PLEG): container finished" podID="cff84b10-3c83-47d1-903b-f0a95703ce96" containerID="7918809807c83036acf97d6ec3bf7b7452a3bb45b9d8e9d828932c336a63d344" exitCode=0 Oct 31 00:52:32.699608 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:32.699216 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7d5f4de7-td9pl" event={"ID":"cff84b10-3c83-47d1-903b-f0a95703ce96","Type":"ContainerDied","Data":"7918809807c83036acf97d6ec3bf7b7452a3bb45b9d8e9d828932c336a63d344"} Oct 31 00:52:33.886421 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:33.886389 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7d5f4de7-td9pl" Oct 31 00:52:34.011836 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:34.011755 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cff84b10-3c83-47d1-903b-f0a95703ce96-kube-api-access-26xcj\" (UniqueName: \"kubernetes.io/projected/cff84b10-3c83-47d1-903b-f0a95703ce96-kube-api-access-26xcj\") pod \"cff84b10-3c83-47d1-903b-f0a95703ce96\" (UID: \"cff84b10-3c83-47d1-903b-f0a95703ce96\") " Oct 31 00:52:34.011836 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:34.011824 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cff84b10-3c83-47d1-903b-f0a95703ce96-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cff84b10-3c83-47d1-903b-f0a95703ce96-maas-api-xks-ca\") pod \"cff84b10-3c83-47d1-903b-f0a95703ce96\" (UID: \"cff84b10-3c83-47d1-903b-f0a95703ce96\") " Oct 31 00:52:34.012072 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:34.011859 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cff84b10-3c83-47d1-903b-f0a95703ce96-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cff84b10-3c83-47d1-903b-f0a95703ce96-maas-api-openshift-ca\") pod \"cff84b10-3c83-47d1-903b-f0a95703ce96\" (UID: \"cff84b10-3c83-47d1-903b-f0a95703ce96\") " Oct 31 00:52:34.012448 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:34.012406 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cff84b10-3c83-47d1-903b-f0a95703ce96-maas-api-openshift-ca" pod "cff84b10-3c83-47d1-903b-f0a95703ce96" (UID: "cff84b10-3c83-47d1-903b-f0a95703ce96"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:52:34.014265 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:34.014231 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cff84b10-3c83-47d1-903b-f0a95703ce96-kube-api-access-26xcj" pod "cff84b10-3c83-47d1-903b-f0a95703ce96" (UID: "cff84b10-3c83-47d1-903b-f0a95703ce96"). InnerVolumeSpecName "kube-api-access-26xcj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:52:34.014409 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:34.014239 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cff84b10-3c83-47d1-903b-f0a95703ce96-maas-api-xks-ca" pod "cff84b10-3c83-47d1-903b-f0a95703ce96" (UID: "cff84b10-3c83-47d1-903b-f0a95703ce96"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:52:34.096646 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:34.096612 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-116b593b-h778h"] Oct 31 00:52:34.097273 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:34.097257 5222 cpu_manager.go:414] "containerMap: removing container" podUID="cff84b10-3c83-47d1-903b-f0a95703ce96" containerName="revoke-keys" Oct 31 00:52:34.097329 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:34.097274 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cff84b10-3c83-47d1-903b-f0a95703ce96" containerName="revoke-keys" Oct 31 00:52:34.097468 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:34.097456 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="cff84b10-3c83-47d1-903b-f0a95703ce96" containerName="revoke-keys" Oct 31 00:52:34.101162 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:34.101145 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-116b593b-h778h" Oct 31 00:52:34.114226 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:34.114198 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26xcj\" (UniqueName: \"kubernetes.io/projected/cff84b10-3c83-47d1-903b-f0a95703ce96-kube-api-access-26xcj\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:52:34.114226 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:34.114227 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cff84b10-3c83-47d1-903b-f0a95703ce96-maas-api-xks-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:52:34.114433 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:34.114236 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cff84b10-3c83-47d1-903b-f0a95703ce96-maas-api-openshift-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:52:34.155439 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:34.155407 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-116b593b-h778h"] Oct 31 00:52:34.159923 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:34.159891 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aaf18069-4hwxp"] Oct 31 00:52:34.164495 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:34.164468 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaf18069-4hwxp" Oct 31 00:52:34.215408 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:34.215353 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aaf18069-4hwxp"] Oct 31 00:52:34.216133 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:34.216095 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dv62s\" (UniqueName: \"kubernetes.io/projected/e4399764-d07a-4838-ad80-57801a53c83c-kube-api-access-dv62s\") pod \"maas-api-revoke-sub-aaf18069-4hwxp\" (UID: \"e4399764-d07a-4838-ad80-57801a53c83c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaf18069-4hwxp" Oct 31 00:52:34.216275 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:34.216200 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0aeab769-a2dc-473e-ba35-441312562df8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-116b593b-h778h\" (UID: \"0aeab769-a2dc-473e-ba35-441312562df8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-116b593b-h778h" Oct 31 00:52:34.216341 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:34.216296 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4399764-d07a-4838-ad80-57801a53c83c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aaf18069-4hwxp\" (UID: \"e4399764-d07a-4838-ad80-57801a53c83c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaf18069-4hwxp" Oct 31 00:52:34.216449 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:34.216337 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0aeab769-a2dc-473e-ba35-441312562df8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-116b593b-h778h\" (UID: \"0aeab769-a2dc-473e-ba35-441312562df8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-116b593b-h778h" Oct 31 00:52:34.216449 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:34.216435 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4399764-d07a-4838-ad80-57801a53c83c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aaf18069-4hwxp\" (UID: \"e4399764-d07a-4838-ad80-57801a53c83c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaf18069-4hwxp" Oct 31 00:52:34.216571 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:34.216481 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvvp6\" (UniqueName: \"kubernetes.io/projected/0aeab769-a2dc-473e-ba35-441312562df8-kube-api-access-zvvp6\") pod \"maas-api-revoke-sub-116b593b-h778h\" (UID: \"0aeab769-a2dc-473e-ba35-441312562df8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-116b593b-h778h" Oct 31 00:52:34.317795 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:34.317759 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dv62s\" (UniqueName: \"kubernetes.io/projected/e4399764-d07a-4838-ad80-57801a53c83c-kube-api-access-dv62s\") pod \"maas-api-revoke-sub-aaf18069-4hwxp\" (UID: \"e4399764-d07a-4838-ad80-57801a53c83c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaf18069-4hwxp" Oct 31 00:52:34.317990 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:34.317826 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0aeab769-a2dc-473e-ba35-441312562df8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-116b593b-h778h\" (UID: \"0aeab769-a2dc-473e-ba35-441312562df8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-116b593b-h778h" Oct 31 00:52:34.317990 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:34.317885 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4399764-d07a-4838-ad80-57801a53c83c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aaf18069-4hwxp\" (UID: \"e4399764-d07a-4838-ad80-57801a53c83c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaf18069-4hwxp" Oct 31 00:52:34.317990 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:34.317907 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0aeab769-a2dc-473e-ba35-441312562df8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-116b593b-h778h\" (UID: \"0aeab769-a2dc-473e-ba35-441312562df8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-116b593b-h778h" Oct 31 00:52:34.317990 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:34.317950 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4399764-d07a-4838-ad80-57801a53c83c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aaf18069-4hwxp\" (UID: \"e4399764-d07a-4838-ad80-57801a53c83c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaf18069-4hwxp" Oct 31 00:52:34.317990 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:34.317978 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvvp6\" (UniqueName: \"kubernetes.io/projected/0aeab769-a2dc-473e-ba35-441312562df8-kube-api-access-zvvp6\") pod \"maas-api-revoke-sub-116b593b-h778h\" (UID: \"0aeab769-a2dc-473e-ba35-441312562df8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-116b593b-h778h" Oct 31 00:52:34.318743 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:34.318667 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0aeab769-a2dc-473e-ba35-441312562df8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-116b593b-h778h\" (UID: \"0aeab769-a2dc-473e-ba35-441312562df8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-116b593b-h778h" Oct 31 00:52:34.318743 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:34.318709 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4399764-d07a-4838-ad80-57801a53c83c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aaf18069-4hwxp\" (UID: \"e4399764-d07a-4838-ad80-57801a53c83c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaf18069-4hwxp" Oct 31 00:52:34.320568 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:34.320545 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4399764-d07a-4838-ad80-57801a53c83c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aaf18069-4hwxp\" (UID: \"e4399764-d07a-4838-ad80-57801a53c83c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaf18069-4hwxp" Oct 31 00:52:34.320688 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:34.320667 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0aeab769-a2dc-473e-ba35-441312562df8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-116b593b-h778h\" (UID: \"0aeab769-a2dc-473e-ba35-441312562df8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-116b593b-h778h" Oct 31 00:52:34.380606 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:34.380563 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dv62s\" (UniqueName: \"kubernetes.io/projected/e4399764-d07a-4838-ad80-57801a53c83c-kube-api-access-dv62s\") pod \"maas-api-revoke-sub-aaf18069-4hwxp\" (UID: \"e4399764-d07a-4838-ad80-57801a53c83c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaf18069-4hwxp" Oct 31 00:52:34.380606 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:34.380568 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvvp6\" (UniqueName: \"kubernetes.io/projected/0aeab769-a2dc-473e-ba35-441312562df8-kube-api-access-zvvp6\") pod \"maas-api-revoke-sub-116b593b-h778h\" (UID: \"0aeab769-a2dc-473e-ba35-441312562df8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-116b593b-h778h" Oct 31 00:52:34.419192 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:34.419153 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-116b593b-h778h" Oct 31 00:52:34.480704 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:34.480437 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaf18069-4hwxp" Oct 31 00:52:34.716808 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:34.716768 5222 generic.go:356] "Generic (PLEG): container finished" podID="35e93965-7511-4e3b-ae8e-af51574f79f1" containerID="8b9b985c6a48a3040e67e9a999fa9f63a05981fbf411c49a8ad0d0cfd25d2e2b" exitCode=0 Oct 31 00:52:34.716999 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:34.716851 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-c5pmx" event={"ID":"35e93965-7511-4e3b-ae8e-af51574f79f1","Type":"ContainerDied","Data":"8b9b985c6a48a3040e67e9a999fa9f63a05981fbf411c49a8ad0d0cfd25d2e2b"} Oct 31 00:52:34.718672 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:34.718650 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7d5f4de7-td9pl" Oct 31 00:52:34.718834 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:34.718685 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7d5f4de7-td9pl" event={"ID":"cff84b10-3c83-47d1-903b-f0a95703ce96","Type":"ContainerDied","Data":"4030f90a917187d6ceac416c9611177545c29f3da4894e517a5d3e7a8a804630"} Oct 31 00:52:34.718834 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:34.718728 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4030f90a917187d6ceac416c9611177545c29f3da4894e517a5d3e7a8a804630" Oct 31 00:52:34.803817 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:34.803788 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-116b593b-h778h"] Oct 31 00:52:34.934562 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:34.934155 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aaf18069-4hwxp"] Oct 31 00:52:34.935990 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:34.935633 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-c5pmx" Oct 31 00:52:34.950668 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:52:34.950641 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4399764_d07a_4838_ad80_57801a53c83c.slice/crio-953e002a44b06c7622a1c9e3dca35b4b705c02282c17de10a789ae6a52a41c0e WatchSource:0}: Error finding container 953e002a44b06c7622a1c9e3dca35b4b705c02282c17de10a789ae6a52a41c0e: Status 404 returned error can't find the container with id 953e002a44b06c7622a1c9e3dca35b4b705c02282c17de10a789ae6a52a41c0e Oct 31 00:52:35.049961 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:35.049922 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35e93965-7511-4e3b-ae8e-af51574f79f1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35e93965-7511-4e3b-ae8e-af51574f79f1-workload-socket\") pod \"35e93965-7511-4e3b-ae8e-af51574f79f1\" (UID: \"35e93965-7511-4e3b-ae8e-af51574f79f1\") " Oct 31 00:52:35.050582 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:35.050560 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35e93965-7511-4e3b-ae8e-af51574f79f1-istio-token\" (UniqueName: \"kubernetes.io/projected/35e93965-7511-4e3b-ae8e-af51574f79f1-istio-token\") pod \"35e93965-7511-4e3b-ae8e-af51574f79f1\" (UID: \"35e93965-7511-4e3b-ae8e-af51574f79f1\") " Oct 31 00:52:35.051245 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:35.051226 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35e93965-7511-4e3b-ae8e-af51574f79f1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35e93965-7511-4e3b-ae8e-af51574f79f1-istio-data\") pod \"35e93965-7511-4e3b-ae8e-af51574f79f1\" (UID: \"35e93965-7511-4e3b-ae8e-af51574f79f1\") " Oct 31 00:52:35.051442 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:35.051425 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35e93965-7511-4e3b-ae8e-af51574f79f1-kube-api-access-mz5tj\" (UniqueName: \"kubernetes.io/projected/35e93965-7511-4e3b-ae8e-af51574f79f1-kube-api-access-mz5tj\") pod \"35e93965-7511-4e3b-ae8e-af51574f79f1\" (UID: \"35e93965-7511-4e3b-ae8e-af51574f79f1\") " Oct 31 00:52:35.051573 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:35.051557 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35e93965-7511-4e3b-ae8e-af51574f79f1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35e93965-7511-4e3b-ae8e-af51574f79f1-workload-certs\") pod \"35e93965-7511-4e3b-ae8e-af51574f79f1\" (UID: \"35e93965-7511-4e3b-ae8e-af51574f79f1\") " Oct 31 00:52:35.051707 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:35.051694 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35e93965-7511-4e3b-ae8e-af51574f79f1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35e93965-7511-4e3b-ae8e-af51574f79f1-istio-envoy\") pod \"35e93965-7511-4e3b-ae8e-af51574f79f1\" (UID: \"35e93965-7511-4e3b-ae8e-af51574f79f1\") " Oct 31 00:52:35.051860 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:35.051844 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/35e93965-7511-4e3b-ae8e-af51574f79f1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35e93965-7511-4e3b-ae8e-af51574f79f1-istio-podinfo\") pod \"35e93965-7511-4e3b-ae8e-af51574f79f1\" (UID: \"35e93965-7511-4e3b-ae8e-af51574f79f1\") " Oct 31 00:52:35.051987 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:35.051974 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35e93965-7511-4e3b-ae8e-af51574f79f1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35e93965-7511-4e3b-ae8e-af51574f79f1-credential-socket\") pod \"35e93965-7511-4e3b-ae8e-af51574f79f1\" (UID: \"35e93965-7511-4e3b-ae8e-af51574f79f1\") " Oct 31 00:52:35.052179 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:35.052165 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35e93965-7511-4e3b-ae8e-af51574f79f1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35e93965-7511-4e3b-ae8e-af51574f79f1-istiod-ca-cert\") pod \"35e93965-7511-4e3b-ae8e-af51574f79f1\" (UID: \"35e93965-7511-4e3b-ae8e-af51574f79f1\") " Oct 31 00:52:35.067558 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:35.050464 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35e93965-7511-4e3b-ae8e-af51574f79f1-workload-socket" pod "35e93965-7511-4e3b-ae8e-af51574f79f1" (UID: "35e93965-7511-4e3b-ae8e-af51574f79f1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:52:35.067558 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:35.053637 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35e93965-7511-4e3b-ae8e-af51574f79f1-workload-certs" pod "35e93965-7511-4e3b-ae8e-af51574f79f1" (UID: "35e93965-7511-4e3b-ae8e-af51574f79f1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:52:35.067558 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:35.053797 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35e93965-7511-4e3b-ae8e-af51574f79f1-istio-data" pod "35e93965-7511-4e3b-ae8e-af51574f79f1" (UID: "35e93965-7511-4e3b-ae8e-af51574f79f1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:52:35.067558 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:35.054863 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35e93965-7511-4e3b-ae8e-af51574f79f1-credential-socket" pod "35e93965-7511-4e3b-ae8e-af51574f79f1" (UID: "35e93965-7511-4e3b-ae8e-af51574f79f1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:52:35.067558 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:35.055266 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35e93965-7511-4e3b-ae8e-af51574f79f1-istiod-ca-cert" pod "35e93965-7511-4e3b-ae8e-af51574f79f1" (UID: "35e93965-7511-4e3b-ae8e-af51574f79f1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:52:35.086613 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:35.083744 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35e93965-7511-4e3b-ae8e-af51574f79f1-istio-token" pod "35e93965-7511-4e3b-ae8e-af51574f79f1" (UID: "35e93965-7511-4e3b-ae8e-af51574f79f1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:52:35.086613 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:35.083919 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35e93965-7511-4e3b-ae8e-af51574f79f1-istio-envoy" pod "35e93965-7511-4e3b-ae8e-af51574f79f1" (UID: "35e93965-7511-4e3b-ae8e-af51574f79f1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:52:35.118444 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:35.105443 5222 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35e93965-7511-4e3b-ae8e-af51574f79f1-istio-envoy\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:52:35.118444 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:35.105476 5222 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35e93965-7511-4e3b-ae8e-af51574f79f1-credential-socket\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:52:35.118444 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:35.105493 5222 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35e93965-7511-4e3b-ae8e-af51574f79f1-istiod-ca-cert\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:52:35.118444 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:35.105504 5222 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35e93965-7511-4e3b-ae8e-af51574f79f1-workload-socket\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:52:35.118444 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:35.105514 5222 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35e93965-7511-4e3b-ae8e-af51574f79f1-istio-token\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:52:35.118444 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:35.105522 5222 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35e93965-7511-4e3b-ae8e-af51574f79f1-istio-data\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:52:35.118444 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:35.105531 5222 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35e93965-7511-4e3b-ae8e-af51574f79f1-workload-certs\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:52:35.120936 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:35.120898 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/35e93965-7511-4e3b-ae8e-af51574f79f1-istio-podinfo" pod "35e93965-7511-4e3b-ae8e-af51574f79f1" (UID: "35e93965-7511-4e3b-ae8e-af51574f79f1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:52:35.132504 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:35.132361 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35e93965-7511-4e3b-ae8e-af51574f79f1-kube-api-access-mz5tj" pod "35e93965-7511-4e3b-ae8e-af51574f79f1" (UID: "35e93965-7511-4e3b-ae8e-af51574f79f1"). InnerVolumeSpecName "kube-api-access-mz5tj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:52:35.206430 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:35.206400 5222 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35e93965-7511-4e3b-ae8e-af51574f79f1-istio-podinfo\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:52:35.206430 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:35.206428 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mz5tj\" (UniqueName: \"kubernetes.io/projected/35e93965-7511-4e3b-ae8e-af51574f79f1-kube-api-access-mz5tj\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:52:35.728322 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:35.728224 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaf18069-4hwxp" event={"ID":"e4399764-d07a-4838-ad80-57801a53c83c","Type":"ContainerStarted","Data":"40bf70fd874c54561704082986718d734650ca8c777c78094d3731aa7d590e50"} Oct 31 00:52:35.728322 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:35.728289 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaf18069-4hwxp" event={"ID":"e4399764-d07a-4838-ad80-57801a53c83c","Type":"ContainerStarted","Data":"953e002a44b06c7622a1c9e3dca35b4b705c02282c17de10a789ae6a52a41c0e"} Oct 31 00:52:35.730095 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:35.730066 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-116b593b-h778h" event={"ID":"0aeab769-a2dc-473e-ba35-441312562df8","Type":"ContainerStarted","Data":"401c36cad00d2004be52c4e07dfd34e09674bdebbf699456249b221926fb0470"} Oct 31 00:52:35.730231 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:35.730107 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-116b593b-h778h" event={"ID":"0aeab769-a2dc-473e-ba35-441312562df8","Type":"ContainerStarted","Data":"71307bc01d00f455cb373bbdae0811db5ab6e67fdb093bd61961d72af7eb7d86"} Oct 31 00:52:35.731543 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:35.731520 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-c5pmx" event={"ID":"35e93965-7511-4e3b-ae8e-af51574f79f1","Type":"ContainerDied","Data":"55e50c731ed697309ccc5dd2d8baeaa60e4d4f0de83e0c3af2d77985f1c452af"} Oct 31 00:52:35.731667 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:35.731542 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-c5pmx" Oct 31 00:52:35.731667 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:35.731571 5222 scope.go:122] "RemoveContainer" containerID="8b9b985c6a48a3040e67e9a999fa9f63a05981fbf411c49a8ad0d0cfd25d2e2b" Oct 31 00:52:35.849265 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:35.849204 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaf18069-4hwxp" podStartSLOduration=1.8491874 podStartE2EDuration="1.8491874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:52:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:52:35.846047088 +0000 UTC m=+2248.317216838" watchObservedRunningTime="2026-10-31 00:52:35.8491874 +0000 UTC m=+2248.320357148" Oct 31 00:52:35.891765 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:52:35.891726 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 00:52:35.967305 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:35.967234 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-116b593b-h778h" podStartSLOduration=1.967200054 podStartE2EDuration="1.967200054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:52:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:52:35.963712077 +0000 UTC m=+2248.434881826" watchObservedRunningTime="2026-10-31 00:52:35.967200054 +0000 UTC m=+2248.438369807" Oct 31 00:52:36.081120 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:36.081086 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-c5pmx"] Oct 31 00:52:36.086548 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:36.086517 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-c5pmx"] Oct 31 00:52:36.741204 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:36.741176 5222 generic.go:356] "Generic (PLEG): container finished" podID="e4399764-d07a-4838-ad80-57801a53c83c" containerID="40bf70fd874c54561704082986718d734650ca8c777c78094d3731aa7d590e50" exitCode=0 Oct 31 00:52:36.741416 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:36.741264 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaf18069-4hwxp" event={"ID":"e4399764-d07a-4838-ad80-57801a53c83c","Type":"ContainerDied","Data":"40bf70fd874c54561704082986718d734650ca8c777c78094d3731aa7d590e50"} Oct 31 00:52:36.743051 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:36.743028 5222 generic.go:356] "Generic (PLEG): container finished" podID="0aeab769-a2dc-473e-ba35-441312562df8" containerID="401c36cad00d2004be52c4e07dfd34e09674bdebbf699456249b221926fb0470" exitCode=0 Oct 31 00:52:36.743138 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:36.743104 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-116b593b-h778h" event={"ID":"0aeab769-a2dc-473e-ba35-441312562df8","Type":"ContainerDied","Data":"401c36cad00d2004be52c4e07dfd34e09674bdebbf699456249b221926fb0470"} Oct 31 00:52:36.923499 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:36.923468 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-gxxl8"] Oct 31 00:52:36.923667 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:36.923652 5222 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-gxxl8" podUID="86d77382-1376-49f5-abb0-165df5b8cd9f" containerName="istio-proxy" containerID="cri-o://a0f60e1e18927ccbd4a68231441f63ac069243facce24bc79fbc72072276d281" gracePeriod=30 Oct 31 00:52:37.657171 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:37.657133 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35e93965-7511-4e3b-ae8e-af51574f79f1" path="/var/lib/kubelet/pods/35e93965-7511-4e3b-ae8e-af51574f79f1/volumes" Oct 31 00:52:37.925648 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:37.925566 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-116b593b-h778h" Oct 31 00:52:37.960572 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:37.960546 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaf18069-4hwxp" Oct 31 00:52:37.990822 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:37.990782 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0aeab769-a2dc-473e-ba35-441312562df8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0aeab769-a2dc-473e-ba35-441312562df8-maas-api-xks-ca\") pod \"0aeab769-a2dc-473e-ba35-441312562df8\" (UID: \"0aeab769-a2dc-473e-ba35-441312562df8\") " Oct 31 00:52:37.991067 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:37.991048 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4399764-d07a-4838-ad80-57801a53c83c-kube-api-access-dv62s\" (UniqueName: \"kubernetes.io/projected/e4399764-d07a-4838-ad80-57801a53c83c-kube-api-access-dv62s\") pod \"e4399764-d07a-4838-ad80-57801a53c83c\" (UID: \"e4399764-d07a-4838-ad80-57801a53c83c\") " Oct 31 00:52:37.991208 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:37.991193 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4399764-d07a-4838-ad80-57801a53c83c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4399764-d07a-4838-ad80-57801a53c83c-maas-api-openshift-ca\") pod \"e4399764-d07a-4838-ad80-57801a53c83c\" (UID: \"e4399764-d07a-4838-ad80-57801a53c83c\") " Oct 31 00:52:37.991276 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:37.991224 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0aeab769-a2dc-473e-ba35-441312562df8-kube-api-access-zvvp6\" (UniqueName: \"kubernetes.io/projected/0aeab769-a2dc-473e-ba35-441312562df8-kube-api-access-zvvp6\") pod \"0aeab769-a2dc-473e-ba35-441312562df8\" (UID: \"0aeab769-a2dc-473e-ba35-441312562df8\") " Oct 31 00:52:37.991343 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:37.991288 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4399764-d07a-4838-ad80-57801a53c83c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4399764-d07a-4838-ad80-57801a53c83c-maas-api-xks-ca\") pod \"e4399764-d07a-4838-ad80-57801a53c83c\" (UID: \"e4399764-d07a-4838-ad80-57801a53c83c\") " Oct 31 00:52:37.991431 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:37.991390 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0aeab769-a2dc-473e-ba35-441312562df8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0aeab769-a2dc-473e-ba35-441312562df8-maas-api-openshift-ca\") pod \"0aeab769-a2dc-473e-ba35-441312562df8\" (UID: \"0aeab769-a2dc-473e-ba35-441312562df8\") " Oct 31 00:52:37.991952 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:37.991916 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4399764-d07a-4838-ad80-57801a53c83c-maas-api-openshift-ca" pod "e4399764-d07a-4838-ad80-57801a53c83c" (UID: "e4399764-d07a-4838-ad80-57801a53c83c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:52:37.992226 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:37.992012 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0aeab769-a2dc-473e-ba35-441312562df8-maas-api-openshift-ca" pod "0aeab769-a2dc-473e-ba35-441312562df8" (UID: "0aeab769-a2dc-473e-ba35-441312562df8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:52:37.992226 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:37.992021 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4399764-d07a-4838-ad80-57801a53c83c-maas-api-openshift-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:52:37.993643 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:37.993613 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4399764-d07a-4838-ad80-57801a53c83c-kube-api-access-dv62s" pod "e4399764-d07a-4838-ad80-57801a53c83c" (UID: "e4399764-d07a-4838-ad80-57801a53c83c"). InnerVolumeSpecName "kube-api-access-dv62s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:52:37.994010 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:37.993981 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4399764-d07a-4838-ad80-57801a53c83c-maas-api-xks-ca" pod "e4399764-d07a-4838-ad80-57801a53c83c" (UID: "e4399764-d07a-4838-ad80-57801a53c83c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:52:37.994120 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:37.994069 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0aeab769-a2dc-473e-ba35-441312562df8-kube-api-access-zvvp6" pod "0aeab769-a2dc-473e-ba35-441312562df8" (UID: "0aeab769-a2dc-473e-ba35-441312562df8"). InnerVolumeSpecName "kube-api-access-zvvp6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:52:37.994298 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:37.994282 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0aeab769-a2dc-473e-ba35-441312562df8-maas-api-xks-ca" pod "0aeab769-a2dc-473e-ba35-441312562df8" (UID: "0aeab769-a2dc-473e-ba35-441312562df8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:52:38.093562 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:38.093527 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0aeab769-a2dc-473e-ba35-441312562df8-maas-api-xks-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:52:38.093562 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:38.093557 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dv62s\" (UniqueName: \"kubernetes.io/projected/e4399764-d07a-4838-ad80-57801a53c83c-kube-api-access-dv62s\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:52:38.093562 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:38.093569 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvvp6\" (UniqueName: \"kubernetes.io/projected/0aeab769-a2dc-473e-ba35-441312562df8-kube-api-access-zvvp6\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:52:38.093797 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:38.093579 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4399764-d07a-4838-ad80-57801a53c83c-maas-api-xks-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:52:38.093797 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:38.093587 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0aeab769-a2dc-473e-ba35-441312562df8-maas-api-openshift-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:52:38.761464 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:38.761432 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-116b593b-h778h" Oct 31 00:52:38.761890 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:38.761437 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-116b593b-h778h" event={"ID":"0aeab769-a2dc-473e-ba35-441312562df8","Type":"ContainerDied","Data":"71307bc01d00f455cb373bbdae0811db5ab6e67fdb093bd61961d72af7eb7d86"} Oct 31 00:52:38.761890 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:38.761563 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="71307bc01d00f455cb373bbdae0811db5ab6e67fdb093bd61961d72af7eb7d86" Oct 31 00:52:38.763061 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:38.763035 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaf18069-4hwxp" Oct 31 00:52:38.763202 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:38.763048 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaf18069-4hwxp" event={"ID":"e4399764-d07a-4838-ad80-57801a53c83c","Type":"ContainerDied","Data":"953e002a44b06c7622a1c9e3dca35b4b705c02282c17de10a789ae6a52a41c0e"} Oct 31 00:52:38.763202 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:38.763101 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="953e002a44b06c7622a1c9e3dca35b4b705c02282c17de10a789ae6a52a41c0e" Oct 31 00:52:41.951400 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:41.951344 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-79tk8"] Oct 31 00:52:41.986683 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:41.986651 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7d5f4de7-cf6c94fdc-zvq97"] Oct 31 00:52:42.089625 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:42.089588 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7d5f4de7-ffd6698f9-2r7sb"] Oct 31 00:52:42.206801 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:42.206736 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-gxxl8" Oct 31 00:52:42.354229 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:42.354197 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86d77382-1376-49f5-abb0-165df5b8cd9f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86d77382-1376-49f5-abb0-165df5b8cd9f-istio-envoy\") pod \"86d77382-1376-49f5-abb0-165df5b8cd9f\" (UID: \"86d77382-1376-49f5-abb0-165df5b8cd9f\") " Oct 31 00:52:42.354414 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:42.354240 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86d77382-1376-49f5-abb0-165df5b8cd9f-istio-token\" (UniqueName: \"kubernetes.io/projected/86d77382-1376-49f5-abb0-165df5b8cd9f-istio-token\") pod \"86d77382-1376-49f5-abb0-165df5b8cd9f\" (UID: \"86d77382-1376-49f5-abb0-165df5b8cd9f\") " Oct 31 00:52:42.354414 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:42.354263 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86d77382-1376-49f5-abb0-165df5b8cd9f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86d77382-1376-49f5-abb0-165df5b8cd9f-workload-certs\") pod \"86d77382-1376-49f5-abb0-165df5b8cd9f\" (UID: \"86d77382-1376-49f5-abb0-165df5b8cd9f\") " Oct 31 00:52:42.354414 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:42.354382 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86d77382-1376-49f5-abb0-165df5b8cd9f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86d77382-1376-49f5-abb0-165df5b8cd9f-workload-socket\") pod \"86d77382-1376-49f5-abb0-165df5b8cd9f\" (UID: \"86d77382-1376-49f5-abb0-165df5b8cd9f\") " Oct 31 00:52:42.354564 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:42.354456 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86d77382-1376-49f5-abb0-165df5b8cd9f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86d77382-1376-49f5-abb0-165df5b8cd9f-credential-socket\") pod \"86d77382-1376-49f5-abb0-165df5b8cd9f\" (UID: \"86d77382-1376-49f5-abb0-165df5b8cd9f\") " Oct 31 00:52:42.354564 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:42.354489 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86d77382-1376-49f5-abb0-165df5b8cd9f-kube-api-access-sgmr8\" (UniqueName: \"kubernetes.io/projected/86d77382-1376-49f5-abb0-165df5b8cd9f-kube-api-access-sgmr8\") pod \"86d77382-1376-49f5-abb0-165df5b8cd9f\" (UID: \"86d77382-1376-49f5-abb0-165df5b8cd9f\") " Oct 31 00:52:42.354673 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:42.354571 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/86d77382-1376-49f5-abb0-165df5b8cd9f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86d77382-1376-49f5-abb0-165df5b8cd9f-istio-podinfo\") pod \"86d77382-1376-49f5-abb0-165df5b8cd9f\" (UID: \"86d77382-1376-49f5-abb0-165df5b8cd9f\") " Oct 31 00:52:42.354673 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:42.354597 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86d77382-1376-49f5-abb0-165df5b8cd9f-workload-certs" pod "86d77382-1376-49f5-abb0-165df5b8cd9f" (UID: "86d77382-1376-49f5-abb0-165df5b8cd9f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:52:42.354792 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:42.354685 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86d77382-1376-49f5-abb0-165df5b8cd9f-workload-socket" pod "86d77382-1376-49f5-abb0-165df5b8cd9f" (UID: "86d77382-1376-49f5-abb0-165df5b8cd9f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:52:42.354792 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:42.354747 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86d77382-1376-49f5-abb0-165df5b8cd9f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86d77382-1376-49f5-abb0-165df5b8cd9f-istio-data\") pod \"86d77382-1376-49f5-abb0-165df5b8cd9f\" (UID: \"86d77382-1376-49f5-abb0-165df5b8cd9f\") " Oct 31 00:52:42.354909 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:42.354863 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86d77382-1376-49f5-abb0-165df5b8cd9f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86d77382-1376-49f5-abb0-165df5b8cd9f-istiod-ca-cert\") pod \"86d77382-1376-49f5-abb0-165df5b8cd9f\" (UID: \"86d77382-1376-49f5-abb0-165df5b8cd9f\") " Oct 31 00:52:42.355057 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:42.355042 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86d77382-1376-49f5-abb0-165df5b8cd9f-credential-socket" pod "86d77382-1376-49f5-abb0-165df5b8cd9f" (UID: "86d77382-1376-49f5-abb0-165df5b8cd9f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:52:42.355329 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:42.355305 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86d77382-1376-49f5-abb0-165df5b8cd9f-istio-data" pod "86d77382-1376-49f5-abb0-165df5b8cd9f" (UID: "86d77382-1376-49f5-abb0-165df5b8cd9f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:52:42.355466 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:42.355415 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86d77382-1376-49f5-abb0-165df5b8cd9f-istiod-ca-cert" pod "86d77382-1376-49f5-abb0-165df5b8cd9f" (UID: "86d77382-1376-49f5-abb0-165df5b8cd9f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:52:42.355770 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:42.355746 5222 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86d77382-1376-49f5-abb0-165df5b8cd9f-credential-socket\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:52:42.355913 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:42.355774 5222 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86d77382-1376-49f5-abb0-165df5b8cd9f-istio-data\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:52:42.355913 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:42.355790 5222 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86d77382-1376-49f5-abb0-165df5b8cd9f-istiod-ca-cert\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:52:42.355913 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:42.355804 5222 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86d77382-1376-49f5-abb0-165df5b8cd9f-workload-certs\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:52:42.355913 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:42.355820 5222 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86d77382-1376-49f5-abb0-165df5b8cd9f-workload-socket\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:52:42.356855 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:42.356832 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86d77382-1376-49f5-abb0-165df5b8cd9f-istio-token" pod "86d77382-1376-49f5-abb0-165df5b8cd9f" (UID: "86d77382-1376-49f5-abb0-165df5b8cd9f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:52:42.356855 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:42.356846 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86d77382-1376-49f5-abb0-165df5b8cd9f-istio-envoy" pod "86d77382-1376-49f5-abb0-165df5b8cd9f" (UID: "86d77382-1376-49f5-abb0-165df5b8cd9f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:52:42.357170 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:42.357156 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/86d77382-1376-49f5-abb0-165df5b8cd9f-istio-podinfo" pod "86d77382-1376-49f5-abb0-165df5b8cd9f" (UID: "86d77382-1376-49f5-abb0-165df5b8cd9f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:52:42.357320 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:42.357301 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86d77382-1376-49f5-abb0-165df5b8cd9f-kube-api-access-sgmr8" pod "86d77382-1376-49f5-abb0-165df5b8cd9f" (UID: "86d77382-1376-49f5-abb0-165df5b8cd9f"). InnerVolumeSpecName "kube-api-access-sgmr8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:52:42.462235 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:42.462138 5222 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86d77382-1376-49f5-abb0-165df5b8cd9f-istio-envoy\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:52:42.462235 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:42.462167 5222 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86d77382-1376-49f5-abb0-165df5b8cd9f-istio-token\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:52:42.462235 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:42.462178 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgmr8\" (UniqueName: \"kubernetes.io/projected/86d77382-1376-49f5-abb0-165df5b8cd9f-kube-api-access-sgmr8\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:52:42.462235 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:42.462187 5222 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86d77382-1376-49f5-abb0-165df5b8cd9f-istio-podinfo\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:52:42.877409 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:42.873432 5222 generic.go:356] "Generic (PLEG): container finished" podID="86d77382-1376-49f5-abb0-165df5b8cd9f" containerID="a0f60e1e18927ccbd4a68231441f63ac069243facce24bc79fbc72072276d281" exitCode=0 Oct 31 00:52:42.877409 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:42.873574 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-gxxl8" event={"ID":"86d77382-1376-49f5-abb0-165df5b8cd9f","Type":"ContainerDied","Data":"a0f60e1e18927ccbd4a68231441f63ac069243facce24bc79fbc72072276d281"} Oct 31 00:52:42.877409 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:42.873629 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-gxxl8" event={"ID":"86d77382-1376-49f5-abb0-165df5b8cd9f","Type":"ContainerDied","Data":"120df8b9096b816a76274294a3f61e5f137354df13ea20b519cf46f4a2ee2d5a"} Oct 31 00:52:42.877409 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:42.873657 5222 scope.go:122] "RemoveContainer" containerID="a0f60e1e18927ccbd4a68231441f63ac069243facce24bc79fbc72072276d281" Oct 31 00:52:42.877409 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:42.873865 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-gxxl8" Oct 31 00:52:42.906340 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:42.906316 5222 scope.go:122] "RemoveContainer" containerID="a0f60e1e18927ccbd4a68231441f63ac069243facce24bc79fbc72072276d281" Oct 31 00:52:42.906769 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:52:42.906749 5222 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0f60e1e18927ccbd4a68231441f63ac069243facce24bc79fbc72072276d281\": container with ID starting with a0f60e1e18927ccbd4a68231441f63ac069243facce24bc79fbc72072276d281 not found: ID does not exist" containerID="a0f60e1e18927ccbd4a68231441f63ac069243facce24bc79fbc72072276d281" Oct 31 00:52:42.906861 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:42.906780 5222 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0f60e1e18927ccbd4a68231441f63ac069243facce24bc79fbc72072276d281"} err="failed to get container status \"a0f60e1e18927ccbd4a68231441f63ac069243facce24bc79fbc72072276d281\": rpc error: code = NotFound desc = could not find container \"a0f60e1e18927ccbd4a68231441f63ac069243facce24bc79fbc72072276d281\": container with ID starting with a0f60e1e18927ccbd4a68231441f63ac069243facce24bc79fbc72072276d281 not found: ID does not exist" Oct 31 00:52:43.108498 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:43.108462 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-gxxl8"] Oct 31 00:52:43.130051 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:43.128837 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-gxxl8"] Oct 31 00:52:43.676881 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:43.676295 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86d77382-1376-49f5-abb0-165df5b8cd9f" path="/var/lib/kubelet/pods/86d77382-1376-49f5-abb0-165df5b8cd9f/volumes" Oct 31 00:52:46.987959 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:46.987860 5222 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-7d5f4de7-cf6c94fdc-zvq97" podUID="cb02bae7-ad38-4a84-b668-2871c1c581e9" containerName="payload-processing" containerID="cri-o://3a3243157e7aef86a98679813352d0987039c4a72e713550f006bccd33dabefc" gracePeriod=25 Oct 31 00:52:47.089996 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:47.089945 5222 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-7d5f4de7-ffd6698f9-2r7sb" podUID="64f10e01-81b0-4bff-a597-19b2a0106639" containerName="payload-pre-processing" containerID="cri-o://79e79d70ad957d72fd45d3ddb364ea2e937e7e0e09a3831dabfb40a822b0dcb0" gracePeriod=25 Oct 31 00:52:47.301051 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:47.301009 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7d5f4de7-cf6c94fdc-zvq97" Oct 31 00:52:47.372347 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:47.372321 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb02bae7-ad38-4a84-b668-2871c1c581e9-kube-api-access-7fcdf\" (UniqueName: \"kubernetes.io/projected/cb02bae7-ad38-4a84-b668-2871c1c581e9-kube-api-access-7fcdf\") pod \"cb02bae7-ad38-4a84-b668-2871c1c581e9\" (UID: \"cb02bae7-ad38-4a84-b668-2871c1c581e9\") " Oct 31 00:52:47.372526 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:47.372476 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb02bae7-ad38-4a84-b668-2871c1c581e9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb02bae7-ad38-4a84-b668-2871c1c581e9-plugins-config-volume\") pod \"cb02bae7-ad38-4a84-b668-2871c1c581e9\" (UID: \"cb02bae7-ad38-4a84-b668-2871c1c581e9\") " Oct 31 00:52:47.372839 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:47.372814 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb02bae7-ad38-4a84-b668-2871c1c581e9-plugins-config-volume" pod "cb02bae7-ad38-4a84-b668-2871c1c581e9" (UID: "cb02bae7-ad38-4a84-b668-2871c1c581e9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:52:47.372986 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:47.372969 5222 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb02bae7-ad38-4a84-b668-2871c1c581e9-plugins-config-volume\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:52:47.374672 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:47.374651 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb02bae7-ad38-4a84-b668-2871c1c581e9-kube-api-access-7fcdf" pod "cb02bae7-ad38-4a84-b668-2871c1c581e9" (UID: "cb02bae7-ad38-4a84-b668-2871c1c581e9"). InnerVolumeSpecName "kube-api-access-7fcdf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:52:47.388001 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:47.387974 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7d5f4de7-ffd6698f9-2r7sb" Oct 31 00:52:47.473481 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:47.473449 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64f10e01-81b0-4bff-a597-19b2a0106639-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64f10e01-81b0-4bff-a597-19b2a0106639-plugins-config-volume\") pod \"64f10e01-81b0-4bff-a597-19b2a0106639\" (UID: \"64f10e01-81b0-4bff-a597-19b2a0106639\") " Oct 31 00:52:47.473693 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:47.473630 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64f10e01-81b0-4bff-a597-19b2a0106639-kube-api-access-425zb\" (UniqueName: \"kubernetes.io/projected/64f10e01-81b0-4bff-a597-19b2a0106639-kube-api-access-425zb\") pod \"64f10e01-81b0-4bff-a597-19b2a0106639\" (UID: \"64f10e01-81b0-4bff-a597-19b2a0106639\") " Oct 31 00:52:47.473885 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:47.473859 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64f10e01-81b0-4bff-a597-19b2a0106639-plugins-config-volume" pod "64f10e01-81b0-4bff-a597-19b2a0106639" (UID: "64f10e01-81b0-4bff-a597-19b2a0106639"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:52:47.474192 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:47.474167 5222 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64f10e01-81b0-4bff-a597-19b2a0106639-plugins-config-volume\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:52:47.474192 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:47.474188 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fcdf\" (UniqueName: \"kubernetes.io/projected/cb02bae7-ad38-4a84-b668-2871c1c581e9-kube-api-access-7fcdf\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:52:47.475912 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:47.475891 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64f10e01-81b0-4bff-a597-19b2a0106639-kube-api-access-425zb" pod "64f10e01-81b0-4bff-a597-19b2a0106639" (UID: "64f10e01-81b0-4bff-a597-19b2a0106639"). InnerVolumeSpecName "kube-api-access-425zb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:52:47.575753 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:47.575709 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-425zb\" (UniqueName: \"kubernetes.io/projected/64f10e01-81b0-4bff-a597-19b2a0106639-kube-api-access-425zb\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:52:47.917270 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:47.917159 5222 generic.go:356] "Generic (PLEG): container finished" podID="64f10e01-81b0-4bff-a597-19b2a0106639" containerID="79e79d70ad957d72fd45d3ddb364ea2e937e7e0e09a3831dabfb40a822b0dcb0" exitCode=0 Oct 31 00:52:47.917270 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:47.917208 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7d5f4de7-ffd6698f9-2r7sb" Oct 31 00:52:47.917270 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:47.917244 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7d5f4de7-ffd6698f9-2r7sb" event={"ID":"64f10e01-81b0-4bff-a597-19b2a0106639","Type":"ContainerDied","Data":"79e79d70ad957d72fd45d3ddb364ea2e937e7e0e09a3831dabfb40a822b0dcb0"} Oct 31 00:52:47.917548 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:47.917296 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7d5f4de7-ffd6698f9-2r7sb" event={"ID":"64f10e01-81b0-4bff-a597-19b2a0106639","Type":"ContainerDied","Data":"5ed0a82869793a182cd000fb65487024a06e9c2be3baa3af0ac4518e31562747"} Oct 31 00:52:47.917548 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:47.917330 5222 scope.go:122] "RemoveContainer" containerID="79e79d70ad957d72fd45d3ddb364ea2e937e7e0e09a3831dabfb40a822b0dcb0" Oct 31 00:52:47.919092 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:47.919031 5222 generic.go:356] "Generic (PLEG): container finished" podID="cb02bae7-ad38-4a84-b668-2871c1c581e9" containerID="3a3243157e7aef86a98679813352d0987039c4a72e713550f006bccd33dabefc" exitCode=0 Oct 31 00:52:47.919250 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:47.919206 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7d5f4de7-cf6c94fdc-zvq97" event={"ID":"cb02bae7-ad38-4a84-b668-2871c1c581e9","Type":"ContainerDied","Data":"3a3243157e7aef86a98679813352d0987039c4a72e713550f006bccd33dabefc"} Oct 31 00:52:47.919334 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:47.919280 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7d5f4de7-cf6c94fdc-zvq97" event={"ID":"cb02bae7-ad38-4a84-b668-2871c1c581e9","Type":"ContainerDied","Data":"8ca82d22b5b4b85076e4a96d7130992656d373dac069b875b66492399212444e"} Oct 31 00:52:47.920703 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:47.920671 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7d5f4de7-cf6c94fdc-zvq97" Oct 31 00:52:47.935628 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:47.935601 5222 scope.go:122] "RemoveContainer" containerID="79e79d70ad957d72fd45d3ddb364ea2e937e7e0e09a3831dabfb40a822b0dcb0" Oct 31 00:52:47.936055 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:52:47.936025 5222 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79e79d70ad957d72fd45d3ddb364ea2e937e7e0e09a3831dabfb40a822b0dcb0\": container with ID starting with 79e79d70ad957d72fd45d3ddb364ea2e937e7e0e09a3831dabfb40a822b0dcb0 not found: ID does not exist" containerID="79e79d70ad957d72fd45d3ddb364ea2e937e7e0e09a3831dabfb40a822b0dcb0" Oct 31 00:52:47.936151 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:47.936068 5222 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79e79d70ad957d72fd45d3ddb364ea2e937e7e0e09a3831dabfb40a822b0dcb0"} err="failed to get container status \"79e79d70ad957d72fd45d3ddb364ea2e937e7e0e09a3831dabfb40a822b0dcb0\": rpc error: code = NotFound desc = could not find container \"79e79d70ad957d72fd45d3ddb364ea2e937e7e0e09a3831dabfb40a822b0dcb0\": container with ID starting with 79e79d70ad957d72fd45d3ddb364ea2e937e7e0e09a3831dabfb40a822b0dcb0 not found: ID does not exist" Oct 31 00:52:47.936151 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:47.936093 5222 scope.go:122] "RemoveContainer" containerID="3a3243157e7aef86a98679813352d0987039c4a72e713550f006bccd33dabefc" Oct 31 00:52:47.950133 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:47.950100 5222 scope.go:122] "RemoveContainer" containerID="3a3243157e7aef86a98679813352d0987039c4a72e713550f006bccd33dabefc" Oct 31 00:52:47.950553 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:52:47.950526 5222 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a3243157e7aef86a98679813352d0987039c4a72e713550f006bccd33dabefc\": container with ID starting with 3a3243157e7aef86a98679813352d0987039c4a72e713550f006bccd33dabefc not found: ID does not exist" containerID="3a3243157e7aef86a98679813352d0987039c4a72e713550f006bccd33dabefc" Oct 31 00:52:47.950654 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:47.950567 5222 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a3243157e7aef86a98679813352d0987039c4a72e713550f006bccd33dabefc"} err="failed to get container status \"3a3243157e7aef86a98679813352d0987039c4a72e713550f006bccd33dabefc\": rpc error: code = NotFound desc = could not find container \"3a3243157e7aef86a98679813352d0987039c4a72e713550f006bccd33dabefc\": container with ID starting with 3a3243157e7aef86a98679813352d0987039c4a72e713550f006bccd33dabefc not found: ID does not exist" Oct 31 00:52:48.089405 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:48.089354 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7d5f4de7-cf6c94fdc-zvq97"] Oct 31 00:52:48.093002 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:48.092981 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7d5f4de7-cf6c94fdc-zvq97"] Oct 31 00:52:48.205351 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:48.205271 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7d5f4de7-ffd6698f9-2r7sb"] Oct 31 00:52:48.210667 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:48.210636 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7d5f4de7-ffd6698f9-2r7sb"] Oct 31 00:52:49.657356 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:49.657322 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64f10e01-81b0-4bff-a597-19b2a0106639" path="/var/lib/kubelet/pods/64f10e01-81b0-4bff-a597-19b2a0106639/volumes" Oct 31 00:52:49.657806 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:49.657794 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb02bae7-ad38-4a84-b668-2871c1c581e9" path="/var/lib/kubelet/pods/cb02bae7-ad38-4a84-b668-2871c1c581e9/volumes" Oct 31 00:52:59.519804 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:59.519766 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l9jbt"] Oct 31 00:52:59.520437 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:59.520419 5222 cpu_manager.go:414] "containerMap: removing container" podUID="cb02bae7-ad38-4a84-b668-2871c1c581e9" containerName="payload-processing" Oct 31 00:52:59.520437 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:59.520438 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb02bae7-ad38-4a84-b668-2871c1c581e9" containerName="payload-processing" Oct 31 00:52:59.520577 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:59.520451 5222 cpu_manager.go:414] "containerMap: removing container" podUID="35e93965-7511-4e3b-ae8e-af51574f79f1" containerName="istio-proxy" Oct 31 00:52:59.520577 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:59.520456 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35e93965-7511-4e3b-ae8e-af51574f79f1" containerName="istio-proxy" Oct 31 00:52:59.520577 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:59.520472 5222 cpu_manager.go:414] "containerMap: removing container" podUID="0aeab769-a2dc-473e-ba35-441312562df8" containerName="revoke-keys" Oct 31 00:52:59.520577 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:59.520477 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0aeab769-a2dc-473e-ba35-441312562df8" containerName="revoke-keys" Oct 31 00:52:59.520577 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:59.520486 5222 cpu_manager.go:414] "containerMap: removing container" podUID="86d77382-1376-49f5-abb0-165df5b8cd9f" containerName="istio-proxy" Oct 31 00:52:59.520577 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:59.520490 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86d77382-1376-49f5-abb0-165df5b8cd9f" containerName="istio-proxy" Oct 31 00:52:59.520577 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:59.520518 5222 cpu_manager.go:414] "containerMap: removing container" podUID="e4399764-d07a-4838-ad80-57801a53c83c" containerName="revoke-keys" Oct 31 00:52:59.520577 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:59.520522 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4399764-d07a-4838-ad80-57801a53c83c" containerName="revoke-keys" Oct 31 00:52:59.520577 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:59.520539 5222 cpu_manager.go:414] "containerMap: removing container" podUID="64f10e01-81b0-4bff-a597-19b2a0106639" containerName="payload-pre-processing" Oct 31 00:52:59.520577 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:59.520543 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64f10e01-81b0-4bff-a597-19b2a0106639" containerName="payload-pre-processing" Oct 31 00:52:59.521044 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:59.520642 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4399764-d07a-4838-ad80-57801a53c83c" containerName="revoke-keys" Oct 31 00:52:59.521044 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:59.520652 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="64f10e01-81b0-4bff-a597-19b2a0106639" containerName="payload-pre-processing" Oct 31 00:52:59.521044 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:59.520658 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb02bae7-ad38-4a84-b668-2871c1c581e9" containerName="payload-processing" Oct 31 00:52:59.521044 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:59.520665 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="35e93965-7511-4e3b-ae8e-af51574f79f1" containerName="istio-proxy" Oct 31 00:52:59.521044 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:59.520672 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="0aeab769-a2dc-473e-ba35-441312562df8" containerName="revoke-keys" Oct 31 00:52:59.521044 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:59.520679 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="86d77382-1376-49f5-abb0-165df5b8cd9f" containerName="istio-proxy" Oct 31 00:52:59.523896 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:59.523877 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l9jbt" Oct 31 00:52:59.570464 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:59.570430 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2rzpl"] Oct 31 00:52:59.574847 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:59.574825 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2rzpl" Oct 31 00:52:59.625692 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:59.625657 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l9jbt"] Oct 31 00:52:59.637204 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:59.637168 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a711fef-fe18-4853-b9c5-c3138bdb9cbf-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-2rzpl\" (UID: \"5a711fef-fe18-4853-b9c5-c3138bdb9cbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2rzpl" Oct 31 00:52:59.637204 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:59.637210 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6ca1829-40e1-442b-b711-1d57b9900f4d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-l9jbt\" (UID: \"a6ca1829-40e1-442b-b711-1d57b9900f4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l9jbt" Oct 31 00:52:59.637506 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:59.637231 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a711fef-fe18-4853-b9c5-c3138bdb9cbf-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-2rzpl\" (UID: \"5a711fef-fe18-4853-b9c5-c3138bdb9cbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2rzpl" Oct 31 00:52:59.637506 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:59.637248 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a711fef-fe18-4853-b9c5-c3138bdb9cbf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-2rzpl\" (UID: \"5a711fef-fe18-4853-b9c5-c3138bdb9cbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2rzpl" Oct 31 00:52:59.637506 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:59.637436 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6ca1829-40e1-442b-b711-1d57b9900f4d-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-l9jbt\" (UID: \"a6ca1829-40e1-442b-b711-1d57b9900f4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l9jbt" Oct 31 00:52:59.637506 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:59.637479 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6ca1829-40e1-442b-b711-1d57b9900f4d-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-l9jbt\" (UID: \"a6ca1829-40e1-442b-b711-1d57b9900f4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l9jbt" Oct 31 00:52:59.637695 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:59.637536 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a711fef-fe18-4853-b9c5-c3138bdb9cbf-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-2rzpl\" (UID: \"5a711fef-fe18-4853-b9c5-c3138bdb9cbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2rzpl" Oct 31 00:52:59.637695 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:59.637597 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a711fef-fe18-4853-b9c5-c3138bdb9cbf-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-2rzpl\" (UID: \"5a711fef-fe18-4853-b9c5-c3138bdb9cbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2rzpl" Oct 31 00:52:59.637695 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:59.637670 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a711fef-fe18-4853-b9c5-c3138bdb9cbf-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-2rzpl\" (UID: \"5a711fef-fe18-4853-b9c5-c3138bdb9cbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2rzpl" Oct 31 00:52:59.637695 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:59.637690 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6ca1829-40e1-442b-b711-1d57b9900f4d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-l9jbt\" (UID: \"a6ca1829-40e1-442b-b711-1d57b9900f4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l9jbt" Oct 31 00:52:59.637843 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:59.637748 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6ca1829-40e1-442b-b711-1d57b9900f4d-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-l9jbt\" (UID: \"a6ca1829-40e1-442b-b711-1d57b9900f4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l9jbt" Oct 31 00:52:59.637843 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:59.637784 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgdbr\" (UniqueName: \"kubernetes.io/projected/a6ca1829-40e1-442b-b711-1d57b9900f4d-kube-api-access-tgdbr\") pod \"maas-default-gateway-openshift-default-6886b96f9-l9jbt\" (UID: \"a6ca1829-40e1-442b-b711-1d57b9900f4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l9jbt" Oct 31 00:52:59.637915 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:59.637869 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6ca1829-40e1-442b-b711-1d57b9900f4d-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-l9jbt\" (UID: \"a6ca1829-40e1-442b-b711-1d57b9900f4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l9jbt" Oct 31 00:52:59.637915 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:59.637905 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hg8q\" (UniqueName: \"kubernetes.io/projected/5a711fef-fe18-4853-b9c5-c3138bdb9cbf-kube-api-access-7hg8q\") pod \"maas-default-gateway-openshift-default-6886b96f9-2rzpl\" (UID: \"5a711fef-fe18-4853-b9c5-c3138bdb9cbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2rzpl" Oct 31 00:52:59.638021 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:59.637975 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6ca1829-40e1-442b-b711-1d57b9900f4d-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-l9jbt\" (UID: \"a6ca1829-40e1-442b-b711-1d57b9900f4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l9jbt" Oct 31 00:52:59.638074 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:59.638059 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a711fef-fe18-4853-b9c5-c3138bdb9cbf-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-2rzpl\" (UID: \"5a711fef-fe18-4853-b9c5-c3138bdb9cbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2rzpl" Oct 31 00:52:59.638130 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:59.638112 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a711fef-fe18-4853-b9c5-c3138bdb9cbf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-2rzpl\" (UID: \"5a711fef-fe18-4853-b9c5-c3138bdb9cbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2rzpl" Oct 31 00:52:59.638182 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:59.638153 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6ca1829-40e1-442b-b711-1d57b9900f4d-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-l9jbt\" (UID: \"a6ca1829-40e1-442b-b711-1d57b9900f4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l9jbt" Oct 31 00:52:59.701971 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:59.701937 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2rzpl"] Oct 31 00:52:59.749028 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:59.746271 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a711fef-fe18-4853-b9c5-c3138bdb9cbf-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-2rzpl\" (UID: \"5a711fef-fe18-4853-b9c5-c3138bdb9cbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2rzpl" Oct 31 00:52:59.749028 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:59.746337 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6ca1829-40e1-442b-b711-1d57b9900f4d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-l9jbt\" (UID: \"a6ca1829-40e1-442b-b711-1d57b9900f4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l9jbt" Oct 31 00:52:59.749028 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:59.746398 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a711fef-fe18-4853-b9c5-c3138bdb9cbf-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-2rzpl\" (UID: \"5a711fef-fe18-4853-b9c5-c3138bdb9cbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2rzpl" Oct 31 00:52:59.749028 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:59.746426 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a711fef-fe18-4853-b9c5-c3138bdb9cbf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-2rzpl\" (UID: \"5a711fef-fe18-4853-b9c5-c3138bdb9cbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2rzpl" Oct 31 00:52:59.749028 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:59.746539 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6ca1829-40e1-442b-b711-1d57b9900f4d-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-l9jbt\" (UID: \"a6ca1829-40e1-442b-b711-1d57b9900f4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l9jbt" Oct 31 00:52:59.749028 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:59.746558 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6ca1829-40e1-442b-b711-1d57b9900f4d-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-l9jbt\" (UID: \"a6ca1829-40e1-442b-b711-1d57b9900f4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l9jbt" Oct 31 00:52:59.749028 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:59.746725 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a711fef-fe18-4853-b9c5-c3138bdb9cbf-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-2rzpl\" (UID: \"5a711fef-fe18-4853-b9c5-c3138bdb9cbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2rzpl" Oct 31 00:52:59.749028 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:59.747025 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a711fef-fe18-4853-b9c5-c3138bdb9cbf-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-2rzpl\" (UID: \"5a711fef-fe18-4853-b9c5-c3138bdb9cbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2rzpl" Oct 31 00:52:59.749028 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:59.747060 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a711fef-fe18-4853-b9c5-c3138bdb9cbf-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-2rzpl\" (UID: \"5a711fef-fe18-4853-b9c5-c3138bdb9cbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2rzpl" Oct 31 00:52:59.749028 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:59.747100 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a711fef-fe18-4853-b9c5-c3138bdb9cbf-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-2rzpl\" (UID: \"5a711fef-fe18-4853-b9c5-c3138bdb9cbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2rzpl" Oct 31 00:52:59.749028 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:59.747125 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6ca1829-40e1-442b-b711-1d57b9900f4d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-l9jbt\" (UID: \"a6ca1829-40e1-442b-b711-1d57b9900f4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l9jbt" Oct 31 00:52:59.749028 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:59.747158 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a711fef-fe18-4853-b9c5-c3138bdb9cbf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-2rzpl\" (UID: \"5a711fef-fe18-4853-b9c5-c3138bdb9cbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2rzpl" Oct 31 00:52:59.749028 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:59.747409 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6ca1829-40e1-442b-b711-1d57b9900f4d-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-l9jbt\" (UID: \"a6ca1829-40e1-442b-b711-1d57b9900f4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l9jbt" Oct 31 00:52:59.749028 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:59.747489 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a711fef-fe18-4853-b9c5-c3138bdb9cbf-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-2rzpl\" (UID: \"5a711fef-fe18-4853-b9c5-c3138bdb9cbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2rzpl" Oct 31 00:52:59.749028 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:59.747163 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6ca1829-40e1-442b-b711-1d57b9900f4d-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-l9jbt\" (UID: \"a6ca1829-40e1-442b-b711-1d57b9900f4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l9jbt" Oct 31 00:52:59.749028 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:59.747569 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a711fef-fe18-4853-b9c5-c3138bdb9cbf-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-2rzpl\" (UID: \"5a711fef-fe18-4853-b9c5-c3138bdb9cbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2rzpl" Oct 31 00:52:59.749028 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:59.747658 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgdbr\" (UniqueName: \"kubernetes.io/projected/a6ca1829-40e1-442b-b711-1d57b9900f4d-kube-api-access-tgdbr\") pod \"maas-default-gateway-openshift-default-6886b96f9-l9jbt\" (UID: \"a6ca1829-40e1-442b-b711-1d57b9900f4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l9jbt" Oct 31 00:52:59.749028 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:59.747759 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a711fef-fe18-4853-b9c5-c3138bdb9cbf-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-2rzpl\" (UID: \"5a711fef-fe18-4853-b9c5-c3138bdb9cbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2rzpl" Oct 31 00:52:59.749028 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:59.747840 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6ca1829-40e1-442b-b711-1d57b9900f4d-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-l9jbt\" (UID: \"a6ca1829-40e1-442b-b711-1d57b9900f4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l9jbt" Oct 31 00:52:59.749028 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:59.747894 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hg8q\" (UniqueName: \"kubernetes.io/projected/5a711fef-fe18-4853-b9c5-c3138bdb9cbf-kube-api-access-7hg8q\") pod \"maas-default-gateway-openshift-default-6886b96f9-2rzpl\" (UID: \"5a711fef-fe18-4853-b9c5-c3138bdb9cbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2rzpl" Oct 31 00:52:59.749028 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:59.747935 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6ca1829-40e1-442b-b711-1d57b9900f4d-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-l9jbt\" (UID: \"a6ca1829-40e1-442b-b711-1d57b9900f4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l9jbt" Oct 31 00:52:59.749028 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:59.747964 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6ca1829-40e1-442b-b711-1d57b9900f4d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-l9jbt\" (UID: \"a6ca1829-40e1-442b-b711-1d57b9900f4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l9jbt" Oct 31 00:52:59.749028 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:59.748022 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a711fef-fe18-4853-b9c5-c3138bdb9cbf-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-2rzpl\" (UID: \"5a711fef-fe18-4853-b9c5-c3138bdb9cbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2rzpl" Oct 31 00:52:59.749028 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:59.748107 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a711fef-fe18-4853-b9c5-c3138bdb9cbf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-2rzpl\" (UID: \"5a711fef-fe18-4853-b9c5-c3138bdb9cbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2rzpl" Oct 31 00:52:59.749028 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:59.748180 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6ca1829-40e1-442b-b711-1d57b9900f4d-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-l9jbt\" (UID: \"a6ca1829-40e1-442b-b711-1d57b9900f4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l9jbt" Oct 31 00:52:59.749028 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:59.748287 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6ca1829-40e1-442b-b711-1d57b9900f4d-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-l9jbt\" (UID: \"a6ca1829-40e1-442b-b711-1d57b9900f4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l9jbt" Oct 31 00:52:59.749028 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:59.748624 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6ca1829-40e1-442b-b711-1d57b9900f4d-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-l9jbt\" (UID: \"a6ca1829-40e1-442b-b711-1d57b9900f4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l9jbt" Oct 31 00:52:59.749028 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:59.748798 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6ca1829-40e1-442b-b711-1d57b9900f4d-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-l9jbt\" (UID: \"a6ca1829-40e1-442b-b711-1d57b9900f4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l9jbt" Oct 31 00:52:59.750262 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:59.749866 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6ca1829-40e1-442b-b711-1d57b9900f4d-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-l9jbt\" (UID: \"a6ca1829-40e1-442b-b711-1d57b9900f4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l9jbt" Oct 31 00:52:59.750338 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:59.750306 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6ca1829-40e1-442b-b711-1d57b9900f4d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-l9jbt\" (UID: \"a6ca1829-40e1-442b-b711-1d57b9900f4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l9jbt" Oct 31 00:52:59.751426 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:59.751383 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a711fef-fe18-4853-b9c5-c3138bdb9cbf-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-2rzpl\" (UID: \"5a711fef-fe18-4853-b9c5-c3138bdb9cbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2rzpl" Oct 31 00:52:59.751644 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:59.751625 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a711fef-fe18-4853-b9c5-c3138bdb9cbf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-2rzpl\" (UID: \"5a711fef-fe18-4853-b9c5-c3138bdb9cbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2rzpl" Oct 31 00:52:59.809202 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:59.809117 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a711fef-fe18-4853-b9c5-c3138bdb9cbf-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-2rzpl\" (UID: \"5a711fef-fe18-4853-b9c5-c3138bdb9cbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2rzpl" Oct 31 00:52:59.809517 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:59.809494 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6ca1829-40e1-442b-b711-1d57b9900f4d-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-l9jbt\" (UID: \"a6ca1829-40e1-442b-b711-1d57b9900f4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l9jbt" Oct 31 00:52:59.809680 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:59.809659 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgdbr\" (UniqueName: \"kubernetes.io/projected/a6ca1829-40e1-442b-b711-1d57b9900f4d-kube-api-access-tgdbr\") pod \"maas-default-gateway-openshift-default-6886b96f9-l9jbt\" (UID: \"a6ca1829-40e1-442b-b711-1d57b9900f4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l9jbt" Oct 31 00:52:59.809747 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:59.809722 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hg8q\" (UniqueName: \"kubernetes.io/projected/5a711fef-fe18-4853-b9c5-c3138bdb9cbf-kube-api-access-7hg8q\") pod \"maas-default-gateway-openshift-default-6886b96f9-2rzpl\" (UID: \"5a711fef-fe18-4853-b9c5-c3138bdb9cbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2rzpl" Oct 31 00:52:59.840086 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:59.840050 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l9jbt" Oct 31 00:52:59.889637 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:52:59.889594 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2rzpl" Oct 31 00:53:00.334575 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:53:00.334541 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6ca1829_40e1_442b_b711_1d57b9900f4d.slice/crio-9aa9327b3bb8fae82ae9ea22034af05b1a5505dc289f8b569edb8987809d7621 WatchSource:0}: Error finding container 9aa9327b3bb8fae82ae9ea22034af05b1a5505dc289f8b569edb8987809d7621: Status 404 returned error can't find the container with id 9aa9327b3bb8fae82ae9ea22034af05b1a5505dc289f8b569edb8987809d7621 Oct 31 00:53:00.336971 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:00.336932 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l9jbt"] Oct 31 00:53:00.337846 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:00.337813 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:53:00.337948 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:00.337878 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:53:00.337948 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:00.337909 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:53:00.432223 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:53:00.432190 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a711fef_fe18_4853_b9c5_c3138bdb9cbf.slice/crio-fb673a6e705ad63f4dad815e5779fbfe8711278aa01c81f46009f0e71d034ef0 WatchSource:0}: Error finding container fb673a6e705ad63f4dad815e5779fbfe8711278aa01c81f46009f0e71d034ef0: Status 404 returned error can't find the container with id fb673a6e705ad63f4dad815e5779fbfe8711278aa01c81f46009f0e71d034ef0 Oct 31 00:53:00.433843 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:00.433814 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2rzpl"] Oct 31 00:53:00.435352 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:00.435312 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:53:00.435514 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:00.435411 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:53:00.435514 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:00.435444 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:53:01.022035 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:01.022000 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l9jbt" event={"ID":"a6ca1829-40e1-442b-b711-1d57b9900f4d","Type":"ContainerStarted","Data":"6c765868dfae014b2d1ae045ff92020b741de13cc5887a032fc66672618370d1"} Oct 31 00:53:01.022592 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:01.022564 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l9jbt" event={"ID":"a6ca1829-40e1-442b-b711-1d57b9900f4d","Type":"ContainerStarted","Data":"9aa9327b3bb8fae82ae9ea22034af05b1a5505dc289f8b569edb8987809d7621"} Oct 31 00:53:01.024246 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:01.024220 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2rzpl" event={"ID":"5a711fef-fe18-4853-b9c5-c3138bdb9cbf","Type":"ContainerStarted","Data":"e950c3bd9cbbbc103d2bd5de16a582b3def0c5988d812ad96e3c9a64b6a234a3"} Oct 31 00:53:01.024409 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:01.024276 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2rzpl" event={"ID":"5a711fef-fe18-4853-b9c5-c3138bdb9cbf","Type":"ContainerStarted","Data":"fb673a6e705ad63f4dad815e5779fbfe8711278aa01c81f46009f0e71d034ef0"} Oct 31 00:53:01.199248 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:01.199172 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l9jbt" podStartSLOduration=2.199149266 podStartE2EDuration="2.199149266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:52:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:53:01.14894779 +0000 UTC m=+2273.620117539" watchObservedRunningTime="2026-10-31 00:53:01.199149266 +0000 UTC m=+2273.670319018" Oct 31 00:53:01.280313 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:01.280183 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2rzpl" podStartSLOduration=2.280140888 podStartE2EDuration="2.280140888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:52:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:53:01.276004632 +0000 UTC m=+2273.747174434" watchObservedRunningTime="2026-10-31 00:53:01.280140888 +0000 UTC m=+2273.751310638" Oct 31 00:53:01.849641 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:01.841839 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l9jbt" Oct 31 00:53:01.853393 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:01.850251 5222 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-l9jbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" start-of-body= Oct 31 00:53:01.853393 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:01.850315 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l9jbt" podUID="a6ca1829-40e1-442b-b711-1d57b9900f4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" Oct 31 00:53:01.891391 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:01.890836 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2rzpl" Oct 31 00:53:01.892492 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:01.892464 5222 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-2rzpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" start-of-body= Oct 31 00:53:01.892667 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:01.892517 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2rzpl" podUID="5a711fef-fe18-4853-b9c5-c3138bdb9cbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" Oct 31 00:53:01.988594 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:01.988490 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5b4db8-c2v24"] Oct 31 00:53:02.023562 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:02.023518 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5b4db8-c2v24"] Oct 31 00:53:02.845398 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:02.841793 5222 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-l9jbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" start-of-body= Oct 31 00:53:02.845398 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:02.841861 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l9jbt" podUID="a6ca1829-40e1-442b-b711-1d57b9900f4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" Oct 31 00:53:02.890842 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:02.890803 5222 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-2rzpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" start-of-body= Oct 31 00:53:02.891042 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:02.890861 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2rzpl" podUID="5a711fef-fe18-4853-b9c5-c3138bdb9cbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" Oct 31 00:53:03.691330 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:03.691294 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7c13393-b6f2-4212-8c35-4564200ab329" path="/var/lib/kubelet/pods/f7c13393-b6f2-4212-8c35-4564200ab329/volumes" Oct 31 00:53:03.865565 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:03.865525 5222 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-l9jbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" start-of-body= Oct 31 00:53:03.865860 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:03.865832 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l9jbt" podUID="a6ca1829-40e1-442b-b711-1d57b9900f4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" Oct 31 00:53:03.890998 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:03.890958 5222 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-2rzpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" start-of-body= Oct 31 00:53:03.891223 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:03.891037 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2rzpl" podUID="5a711fef-fe18-4853-b9c5-c3138bdb9cbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" Oct 31 00:53:04.842397 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:04.841808 5222 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-l9jbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" start-of-body= Oct 31 00:53:04.842397 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:04.841876 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l9jbt" podUID="a6ca1829-40e1-442b-b711-1d57b9900f4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" Oct 31 00:53:04.890961 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:04.890919 5222 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-2rzpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" start-of-body= Oct 31 00:53:04.891151 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:04.890990 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2rzpl" podUID="5a711fef-fe18-4853-b9c5-c3138bdb9cbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" Oct 31 00:53:05.841866 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:05.841823 5222 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-l9jbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" start-of-body= Oct 31 00:53:05.842180 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:05.842142 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l9jbt" podUID="a6ca1829-40e1-442b-b711-1d57b9900f4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" Oct 31 00:53:05.890160 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:05.890119 5222 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-2rzpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" start-of-body= Oct 31 00:53:05.890697 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:05.890190 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2rzpl" podUID="5a711fef-fe18-4853-b9c5-c3138bdb9cbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" Oct 31 00:53:05.912688 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:53:05.912627 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 00:53:06.840859 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:06.840822 5222 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-l9jbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" start-of-body= Oct 31 00:53:06.841045 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:06.840885 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l9jbt" podUID="a6ca1829-40e1-442b-b711-1d57b9900f4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" Oct 31 00:53:06.890201 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:06.890156 5222 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-2rzpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" start-of-body= Oct 31 00:53:06.890769 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:06.890238 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2rzpl" podUID="5a711fef-fe18-4853-b9c5-c3138bdb9cbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" Oct 31 00:53:07.840807 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:07.840763 5222 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-l9jbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" start-of-body= Oct 31 00:53:07.841035 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:07.840834 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l9jbt" podUID="a6ca1829-40e1-442b-b711-1d57b9900f4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" Oct 31 00:53:07.890602 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:07.890547 5222 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-2rzpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" start-of-body= Oct 31 00:53:07.891137 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:07.890634 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2rzpl" podUID="5a711fef-fe18-4853-b9c5-c3138bdb9cbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" Oct 31 00:53:09.225566 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:09.225527 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-tf2dv"] Oct 31 00:53:09.226457 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:09.226419 5222 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-tf2dv" podUID="b191ce67-4983-446c-bd11-394b1f008034" containerName="istio-proxy" containerID="cri-o://6d55730e52ec751adbaa2c9af21dad117a8ec237252148846f89c78a720a0be9" gracePeriod=30 Oct 31 00:53:09.578272 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:09.578227 5222 scope.go:122] "RemoveContainer" containerID="1b53ba0c0585ba3b15fddb818908d77734c1620fbdf074a161c4b104a9916bc9" Oct 31 00:53:09.641037 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:09.640324 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l9jbt" Oct 31 00:53:09.641582 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:09.641390 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l9jbt" Oct 31 00:53:09.643766 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:09.643753 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l9jbt" Oct 31 00:53:09.699666 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:09.699633 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2rzpl" Oct 31 00:53:09.700057 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:09.700037 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2rzpl" Oct 31 00:53:09.700852 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:09.700827 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2rzpl" Oct 31 00:53:10.056763 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:10.056723 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-079967a3-5l6db"] Oct 31 00:53:10.063265 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:10.063229 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-079967a3-5l6db"] Oct 31 00:53:10.095132 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:10.095093 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-50a19d-stls7"] Oct 31 00:53:10.100753 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:10.100727 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-50a19d-stls7" Oct 31 00:53:10.153509 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:10.153469 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-50a19d-stls7"] Oct 31 00:53:10.154995 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:10.154960 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:53:10.154995 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:10.154960 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:53:10.187466 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:10.187427 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c514b249-2a36-467e-9887-98684a616d22-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-50a19d-stls7\" (UID: \"c514b249-2a36-467e-9887-98684a616d22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-50a19d-stls7" Oct 31 00:53:10.187732 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:10.187707 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccqkm\" (UniqueName: \"kubernetes.io/projected/c514b249-2a36-467e-9887-98684a616d22-kube-api-access-ccqkm\") pod \"maas-api-revoke-keys-e2e-api-b-50a19d-stls7\" (UID: \"c514b249-2a36-467e-9887-98684a616d22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-50a19d-stls7" Oct 31 00:53:10.187828 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:10.187756 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c514b249-2a36-467e-9887-98684a616d22-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-50a19d-stls7\" (UID: \"c514b249-2a36-467e-9887-98684a616d22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-50a19d-stls7" Oct 31 00:53:10.214186 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:10.214153 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:53:10.214363 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:10.214322 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfz9z\"" Oct 31 00:53:10.289476 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:10.289431 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccqkm\" (UniqueName: \"kubernetes.io/projected/c514b249-2a36-467e-9887-98684a616d22-kube-api-access-ccqkm\") pod \"maas-api-revoke-keys-e2e-api-b-50a19d-stls7\" (UID: \"c514b249-2a36-467e-9887-98684a616d22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-50a19d-stls7" Oct 31 00:53:10.289961 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:10.289500 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c514b249-2a36-467e-9887-98684a616d22-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-50a19d-stls7\" (UID: \"c514b249-2a36-467e-9887-98684a616d22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-50a19d-stls7" Oct 31 00:53:10.289961 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:10.289651 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c514b249-2a36-467e-9887-98684a616d22-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-50a19d-stls7\" (UID: \"c514b249-2a36-467e-9887-98684a616d22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-50a19d-stls7" Oct 31 00:53:10.290359 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:10.290336 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c514b249-2a36-467e-9887-98684a616d22-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-50a19d-stls7\" (UID: \"c514b249-2a36-467e-9887-98684a616d22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-50a19d-stls7" Oct 31 00:53:10.292169 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:10.292144 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c514b249-2a36-467e-9887-98684a616d22-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-50a19d-stls7\" (UID: \"c514b249-2a36-467e-9887-98684a616d22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-50a19d-stls7" Oct 31 00:53:10.352442 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:10.352319 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccqkm\" (UniqueName: \"kubernetes.io/projected/c514b249-2a36-467e-9887-98684a616d22-kube-api-access-ccqkm\") pod \"maas-api-revoke-keys-e2e-api-b-50a19d-stls7\" (UID: \"c514b249-2a36-467e-9887-98684a616d22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-50a19d-stls7" Oct 31 00:53:10.414507 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:10.414467 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-50a19d-stls7" Oct 31 00:53:10.791434 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:53:10.791357 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc514b249_2a36_467e_9887_98684a616d22.slice/crio-ae00e098927e2a43a441884b8da28ece7165817013d8bdcbc8fbfe352bfc64dc WatchSource:0}: Error finding container ae00e098927e2a43a441884b8da28ece7165817013d8bdcbc8fbfe352bfc64dc: Status 404 returned error can't find the container with id ae00e098927e2a43a441884b8da28ece7165817013d8bdcbc8fbfe352bfc64dc Oct 31 00:53:10.795439 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:10.795415 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-50a19d-stls7"] Oct 31 00:53:11.178412 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:11.178351 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-50a19d-stls7" event={"ID":"c514b249-2a36-467e-9887-98684a616d22","Type":"ContainerStarted","Data":"d1c36e9a8e07acc99fdecc76b40d3b3e758987c5e3aff0c47ec5138158610128"} Oct 31 00:53:11.178610 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:11.178436 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-50a19d-stls7" event={"ID":"c514b249-2a36-467e-9887-98684a616d22","Type":"ContainerStarted","Data":"ae00e098927e2a43a441884b8da28ece7165817013d8bdcbc8fbfe352bfc64dc"} Oct 31 00:53:11.300476 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:11.300336 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-50a19d-stls7" podStartSLOduration=1.300315689 podStartE2EDuration="1.300315689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:53:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:53:11.297583865 +0000 UTC m=+2283.768753614" watchObservedRunningTime="2026-10-31 00:53:11.300315689 +0000 UTC m=+2283.771485439" Oct 31 00:53:11.656974 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:11.656944 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c9a1b97-2895-492c-a5ae-97a6db93bd47" path="/var/lib/kubelet/pods/2c9a1b97-2895-492c-a5ae-97a6db93bd47/volumes" Oct 31 00:53:12.188118 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:12.188081 5222 generic.go:356] "Generic (PLEG): container finished" podID="c514b249-2a36-467e-9887-98684a616d22" containerID="d1c36e9a8e07acc99fdecc76b40d3b3e758987c5e3aff0c47ec5138158610128" exitCode=0 Oct 31 00:53:12.188359 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:12.188170 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-50a19d-stls7" event={"ID":"c514b249-2a36-467e-9887-98684a616d22","Type":"ContainerDied","Data":"d1c36e9a8e07acc99fdecc76b40d3b3e758987c5e3aff0c47ec5138158610128"} Oct 31 00:53:13.378246 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:13.378219 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-50a19d-stls7" Oct 31 00:53:13.464321 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:13.464223 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c514b249-2a36-467e-9887-98684a616d22-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c514b249-2a36-467e-9887-98684a616d22-maas-api-xks-ca\") pod \"c514b249-2a36-467e-9887-98684a616d22\" (UID: \"c514b249-2a36-467e-9887-98684a616d22\") " Oct 31 00:53:13.464577 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:13.464452 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c514b249-2a36-467e-9887-98684a616d22-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c514b249-2a36-467e-9887-98684a616d22-maas-api-openshift-ca\") pod \"c514b249-2a36-467e-9887-98684a616d22\" (UID: \"c514b249-2a36-467e-9887-98684a616d22\") " Oct 31 00:53:13.464577 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:13.464548 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c514b249-2a36-467e-9887-98684a616d22-kube-api-access-ccqkm\" (UniqueName: \"kubernetes.io/projected/c514b249-2a36-467e-9887-98684a616d22-kube-api-access-ccqkm\") pod \"c514b249-2a36-467e-9887-98684a616d22\" (UID: \"c514b249-2a36-467e-9887-98684a616d22\") " Oct 31 00:53:13.464811 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:13.464787 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c514b249-2a36-467e-9887-98684a616d22-maas-api-openshift-ca" pod "c514b249-2a36-467e-9887-98684a616d22" (UID: "c514b249-2a36-467e-9887-98684a616d22"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:53:13.465141 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:13.465121 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c514b249-2a36-467e-9887-98684a616d22-maas-api-openshift-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:53:13.466691 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:13.466667 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c514b249-2a36-467e-9887-98684a616d22-maas-api-xks-ca" pod "c514b249-2a36-467e-9887-98684a616d22" (UID: "c514b249-2a36-467e-9887-98684a616d22"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:53:13.466922 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:13.466899 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c514b249-2a36-467e-9887-98684a616d22-kube-api-access-ccqkm" pod "c514b249-2a36-467e-9887-98684a616d22" (UID: "c514b249-2a36-467e-9887-98684a616d22"). InnerVolumeSpecName "kube-api-access-ccqkm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:53:13.566032 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:13.565997 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccqkm\" (UniqueName: \"kubernetes.io/projected/c514b249-2a36-467e-9887-98684a616d22-kube-api-access-ccqkm\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:53:13.566032 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:13.566029 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c514b249-2a36-467e-9887-98684a616d22-maas-api-xks-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:53:13.706146 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:13.706110 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7d5f4de7-td9pl"] Oct 31 00:53:13.710665 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:13.710637 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7d5f4de7-td9pl"] Oct 31 00:53:13.825456 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:13.825421 5222 patch_prober.go:29] interesting pod/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-tf2dv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:53:13.825630 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:13.825497 5222 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-tf2dv" podUID="b191ce67-4983-446c-bd11-394b1f008034" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:53:14.203766 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:14.203684 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-50a19d-stls7" Oct 31 00:53:14.203766 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:14.203706 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-50a19d-stls7" event={"ID":"c514b249-2a36-467e-9887-98684a616d22","Type":"ContainerDied","Data":"ae00e098927e2a43a441884b8da28ece7165817013d8bdcbc8fbfe352bfc64dc"} Oct 31 00:53:14.203766 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:14.203754 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae00e098927e2a43a441884b8da28ece7165817013d8bdcbc8fbfe352bfc64dc" Oct 31 00:53:14.425289 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:14.425263 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-tf2dv" Oct 31 00:53:14.481301 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:14.481215 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b191ce67-4983-446c-bd11-394b1f008034-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b191ce67-4983-446c-bd11-394b1f008034-istio-data\") pod \"b191ce67-4983-446c-bd11-394b1f008034\" (UID: \"b191ce67-4983-446c-bd11-394b1f008034\") " Oct 31 00:53:14.481301 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:14.481277 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b191ce67-4983-446c-bd11-394b1f008034-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b191ce67-4983-446c-bd11-394b1f008034-workload-certs\") pod \"b191ce67-4983-446c-bd11-394b1f008034\" (UID: \"b191ce67-4983-446c-bd11-394b1f008034\") " Oct 31 00:53:14.481556 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:14.481336 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b191ce67-4983-446c-bd11-394b1f008034-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b191ce67-4983-446c-bd11-394b1f008034-istio-podinfo\") pod \"b191ce67-4983-446c-bd11-394b1f008034\" (UID: \"b191ce67-4983-446c-bd11-394b1f008034\") " Oct 31 00:53:14.481556 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:14.481427 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b191ce67-4983-446c-bd11-394b1f008034-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b191ce67-4983-446c-bd11-394b1f008034-credential-socket\") pod \"b191ce67-4983-446c-bd11-394b1f008034\" (UID: \"b191ce67-4983-446c-bd11-394b1f008034\") " Oct 31 00:53:14.481556 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:14.481480 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b191ce67-4983-446c-bd11-394b1f008034-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b191ce67-4983-446c-bd11-394b1f008034-workload-socket\") pod \"b191ce67-4983-446c-bd11-394b1f008034\" (UID: \"b191ce67-4983-446c-bd11-394b1f008034\") " Oct 31 00:53:14.481732 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:14.481554 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b191ce67-4983-446c-bd11-394b1f008034-istio-data" pod "b191ce67-4983-446c-bd11-394b1f008034" (UID: "b191ce67-4983-446c-bd11-394b1f008034"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:53:14.481732 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:14.481607 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b191ce67-4983-446c-bd11-394b1f008034-kube-api-access-v5b48\" (UniqueName: \"kubernetes.io/projected/b191ce67-4983-446c-bd11-394b1f008034-kube-api-access-v5b48\") pod \"b191ce67-4983-446c-bd11-394b1f008034\" (UID: \"b191ce67-4983-446c-bd11-394b1f008034\") " Oct 31 00:53:14.481732 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:14.481618 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b191ce67-4983-446c-bd11-394b1f008034-workload-certs" pod "b191ce67-4983-446c-bd11-394b1f008034" (UID: "b191ce67-4983-446c-bd11-394b1f008034"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:53:14.481732 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:14.481720 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b191ce67-4983-446c-bd11-394b1f008034-istio-token\" (UniqueName: \"kubernetes.io/projected/b191ce67-4983-446c-bd11-394b1f008034-istio-token\") pod \"b191ce67-4983-446c-bd11-394b1f008034\" (UID: \"b191ce67-4983-446c-bd11-394b1f008034\") " Oct 31 00:53:14.481732 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:14.481727 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b191ce67-4983-446c-bd11-394b1f008034-credential-socket" pod "b191ce67-4983-446c-bd11-394b1f008034" (UID: "b191ce67-4983-446c-bd11-394b1f008034"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:53:14.481970 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:14.481763 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b191ce67-4983-446c-bd11-394b1f008034-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b191ce67-4983-446c-bd11-394b1f008034-istiod-ca-cert\") pod \"b191ce67-4983-446c-bd11-394b1f008034\" (UID: \"b191ce67-4983-446c-bd11-394b1f008034\") " Oct 31 00:53:14.481970 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:14.481798 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b191ce67-4983-446c-bd11-394b1f008034-workload-socket" pod "b191ce67-4983-446c-bd11-394b1f008034" (UID: "b191ce67-4983-446c-bd11-394b1f008034"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:53:14.481970 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:14.481893 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b191ce67-4983-446c-bd11-394b1f008034-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b191ce67-4983-446c-bd11-394b1f008034-istio-envoy\") pod \"b191ce67-4983-446c-bd11-394b1f008034\" (UID: \"b191ce67-4983-446c-bd11-394b1f008034\") " Oct 31 00:53:14.482182 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:14.482153 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b191ce67-4983-446c-bd11-394b1f008034-istiod-ca-cert" pod "b191ce67-4983-446c-bd11-394b1f008034" (UID: "b191ce67-4983-446c-bd11-394b1f008034"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:53:14.482854 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:14.482789 5222 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b191ce67-4983-446c-bd11-394b1f008034-credential-socket\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:53:14.482854 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:14.482817 5222 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b191ce67-4983-446c-bd11-394b1f008034-workload-socket\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:53:14.482854 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:14.482831 5222 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b191ce67-4983-446c-bd11-394b1f008034-istiod-ca-cert\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:53:14.482854 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:14.482846 5222 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b191ce67-4983-446c-bd11-394b1f008034-istio-data\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:53:14.483229 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:14.482863 5222 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b191ce67-4983-446c-bd11-394b1f008034-workload-certs\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:53:14.484015 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:14.483989 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b191ce67-4983-446c-bd11-394b1f008034-istio-token" pod "b191ce67-4983-446c-bd11-394b1f008034" (UID: "b191ce67-4983-446c-bd11-394b1f008034"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:53:14.484119 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:14.484019 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b191ce67-4983-446c-bd11-394b1f008034-istio-podinfo" pod "b191ce67-4983-446c-bd11-394b1f008034" (UID: "b191ce67-4983-446c-bd11-394b1f008034"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:53:14.484119 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:14.484029 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b191ce67-4983-446c-bd11-394b1f008034-kube-api-access-v5b48" pod "b191ce67-4983-446c-bd11-394b1f008034" (UID: "b191ce67-4983-446c-bd11-394b1f008034"). InnerVolumeSpecName "kube-api-access-v5b48". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:53:14.484303 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:14.484288 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b191ce67-4983-446c-bd11-394b1f008034-istio-envoy" pod "b191ce67-4983-446c-bd11-394b1f008034" (UID: "b191ce67-4983-446c-bd11-394b1f008034"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:53:14.584774 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:14.584740 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5b48\" (UniqueName: \"kubernetes.io/projected/b191ce67-4983-446c-bd11-394b1f008034-kube-api-access-v5b48\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:53:14.584774 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:14.584770 5222 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b191ce67-4983-446c-bd11-394b1f008034-istio-token\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:53:14.584774 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:14.584779 5222 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b191ce67-4983-446c-bd11-394b1f008034-istio-envoy\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:53:14.585023 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:14.584788 5222 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b191ce67-4983-446c-bd11-394b1f008034-istio-podinfo\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:53:15.213432 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:15.213399 5222 generic.go:356] "Generic (PLEG): container finished" podID="b191ce67-4983-446c-bd11-394b1f008034" containerID="6d55730e52ec751adbaa2c9af21dad117a8ec237252148846f89c78a720a0be9" exitCode=0 Oct 31 00:53:15.213632 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:15.213523 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-tf2dv" Oct 31 00:53:15.213632 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:15.213541 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-tf2dv" event={"ID":"b191ce67-4983-446c-bd11-394b1f008034","Type":"ContainerDied","Data":"6d55730e52ec751adbaa2c9af21dad117a8ec237252148846f89c78a720a0be9"} Oct 31 00:53:15.213632 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:15.213599 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-tf2dv" event={"ID":"b191ce67-4983-446c-bd11-394b1f008034","Type":"ContainerDied","Data":"a6342241546876a8cbad3e544e655ba3339966e63171367924c9a8c1af401271"} Oct 31 00:53:15.213632 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:15.213626 5222 scope.go:122] "RemoveContainer" containerID="6d55730e52ec751adbaa2c9af21dad117a8ec237252148846f89c78a720a0be9" Oct 31 00:53:15.225427 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:15.225409 5222 scope.go:122] "RemoveContainer" containerID="be0e03299ea7d929c61339c68d60b33a9e7dc6dea816d5a994593d9ae2a5a510" Oct 31 00:53:15.235949 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:15.235926 5222 scope.go:122] "RemoveContainer" containerID="6d55730e52ec751adbaa2c9af21dad117a8ec237252148846f89c78a720a0be9" Oct 31 00:53:15.236281 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:53:15.236251 5222 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d55730e52ec751adbaa2c9af21dad117a8ec237252148846f89c78a720a0be9\": container with ID starting with 6d55730e52ec751adbaa2c9af21dad117a8ec237252148846f89c78a720a0be9 not found: ID does not exist" containerID="6d55730e52ec751adbaa2c9af21dad117a8ec237252148846f89c78a720a0be9" Oct 31 00:53:15.236345 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:15.236290 5222 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d55730e52ec751adbaa2c9af21dad117a8ec237252148846f89c78a720a0be9"} err="failed to get container status \"6d55730e52ec751adbaa2c9af21dad117a8ec237252148846f89c78a720a0be9\": rpc error: code = NotFound desc = could not find container \"6d55730e52ec751adbaa2c9af21dad117a8ec237252148846f89c78a720a0be9\": container with ID starting with 6d55730e52ec751adbaa2c9af21dad117a8ec237252148846f89c78a720a0be9 not found: ID does not exist" Oct 31 00:53:15.236345 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:15.236308 5222 scope.go:122] "RemoveContainer" containerID="be0e03299ea7d929c61339c68d60b33a9e7dc6dea816d5a994593d9ae2a5a510" Oct 31 00:53:15.236615 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:53:15.236597 5222 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be0e03299ea7d929c61339c68d60b33a9e7dc6dea816d5a994593d9ae2a5a510\": container with ID starting with be0e03299ea7d929c61339c68d60b33a9e7dc6dea816d5a994593d9ae2a5a510 not found: ID does not exist" containerID="be0e03299ea7d929c61339c68d60b33a9e7dc6dea816d5a994593d9ae2a5a510" Oct 31 00:53:15.236675 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:15.236620 5222 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be0e03299ea7d929c61339c68d60b33a9e7dc6dea816d5a994593d9ae2a5a510"} err="failed to get container status \"be0e03299ea7d929c61339c68d60b33a9e7dc6dea816d5a994593d9ae2a5a510\": rpc error: code = NotFound desc = could not find container \"be0e03299ea7d929c61339c68d60b33a9e7dc6dea816d5a994593d9ae2a5a510\": container with ID starting with be0e03299ea7d929c61339c68d60b33a9e7dc6dea816d5a994593d9ae2a5a510 not found: ID does not exist" Oct 31 00:53:15.397436 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:15.397397 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-tf2dv"] Oct 31 00:53:15.401096 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:15.401064 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-tf2dv"] Oct 31 00:53:15.657219 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:15.657185 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b191ce67-4983-446c-bd11-394b1f008034" path="/var/lib/kubelet/pods/b191ce67-4983-446c-bd11-394b1f008034/volumes" Oct 31 00:53:15.657746 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:15.657730 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cff84b10-3c83-47d1-903b-f0a95703ce96" path="/var/lib/kubelet/pods/cff84b10-3c83-47d1-903b-f0a95703ce96/volumes" Oct 31 00:53:16.813249 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:16.813210 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-79tk8"] Oct 31 00:53:19.124396 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:19.115500 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-7d5f4de7/e2e-lifecycle-model-7d5f4de7-kserve-799dd698b9-p8scb"] Oct 31 00:53:21.527065 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:21.527035 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-50a19d-867c4c57b5-xknkf"] Oct 31 00:53:21.635818 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:21.635782 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-50a19d-9874b478c-mqhwf"] Oct 31 00:53:26.528094 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:26.528031 5222 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-50a19d-867c4c57b5-xknkf" podUID="29b749ef-a951-4833-bb86-8cb2317f6f15" containerName="payload-processing" containerID="cri-o://7820dec6e9b18ce0033ab5257f398af1893e869bbed7b802c297796059ce8e80" gracePeriod=25 Oct 31 00:53:26.636746 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:26.636682 5222 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-50a19d-9874b478c-mqhwf" podUID="04cf35a1-79f2-4fbd-8a84-7df54fbc0029" containerName="payload-pre-processing" containerID="cri-o://4f55dbcc7d66fa4ad84ed926aa1c1877809d114ae0aec61bff70059b1dc8b805" gracePeriod=25 Oct 31 00:53:26.828709 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:26.828358 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-50a19d-867c4c57b5-xknkf" Oct 31 00:53:26.920992 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:26.920964 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-50a19d-9874b478c-mqhwf" Oct 31 00:53:26.995571 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:26.994961 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29b749ef-a951-4833-bb86-8cb2317f6f15-kube-api-access-npwz9\" (UniqueName: \"kubernetes.io/projected/29b749ef-a951-4833-bb86-8cb2317f6f15-kube-api-access-npwz9\") pod \"29b749ef-a951-4833-bb86-8cb2317f6f15\" (UID: \"29b749ef-a951-4833-bb86-8cb2317f6f15\") " Oct 31 00:53:26.995571 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:26.995168 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29b749ef-a951-4833-bb86-8cb2317f6f15-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29b749ef-a951-4833-bb86-8cb2317f6f15-plugins-config-volume\") pod \"29b749ef-a951-4833-bb86-8cb2317f6f15\" (UID: \"29b749ef-a951-4833-bb86-8cb2317f6f15\") " Oct 31 00:53:26.997842 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:26.997804 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29b749ef-a951-4833-bb86-8cb2317f6f15-plugins-config-volume" pod "29b749ef-a951-4833-bb86-8cb2317f6f15" (UID: "29b749ef-a951-4833-bb86-8cb2317f6f15"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:53:26.999755 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:26.998548 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29b749ef-a951-4833-bb86-8cb2317f6f15-kube-api-access-npwz9" pod "29b749ef-a951-4833-bb86-8cb2317f6f15" (UID: "29b749ef-a951-4833-bb86-8cb2317f6f15"). InnerVolumeSpecName "kube-api-access-npwz9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:53:26.999755 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:26.998607 5222 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29b749ef-a951-4833-bb86-8cb2317f6f15-plugins-config-volume\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:53:27.099994 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:27.099898 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04cf35a1-79f2-4fbd-8a84-7df54fbc0029-kube-api-access-zqjgh\" (UniqueName: \"kubernetes.io/projected/04cf35a1-79f2-4fbd-8a84-7df54fbc0029-kube-api-access-zqjgh\") pod \"04cf35a1-79f2-4fbd-8a84-7df54fbc0029\" (UID: \"04cf35a1-79f2-4fbd-8a84-7df54fbc0029\") " Oct 31 00:53:27.100156 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:27.100052 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04cf35a1-79f2-4fbd-8a84-7df54fbc0029-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04cf35a1-79f2-4fbd-8a84-7df54fbc0029-plugins-config-volume\") pod \"04cf35a1-79f2-4fbd-8a84-7df54fbc0029\" (UID: \"04cf35a1-79f2-4fbd-8a84-7df54fbc0029\") " Oct 31 00:53:27.100579 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:27.100547 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04cf35a1-79f2-4fbd-8a84-7df54fbc0029-plugins-config-volume" pod "04cf35a1-79f2-4fbd-8a84-7df54fbc0029" (UID: "04cf35a1-79f2-4fbd-8a84-7df54fbc0029"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:53:27.100874 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:27.100722 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npwz9\" (UniqueName: \"kubernetes.io/projected/29b749ef-a951-4833-bb86-8cb2317f6f15-kube-api-access-npwz9\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:53:27.100874 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:27.100744 5222 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04cf35a1-79f2-4fbd-8a84-7df54fbc0029-plugins-config-volume\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:53:27.102357 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:27.102323 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04cf35a1-79f2-4fbd-8a84-7df54fbc0029-kube-api-access-zqjgh" pod "04cf35a1-79f2-4fbd-8a84-7df54fbc0029" (UID: "04cf35a1-79f2-4fbd-8a84-7df54fbc0029"). InnerVolumeSpecName "kube-api-access-zqjgh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:53:27.201675 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:27.201641 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqjgh\" (UniqueName: \"kubernetes.io/projected/04cf35a1-79f2-4fbd-8a84-7df54fbc0029-kube-api-access-zqjgh\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:53:27.313729 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:27.313694 5222 generic.go:356] "Generic (PLEG): container finished" podID="04cf35a1-79f2-4fbd-8a84-7df54fbc0029" containerID="4f55dbcc7d66fa4ad84ed926aa1c1877809d114ae0aec61bff70059b1dc8b805" exitCode=0 Oct 31 00:53:27.313924 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:27.313761 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-50a19d-9874b478c-mqhwf" Oct 31 00:53:27.313924 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:27.313774 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-50a19d-9874b478c-mqhwf" event={"ID":"04cf35a1-79f2-4fbd-8a84-7df54fbc0029","Type":"ContainerDied","Data":"4f55dbcc7d66fa4ad84ed926aa1c1877809d114ae0aec61bff70059b1dc8b805"} Oct 31 00:53:27.313924 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:27.313869 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-50a19d-9874b478c-mqhwf" event={"ID":"04cf35a1-79f2-4fbd-8a84-7df54fbc0029","Type":"ContainerDied","Data":"b216aba7b0be2e2583f95e49a8c492916dd90dc9336488a469533a56be5adac3"} Oct 31 00:53:27.313924 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:27.313893 5222 scope.go:122] "RemoveContainer" containerID="4f55dbcc7d66fa4ad84ed926aa1c1877809d114ae0aec61bff70059b1dc8b805" Oct 31 00:53:27.315548 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:27.315514 5222 generic.go:356] "Generic (PLEG): container finished" podID="29b749ef-a951-4833-bb86-8cb2317f6f15" containerID="7820dec6e9b18ce0033ab5257f398af1893e869bbed7b802c297796059ce8e80" exitCode=0 Oct 31 00:53:27.315672 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:27.315580 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-50a19d-867c4c57b5-xknkf" event={"ID":"29b749ef-a951-4833-bb86-8cb2317f6f15","Type":"ContainerDied","Data":"7820dec6e9b18ce0033ab5257f398af1893e869bbed7b802c297796059ce8e80"} Oct 31 00:53:27.315672 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:27.315591 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-50a19d-867c4c57b5-xknkf" Oct 31 00:53:27.315672 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:27.315621 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-50a19d-867c4c57b5-xknkf" event={"ID":"29b749ef-a951-4833-bb86-8cb2317f6f15","Type":"ContainerDied","Data":"085d9b2496825831c0b4e7e71a36e3d8b3d2643c18ecf68a930cfa23c2925833"} Oct 31 00:53:27.325977 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:27.325953 5222 scope.go:122] "RemoveContainer" containerID="4f55dbcc7d66fa4ad84ed926aa1c1877809d114ae0aec61bff70059b1dc8b805" Oct 31 00:53:27.326289 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:53:27.326267 5222 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f55dbcc7d66fa4ad84ed926aa1c1877809d114ae0aec61bff70059b1dc8b805\": container with ID starting with 4f55dbcc7d66fa4ad84ed926aa1c1877809d114ae0aec61bff70059b1dc8b805 not found: ID does not exist" containerID="4f55dbcc7d66fa4ad84ed926aa1c1877809d114ae0aec61bff70059b1dc8b805" Oct 31 00:53:27.326353 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:27.326298 5222 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f55dbcc7d66fa4ad84ed926aa1c1877809d114ae0aec61bff70059b1dc8b805"} err="failed to get container status \"4f55dbcc7d66fa4ad84ed926aa1c1877809d114ae0aec61bff70059b1dc8b805\": rpc error: code = NotFound desc = could not find container \"4f55dbcc7d66fa4ad84ed926aa1c1877809d114ae0aec61bff70059b1dc8b805\": container with ID starting with 4f55dbcc7d66fa4ad84ed926aa1c1877809d114ae0aec61bff70059b1dc8b805 not found: ID does not exist" Oct 31 00:53:27.326353 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:27.326318 5222 scope.go:122] "RemoveContainer" containerID="7820dec6e9b18ce0033ab5257f398af1893e869bbed7b802c297796059ce8e80" Oct 31 00:53:27.336962 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:27.336940 5222 scope.go:122] "RemoveContainer" containerID="7820dec6e9b18ce0033ab5257f398af1893e869bbed7b802c297796059ce8e80" Oct 31 00:53:27.337313 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:53:27.337296 5222 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7820dec6e9b18ce0033ab5257f398af1893e869bbed7b802c297796059ce8e80\": container with ID starting with 7820dec6e9b18ce0033ab5257f398af1893e869bbed7b802c297796059ce8e80 not found: ID does not exist" containerID="7820dec6e9b18ce0033ab5257f398af1893e869bbed7b802c297796059ce8e80" Oct 31 00:53:27.337458 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:27.337324 5222 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7820dec6e9b18ce0033ab5257f398af1893e869bbed7b802c297796059ce8e80"} err="failed to get container status \"7820dec6e9b18ce0033ab5257f398af1893e869bbed7b802c297796059ce8e80\": rpc error: code = NotFound desc = could not find container \"7820dec6e9b18ce0033ab5257f398af1893e869bbed7b802c297796059ce8e80\": container with ID starting with 7820dec6e9b18ce0033ab5257f398af1893e869bbed7b802c297796059ce8e80 not found: ID does not exist" Oct 31 00:53:27.484636 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:27.484545 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-50a19d-867c4c57b5-xknkf"] Oct 31 00:53:27.489049 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:27.489010 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-50a19d-867c4c57b5-xknkf"] Oct 31 00:53:27.648449 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:27.648411 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-50a19d-9874b478c-mqhwf"] Oct 31 00:53:27.660563 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:27.660531 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29b749ef-a951-4833-bb86-8cb2317f6f15" path="/var/lib/kubelet/pods/29b749ef-a951-4833-bb86-8cb2317f6f15/volumes" Oct 31 00:53:27.660911 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:27.660895 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-50a19d-9874b478c-mqhwf"] Oct 31 00:53:29.656683 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:29.656646 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04cf35a1-79f2-4fbd-8a84-7df54fbc0029" path="/var/lib/kubelet/pods/04cf35a1-79f2-4fbd-8a84-7df54fbc0029/volumes" Oct 31 00:53:34.165105 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:34.165058 5222 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-7d5f4de7/e2e-lifecycle-model-7d5f4de7-kserve-799dd698b9-p8scb" podUID="e8849902-5dbf-4045-97e8-94cd53c64a43" containerName="main" containerID="cri-o://7d2209ee2f6f5208d4e3f19d427e32681a81a4719441f8f33705cff08dd5656f" gracePeriod=45 Oct 31 00:53:34.232145 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:34.232099 5222 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-7d5f4de7/e2e-lifecycle-model-7d5f4de7-kserve-799dd698b9-p8scb" podUID="e8849902-5dbf-4045-97e8-94cd53c64a43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.176:8000/ready\": dial tcp 10.134.0.176:8000: connect: connection refused" Oct 31 00:53:34.375432 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:34.375392 5222 generic.go:356] "Generic (PLEG): container finished" podID="e8849902-5dbf-4045-97e8-94cd53c64a43" containerID="7d2209ee2f6f5208d4e3f19d427e32681a81a4719441f8f33705cff08dd5656f" exitCode=0 Oct 31 00:53:34.375432 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:34.375422 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-7d5f4de7/e2e-lifecycle-model-7d5f4de7-kserve-799dd698b9-p8scb" event={"ID":"e8849902-5dbf-4045-97e8-94cd53c64a43","Type":"ContainerDied","Data":"7d2209ee2f6f5208d4e3f19d427e32681a81a4719441f8f33705cff08dd5656f"} Oct 31 00:53:34.446662 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:34.446636 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-7d5f4de7/e2e-lifecycle-model-7d5f4de7-kserve-799dd698b9-p8scb" Oct 31 00:53:34.534060 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:34.534027 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8849902-5dbf-4045-97e8-94cd53c64a43-tls-certs\" (UniqueName: \"kubernetes.io/secret/e8849902-5dbf-4045-97e8-94cd53c64a43-tls-certs\") pod \"e8849902-5dbf-4045-97e8-94cd53c64a43\" (UID: \"e8849902-5dbf-4045-97e8-94cd53c64a43\") " Oct 31 00:53:34.534251 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:34.534147 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8849902-5dbf-4045-97e8-94cd53c64a43-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8849902-5dbf-4045-97e8-94cd53c64a43-dshm\") pod \"e8849902-5dbf-4045-97e8-94cd53c64a43\" (UID: \"e8849902-5dbf-4045-97e8-94cd53c64a43\") " Oct 31 00:53:34.534251 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:34.534177 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8849902-5dbf-4045-97e8-94cd53c64a43-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8849902-5dbf-4045-97e8-94cd53c64a43-tmp-dir\") pod \"e8849902-5dbf-4045-97e8-94cd53c64a43\" (UID: \"e8849902-5dbf-4045-97e8-94cd53c64a43\") " Oct 31 00:53:34.534251 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:34.534222 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8849902-5dbf-4045-97e8-94cd53c64a43-home\" (UniqueName: \"kubernetes.io/empty-dir/e8849902-5dbf-4045-97e8-94cd53c64a43-home\") pod \"e8849902-5dbf-4045-97e8-94cd53c64a43\" (UID: \"e8849902-5dbf-4045-97e8-94cd53c64a43\") " Oct 31 00:53:34.534457 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:34.534278 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8849902-5dbf-4045-97e8-94cd53c64a43-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8849902-5dbf-4045-97e8-94cd53c64a43-model-cache\") pod \"e8849902-5dbf-4045-97e8-94cd53c64a43\" (UID: \"e8849902-5dbf-4045-97e8-94cd53c64a43\") " Oct 31 00:53:34.534523 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:34.534473 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8849902-5dbf-4045-97e8-94cd53c64a43-tmp-dir" pod "e8849902-5dbf-4045-97e8-94cd53c64a43" (UID: "e8849902-5dbf-4045-97e8-94cd53c64a43"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:53:34.534523 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:34.534488 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8849902-5dbf-4045-97e8-94cd53c64a43-home" pod "e8849902-5dbf-4045-97e8-94cd53c64a43" (UID: "e8849902-5dbf-4045-97e8-94cd53c64a43"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:53:34.534639 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:34.534570 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8849902-5dbf-4045-97e8-94cd53c64a43-kube-api-access-5lctf\" (UniqueName: \"kubernetes.io/projected/e8849902-5dbf-4045-97e8-94cd53c64a43-kube-api-access-5lctf\") pod \"e8849902-5dbf-4045-97e8-94cd53c64a43\" (UID: \"e8849902-5dbf-4045-97e8-94cd53c64a43\") " Oct 31 00:53:34.534639 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:34.534607 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8849902-5dbf-4045-97e8-94cd53c64a43-model-cache" pod "e8849902-5dbf-4045-97e8-94cd53c64a43" (UID: "e8849902-5dbf-4045-97e8-94cd53c64a43"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:53:34.535133 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:34.535108 5222 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8849902-5dbf-4045-97e8-94cd53c64a43-tmp-dir\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:53:34.535133 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:34.535127 5222 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8849902-5dbf-4045-97e8-94cd53c64a43-home\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:53:34.535259 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:34.535137 5222 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8849902-5dbf-4045-97e8-94cd53c64a43-model-cache\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:53:34.536754 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:34.536732 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8849902-5dbf-4045-97e8-94cd53c64a43-dshm" pod "e8849902-5dbf-4045-97e8-94cd53c64a43" (UID: "e8849902-5dbf-4045-97e8-94cd53c64a43"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:53:34.536877 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:34.536854 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8849902-5dbf-4045-97e8-94cd53c64a43-tls-certs" pod "e8849902-5dbf-4045-97e8-94cd53c64a43" (UID: "e8849902-5dbf-4045-97e8-94cd53c64a43"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:53:34.537054 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:34.537037 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8849902-5dbf-4045-97e8-94cd53c64a43-kube-api-access-5lctf" pod "e8849902-5dbf-4045-97e8-94cd53c64a43" (UID: "e8849902-5dbf-4045-97e8-94cd53c64a43"). InnerVolumeSpecName "kube-api-access-5lctf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:53:34.636158 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:34.636126 5222 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8849902-5dbf-4045-97e8-94cd53c64a43-dshm\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:53:34.636158 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:34.636155 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lctf\" (UniqueName: \"kubernetes.io/projected/e8849902-5dbf-4045-97e8-94cd53c64a43-kube-api-access-5lctf\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:53:34.636158 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:34.636164 5222 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8849902-5dbf-4045-97e8-94cd53c64a43-tls-certs\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:53:35.383769 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:35.383737 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-7d5f4de7/e2e-lifecycle-model-7d5f4de7-kserve-799dd698b9-p8scb" Oct 31 00:53:35.384271 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:35.383770 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-7d5f4de7/e2e-lifecycle-model-7d5f4de7-kserve-799dd698b9-p8scb" event={"ID":"e8849902-5dbf-4045-97e8-94cd53c64a43","Type":"ContainerDied","Data":"5ed1a83a9c4f6e53338c56b6c60108e2cf8e9c408ebc096c426a2e35a02753b1"} Oct 31 00:53:35.384271 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:35.383827 5222 scope.go:122] "RemoveContainer" containerID="7d2209ee2f6f5208d4e3f19d427e32681a81a4719441f8f33705cff08dd5656f" Oct 31 00:53:35.554951 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:35.554919 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-7d5f4de7/e2e-lifecycle-model-7d5f4de7-kserve-799dd698b9-p8scb"] Oct 31 00:53:35.560474 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:35.560438 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-7d5f4de7/e2e-lifecycle-model-7d5f4de7-kserve-799dd698b9-p8scb"] Oct 31 00:53:35.659903 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:35.659819 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8849902-5dbf-4045-97e8-94cd53c64a43" path="/var/lib/kubelet/pods/e8849902-5dbf-4045-97e8-94cd53c64a43/volumes" Oct 31 00:53:35.891758 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:53:35.891719 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 00:53:41.989716 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:41.989680 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e8ba06-7z5fl"] Oct 31 00:53:42.040669 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:42.040634 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e8ba06-7z5fl"] Oct 31 00:53:43.656699 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:43.656663 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0eb779be-9c69-429a-87b5-2707673ef946" path="/var/lib/kubelet/pods/0eb779be-9c69-429a-87b5-2707673ef946/volumes" Oct 31 00:53:49.185733 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:49.185689 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-cml7r"] Oct 31 00:53:49.186152 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:49.185935 5222 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-cml7r" podUID="f85c9ba2-c597-4786-93a1-930609ff510a" containerName="istio-proxy" containerID="cri-o://1b5d3d06724d455f00d307b92c2c6a4b5554da8c4d8238c849ad38eec38a7fee" gracePeriod=30 Oct 31 00:53:54.152741 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:54.152699 5222 patch_prober.go:29] interesting pod/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-cml7r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:53:54.153146 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:54.152776 5222 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-cml7r" podUID="f85c9ba2-c597-4786-93a1-930609ff510a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:53:54.479440 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:54.479411 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-cml7r" Oct 31 00:53:54.576144 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:54.576109 5222 generic.go:356] "Generic (PLEG): container finished" podID="f85c9ba2-c597-4786-93a1-930609ff510a" containerID="1b5d3d06724d455f00d307b92c2c6a4b5554da8c4d8238c849ad38eec38a7fee" exitCode=0 Oct 31 00:53:54.576363 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:54.576168 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-cml7r" Oct 31 00:53:54.576363 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:54.576196 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-cml7r" event={"ID":"f85c9ba2-c597-4786-93a1-930609ff510a","Type":"ContainerDied","Data":"1b5d3d06724d455f00d307b92c2c6a4b5554da8c4d8238c849ad38eec38a7fee"} Oct 31 00:53:54.576363 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:54.576282 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-cml7r" event={"ID":"f85c9ba2-c597-4786-93a1-930609ff510a","Type":"ContainerDied","Data":"2af46126587414886cedc5d7166cc1452d249bc8e0d2628ef533fefdefa8dccc"} Oct 31 00:53:54.576363 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:54.576309 5222 scope.go:122] "RemoveContainer" containerID="1b5d3d06724d455f00d307b92c2c6a4b5554da8c4d8238c849ad38eec38a7fee" Oct 31 00:53:54.576652 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:54.576610 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f85c9ba2-c597-4786-93a1-930609ff510a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f85c9ba2-c597-4786-93a1-930609ff510a-workload-certs\") pod \"f85c9ba2-c597-4786-93a1-930609ff510a\" (UID: \"f85c9ba2-c597-4786-93a1-930609ff510a\") " Oct 31 00:53:54.576728 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:54.576683 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f85c9ba2-c597-4786-93a1-930609ff510a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f85c9ba2-c597-4786-93a1-930609ff510a-istio-data\") pod \"f85c9ba2-c597-4786-93a1-930609ff510a\" (UID: \"f85c9ba2-c597-4786-93a1-930609ff510a\") " Oct 31 00:53:54.576848 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:54.576800 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f85c9ba2-c597-4786-93a1-930609ff510a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f85c9ba2-c597-4786-93a1-930609ff510a-workload-socket\") pod \"f85c9ba2-c597-4786-93a1-930609ff510a\" (UID: \"f85c9ba2-c597-4786-93a1-930609ff510a\") " Oct 31 00:53:54.576848 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:54.576833 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f85c9ba2-c597-4786-93a1-930609ff510a-istio-token\" (UniqueName: \"kubernetes.io/projected/f85c9ba2-c597-4786-93a1-930609ff510a-istio-token\") pod \"f85c9ba2-c597-4786-93a1-930609ff510a\" (UID: \"f85c9ba2-c597-4786-93a1-930609ff510a\") " Oct 31 00:53:54.576940 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:54.576858 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f85c9ba2-c597-4786-93a1-930609ff510a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f85c9ba2-c597-4786-93a1-930609ff510a-istiod-ca-cert\") pod \"f85c9ba2-c597-4786-93a1-930609ff510a\" (UID: \"f85c9ba2-c597-4786-93a1-930609ff510a\") " Oct 31 00:53:54.576940 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:54.576892 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f85c9ba2-c597-4786-93a1-930609ff510a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f85c9ba2-c597-4786-93a1-930609ff510a-istio-podinfo\") pod \"f85c9ba2-c597-4786-93a1-930609ff510a\" (UID: \"f85c9ba2-c597-4786-93a1-930609ff510a\") " Oct 31 00:53:54.576940 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:54.576899 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f85c9ba2-c597-4786-93a1-930609ff510a-workload-certs" pod "f85c9ba2-c597-4786-93a1-930609ff510a" (UID: "f85c9ba2-c597-4786-93a1-930609ff510a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:53:54.577082 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:54.576945 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f85c9ba2-c597-4786-93a1-930609ff510a-istio-data" pod "f85c9ba2-c597-4786-93a1-930609ff510a" (UID: "f85c9ba2-c597-4786-93a1-930609ff510a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:53:54.577082 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:54.576987 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f85c9ba2-c597-4786-93a1-930609ff510a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f85c9ba2-c597-4786-93a1-930609ff510a-credential-socket\") pod \"f85c9ba2-c597-4786-93a1-930609ff510a\" (UID: \"f85c9ba2-c597-4786-93a1-930609ff510a\") " Oct 31 00:53:54.577190 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:54.577157 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f85c9ba2-c597-4786-93a1-930609ff510a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f85c9ba2-c597-4786-93a1-930609ff510a-istio-envoy\") pod \"f85c9ba2-c597-4786-93a1-930609ff510a\" (UID: \"f85c9ba2-c597-4786-93a1-930609ff510a\") " Oct 31 00:53:54.577254 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:54.577208 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f85c9ba2-c597-4786-93a1-930609ff510a-kube-api-access-p72mr\" (UniqueName: \"kubernetes.io/projected/f85c9ba2-c597-4786-93a1-930609ff510a-kube-api-access-p72mr\") pod \"f85c9ba2-c597-4786-93a1-930609ff510a\" (UID: \"f85c9ba2-c597-4786-93a1-930609ff510a\") " Oct 31 00:53:54.577294 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:54.577274 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f85c9ba2-c597-4786-93a1-930609ff510a-credential-socket" pod "f85c9ba2-c597-4786-93a1-930609ff510a" (UID: "f85c9ba2-c597-4786-93a1-930609ff510a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:53:54.577345 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:54.577289 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f85c9ba2-c597-4786-93a1-930609ff510a-workload-socket" pod "f85c9ba2-c597-4786-93a1-930609ff510a" (UID: "f85c9ba2-c597-4786-93a1-930609ff510a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:53:54.577584 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:54.577559 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f85c9ba2-c597-4786-93a1-930609ff510a-istiod-ca-cert" pod "f85c9ba2-c597-4786-93a1-930609ff510a" (UID: "f85c9ba2-c597-4786-93a1-930609ff510a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:53:54.578172 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:54.577884 5222 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f85c9ba2-c597-4786-93a1-930609ff510a-workload-certs\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:53:54.578172 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:54.578015 5222 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f85c9ba2-c597-4786-93a1-930609ff510a-istio-data\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:53:54.578172 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:54.578030 5222 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f85c9ba2-c597-4786-93a1-930609ff510a-workload-socket\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:53:54.578172 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:54.578045 5222 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f85c9ba2-c597-4786-93a1-930609ff510a-credential-socket\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:53:54.579580 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:54.579543 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f85c9ba2-c597-4786-93a1-930609ff510a-istio-podinfo" pod "f85c9ba2-c597-4786-93a1-930609ff510a" (UID: "f85c9ba2-c597-4786-93a1-930609ff510a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:53:54.579755 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:54.579732 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f85c9ba2-c597-4786-93a1-930609ff510a-istio-token" pod "f85c9ba2-c597-4786-93a1-930609ff510a" (UID: "f85c9ba2-c597-4786-93a1-930609ff510a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:53:54.579866 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:54.579786 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f85c9ba2-c597-4786-93a1-930609ff510a-kube-api-access-p72mr" pod "f85c9ba2-c597-4786-93a1-930609ff510a" (UID: "f85c9ba2-c597-4786-93a1-930609ff510a"). InnerVolumeSpecName "kube-api-access-p72mr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:53:54.579941 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:54.579924 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f85c9ba2-c597-4786-93a1-930609ff510a-istio-envoy" pod "f85c9ba2-c597-4786-93a1-930609ff510a" (UID: "f85c9ba2-c597-4786-93a1-930609ff510a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:53:54.607979 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:54.607958 5222 scope.go:122] "RemoveContainer" containerID="34125c454b0f15ed31fe01b3cb49afe5730ab4bc1d4edef5b8f7784a20daa122" Oct 31 00:53:54.618573 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:54.618549 5222 scope.go:122] "RemoveContainer" containerID="1b5d3d06724d455f00d307b92c2c6a4b5554da8c4d8238c849ad38eec38a7fee" Oct 31 00:53:54.618919 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:53:54.618897 5222 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b5d3d06724d455f00d307b92c2c6a4b5554da8c4d8238c849ad38eec38a7fee\": container with ID starting with 1b5d3d06724d455f00d307b92c2c6a4b5554da8c4d8238c849ad38eec38a7fee not found: ID does not exist" containerID="1b5d3d06724d455f00d307b92c2c6a4b5554da8c4d8238c849ad38eec38a7fee" Oct 31 00:53:54.619000 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:54.618930 5222 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b5d3d06724d455f00d307b92c2c6a4b5554da8c4d8238c849ad38eec38a7fee"} err="failed to get container status \"1b5d3d06724d455f00d307b92c2c6a4b5554da8c4d8238c849ad38eec38a7fee\": rpc error: code = NotFound desc = could not find container \"1b5d3d06724d455f00d307b92c2c6a4b5554da8c4d8238c849ad38eec38a7fee\": container with ID starting with 1b5d3d06724d455f00d307b92c2c6a4b5554da8c4d8238c849ad38eec38a7fee not found: ID does not exist" Oct 31 00:53:54.619000 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:54.618950 5222 scope.go:122] "RemoveContainer" containerID="34125c454b0f15ed31fe01b3cb49afe5730ab4bc1d4edef5b8f7784a20daa122" Oct 31 00:53:54.619204 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:53:54.619190 5222 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34125c454b0f15ed31fe01b3cb49afe5730ab4bc1d4edef5b8f7784a20daa122\": container with ID starting with 34125c454b0f15ed31fe01b3cb49afe5730ab4bc1d4edef5b8f7784a20daa122 not found: ID does not exist" containerID="34125c454b0f15ed31fe01b3cb49afe5730ab4bc1d4edef5b8f7784a20daa122" Oct 31 00:53:54.619246 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:54.619210 5222 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34125c454b0f15ed31fe01b3cb49afe5730ab4bc1d4edef5b8f7784a20daa122"} err="failed to get container status \"34125c454b0f15ed31fe01b3cb49afe5730ab4bc1d4edef5b8f7784a20daa122\": rpc error: code = NotFound desc = could not find container \"34125c454b0f15ed31fe01b3cb49afe5730ab4bc1d4edef5b8f7784a20daa122\": container with ID starting with 34125c454b0f15ed31fe01b3cb49afe5730ab4bc1d4edef5b8f7784a20daa122 not found: ID does not exist" Oct 31 00:53:54.681247 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:54.681213 5222 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f85c9ba2-c597-4786-93a1-930609ff510a-istio-envoy\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:53:54.681247 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:54.681246 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p72mr\" (UniqueName: \"kubernetes.io/projected/f85c9ba2-c597-4786-93a1-930609ff510a-kube-api-access-p72mr\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:53:54.681484 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:54.681258 5222 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f85c9ba2-c597-4786-93a1-930609ff510a-istio-token\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:53:54.681484 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:54.681267 5222 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f85c9ba2-c597-4786-93a1-930609ff510a-istiod-ca-cert\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:53:54.681484 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:54.681275 5222 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f85c9ba2-c597-4786-93a1-930609ff510a-istio-podinfo\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:53:55.002889 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:55.002856 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-cml7r"] Oct 31 00:53:55.008229 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:55.008194 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-cml7r"] Oct 31 00:53:55.656800 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:55.656769 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f85c9ba2-c597-4786-93a1-930609ff510a" path="/var/lib/kubelet/pods/f85c9ba2-c597-4786-93a1-930609ff510a/volumes" Oct 31 00:53:57.913500 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:57.913468 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bef057b-gv7fm"] Oct 31 00:53:57.914068 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:57.914053 5222 cpu_manager.go:414] "containerMap: removing container" podUID="29b749ef-a951-4833-bb86-8cb2317f6f15" containerName="payload-processing" Oct 31 00:53:57.914124 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:57.914069 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29b749ef-a951-4833-bb86-8cb2317f6f15" containerName="payload-processing" Oct 31 00:53:57.914124 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:57.914087 5222 cpu_manager.go:414] "containerMap: removing container" podUID="c514b249-2a36-467e-9887-98684a616d22" containerName="revoke-keys" Oct 31 00:53:57.914124 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:57.914095 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c514b249-2a36-467e-9887-98684a616d22" containerName="revoke-keys" Oct 31 00:53:57.914124 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:57.914104 5222 cpu_manager.go:414] "containerMap: removing container" podUID="e8849902-5dbf-4045-97e8-94cd53c64a43" containerName="main" Oct 31 00:53:57.914124 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:57.914109 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8849902-5dbf-4045-97e8-94cd53c64a43" containerName="main" Oct 31 00:53:57.914124 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:57.914116 5222 cpu_manager.go:414] "containerMap: removing container" podUID="04cf35a1-79f2-4fbd-8a84-7df54fbc0029" containerName="payload-pre-processing" Oct 31 00:53:57.914124 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:57.914121 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04cf35a1-79f2-4fbd-8a84-7df54fbc0029" containerName="payload-pre-processing" Oct 31 00:53:57.914354 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:57.914135 5222 cpu_manager.go:414] "containerMap: removing container" podUID="f85c9ba2-c597-4786-93a1-930609ff510a" containerName="istio-proxy" Oct 31 00:53:57.914354 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:57.914140 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f85c9ba2-c597-4786-93a1-930609ff510a" containerName="istio-proxy" Oct 31 00:53:57.914354 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:57.914156 5222 cpu_manager.go:414] "containerMap: removing container" podUID="b191ce67-4983-446c-bd11-394b1f008034" containerName="istio-proxy" Oct 31 00:53:57.914354 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:57.914161 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b191ce67-4983-446c-bd11-394b1f008034" containerName="istio-proxy" Oct 31 00:53:57.914354 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:57.914174 5222 cpu_manager.go:414] "containerMap: removing container" podUID="b191ce67-4983-446c-bd11-394b1f008034" containerName="istio-proxy" Oct 31 00:53:57.914354 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:57.914179 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b191ce67-4983-446c-bd11-394b1f008034" containerName="istio-proxy" Oct 31 00:53:57.914354 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:57.914281 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="f85c9ba2-c597-4786-93a1-930609ff510a" containerName="istio-proxy" Oct 31 00:53:57.914354 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:57.914291 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="29b749ef-a951-4833-bb86-8cb2317f6f15" containerName="payload-processing" Oct 31 00:53:57.914354 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:57.914302 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="c514b249-2a36-467e-9887-98684a616d22" containerName="revoke-keys" Oct 31 00:53:57.914354 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:57.914312 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8849902-5dbf-4045-97e8-94cd53c64a43" containerName="main" Oct 31 00:53:57.914354 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:57.914318 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="04cf35a1-79f2-4fbd-8a84-7df54fbc0029" containerName="payload-pre-processing" Oct 31 00:53:57.914354 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:57.914324 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="f85c9ba2-c597-4786-93a1-930609ff510a" containerName="istio-proxy" Oct 31 00:53:57.914354 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:57.914332 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="b191ce67-4983-446c-bd11-394b1f008034" containerName="istio-proxy" Oct 31 00:53:57.914354 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:57.914339 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="b191ce67-4983-446c-bd11-394b1f008034" containerName="istio-proxy" Oct 31 00:53:57.919201 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:57.919180 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bef057b-gv7fm" Oct 31 00:53:57.973299 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:57.973253 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bef057b-gv7fm"] Oct 31 00:53:58.019015 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:58.018980 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:53:58.019015 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:58.018981 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:53:58.019015 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:58.019010 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:53:58.023926 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:58.023895 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfz9z\"" Oct 31 00:53:58.036860 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:58.036826 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftdbh\" (UniqueName: \"kubernetes.io/projected/602dd633-f02d-4891-a476-cc693443d0a9-kube-api-access-ftdbh\") pod \"maas-api-revoke-sub-3bef057b-gv7fm\" (UID: \"602dd633-f02d-4891-a476-cc693443d0a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bef057b-gv7fm" Oct 31 00:53:58.037027 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:58.036869 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/602dd633-f02d-4891-a476-cc693443d0a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3bef057b-gv7fm\" (UID: \"602dd633-f02d-4891-a476-cc693443d0a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bef057b-gv7fm" Oct 31 00:53:58.037106 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:58.037087 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/602dd633-f02d-4891-a476-cc693443d0a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3bef057b-gv7fm\" (UID: \"602dd633-f02d-4891-a476-cc693443d0a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bef057b-gv7fm" Oct 31 00:53:58.138098 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:58.138055 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/602dd633-f02d-4891-a476-cc693443d0a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3bef057b-gv7fm\" (UID: \"602dd633-f02d-4891-a476-cc693443d0a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bef057b-gv7fm" Oct 31 00:53:58.138313 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:58.138169 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftdbh\" (UniqueName: \"kubernetes.io/projected/602dd633-f02d-4891-a476-cc693443d0a9-kube-api-access-ftdbh\") pod \"maas-api-revoke-sub-3bef057b-gv7fm\" (UID: \"602dd633-f02d-4891-a476-cc693443d0a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bef057b-gv7fm" Oct 31 00:53:58.138313 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:58.138196 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/602dd633-f02d-4891-a476-cc693443d0a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3bef057b-gv7fm\" (UID: \"602dd633-f02d-4891-a476-cc693443d0a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bef057b-gv7fm" Oct 31 00:53:58.138913 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:58.138891 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/602dd633-f02d-4891-a476-cc693443d0a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3bef057b-gv7fm\" (UID: \"602dd633-f02d-4891-a476-cc693443d0a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bef057b-gv7fm" Oct 31 00:53:58.140702 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:58.140681 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/602dd633-f02d-4891-a476-cc693443d0a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3bef057b-gv7fm\" (UID: \"602dd633-f02d-4891-a476-cc693443d0a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bef057b-gv7fm" Oct 31 00:53:58.200910 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:58.200836 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftdbh\" (UniqueName: \"kubernetes.io/projected/602dd633-f02d-4891-a476-cc693443d0a9-kube-api-access-ftdbh\") pod \"maas-api-revoke-sub-3bef057b-gv7fm\" (UID: \"602dd633-f02d-4891-a476-cc693443d0a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bef057b-gv7fm" Oct 31 00:53:58.232580 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:58.232545 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bef057b-gv7fm" Oct 31 00:53:58.608666 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:53:58.608638 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod602dd633_f02d_4891_a476_cc693443d0a9.slice/crio-401ed7228c8fef31b98aa8ed2adf23ee511526b40cd3d9e23160ad47469db49e WatchSource:0}: Error finding container 401ed7228c8fef31b98aa8ed2adf23ee511526b40cd3d9e23160ad47469db49e: Status 404 returned error can't find the container with id 401ed7228c8fef31b98aa8ed2adf23ee511526b40cd3d9e23160ad47469db49e Oct 31 00:53:58.616626 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:58.615396 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bef057b-gv7fm"] Oct 31 00:53:58.650299 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:58.650256 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bef057b-gv7fm" event={"ID":"602dd633-f02d-4891-a476-cc693443d0a9","Type":"ContainerStarted","Data":"401ed7228c8fef31b98aa8ed2adf23ee511526b40cd3d9e23160ad47469db49e"} Oct 31 00:53:59.661141 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:59.661105 5222 generic.go:356] "Generic (PLEG): container finished" podID="602dd633-f02d-4891-a476-cc693443d0a9" containerID="50ec86e9147149f85040de005652fff27e877a83b824e7ef338ef624eb545c14" exitCode=0 Oct 31 00:53:59.661614 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:53:59.661237 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bef057b-gv7fm" event={"ID":"602dd633-f02d-4891-a476-cc693443d0a9","Type":"ContainerDied","Data":"50ec86e9147149f85040de005652fff27e877a83b824e7ef338ef624eb545c14"} Oct 31 00:54:00.851308 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:00.851279 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bef057b-gv7fm" Oct 31 00:54:00.974880 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:00.974795 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/602dd633-f02d-4891-a476-cc693443d0a9-kube-api-access-ftdbh\" (UniqueName: \"kubernetes.io/projected/602dd633-f02d-4891-a476-cc693443d0a9-kube-api-access-ftdbh\") pod \"602dd633-f02d-4891-a476-cc693443d0a9\" (UID: \"602dd633-f02d-4891-a476-cc693443d0a9\") " Oct 31 00:54:00.974880 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:00.974857 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/602dd633-f02d-4891-a476-cc693443d0a9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/602dd633-f02d-4891-a476-cc693443d0a9-maas-api-xks-ca\") pod \"602dd633-f02d-4891-a476-cc693443d0a9\" (UID: \"602dd633-f02d-4891-a476-cc693443d0a9\") " Oct 31 00:54:00.975117 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:00.974928 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/602dd633-f02d-4891-a476-cc693443d0a9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/602dd633-f02d-4891-a476-cc693443d0a9-maas-api-openshift-ca\") pod \"602dd633-f02d-4891-a476-cc693443d0a9\" (UID: \"602dd633-f02d-4891-a476-cc693443d0a9\") " Oct 31 00:54:00.975325 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:00.975297 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/602dd633-f02d-4891-a476-cc693443d0a9-maas-api-openshift-ca" pod "602dd633-f02d-4891-a476-cc693443d0a9" (UID: "602dd633-f02d-4891-a476-cc693443d0a9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:54:00.975476 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:00.975450 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/602dd633-f02d-4891-a476-cc693443d0a9-maas-api-openshift-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:00.977443 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:00.977417 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/602dd633-f02d-4891-a476-cc693443d0a9-maas-api-xks-ca" pod "602dd633-f02d-4891-a476-cc693443d0a9" (UID: "602dd633-f02d-4891-a476-cc693443d0a9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:54:00.983995 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:00.983958 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/602dd633-f02d-4891-a476-cc693443d0a9-kube-api-access-ftdbh" pod "602dd633-f02d-4891-a476-cc693443d0a9" (UID: "602dd633-f02d-4891-a476-cc693443d0a9"). InnerVolumeSpecName "kube-api-access-ftdbh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:54:01.077204 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:01.077161 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftdbh\" (UniqueName: \"kubernetes.io/projected/602dd633-f02d-4891-a476-cc693443d0a9-kube-api-access-ftdbh\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:01.077204 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:01.077198 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/602dd633-f02d-4891-a476-cc693443d0a9-maas-api-xks-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:01.737120 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:01.737075 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bef057b-gv7fm" event={"ID":"602dd633-f02d-4891-a476-cc693443d0a9","Type":"ContainerDied","Data":"401ed7228c8fef31b98aa8ed2adf23ee511526b40cd3d9e23160ad47469db49e"} Oct 31 00:54:01.737331 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:01.737147 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="401ed7228c8fef31b98aa8ed2adf23ee511526b40cd3d9e23160ad47469db49e" Oct 31 00:54:01.737331 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:01.737226 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bef057b-gv7fm" Oct 31 00:54:02.837936 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:02.837896 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-mwd5g"] Oct 31 00:54:02.838359 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:02.838122 5222 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-mwd5g" podUID="e32ce001-80c9-4be1-8a5a-46de777f11be" containerName="istio-proxy" containerID="cri-o://18004b16438c027ac33f5ecff4c92cc19fb4e0676a8890c022f2c889481e6540" gracePeriod=30 Oct 31 00:54:02.856568 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:02.856530 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-hpj9w"] Oct 31 00:54:02.856795 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:02.856771 5222 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-hpj9w" podUID="cb14bbdd-960a-457c-900c-7de064657e5e" containerName="istio-proxy" containerID="cri-o://218749fd9f675eb9807f080764790e06e6f3cc6f089833c55695c269c84bb890" gracePeriod=30 Oct 31 00:54:02.875304 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:02.875268 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-2rtnt"] Oct 31 00:54:02.875576 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:02.875535 5222 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-2rtnt" podUID="69c94b24-49c5-49b4-825b-50778d04669a" containerName="istio-proxy" containerID="cri-o://1759f6dde775c2a3e85842b471291eb25eeec97618130cd89261cc2a9abdf738" gracePeriod=30 Oct 31 00:54:03.328119 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:03.328080 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-50a19d-stls7"] Oct 31 00:54:03.332843 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:03.332804 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-50a19d-stls7"] Oct 31 00:54:03.657243 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:03.657156 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c514b249-2a36-467e-9887-98684a616d22" path="/var/lib/kubelet/pods/c514b249-2a36-467e-9887-98684a616d22/volumes" Oct 31 00:54:05.919881 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:54:05.919850 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 00:54:06.203964 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:06.203878 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-79tk8"] Oct 31 00:54:08.175437 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:08.175408 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-mwd5g" Oct 31 00:54:08.192214 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:08.192184 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-hpj9w" Oct 31 00:54:08.202198 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:08.202174 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-2rtnt" Oct 31 00:54:08.349416 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:08.349360 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/69c94b24-49c5-49b4-825b-50778d04669a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69c94b24-49c5-49b4-825b-50778d04669a-istio-podinfo\") pod \"69c94b24-49c5-49b4-825b-50778d04669a\" (UID: \"69c94b24-49c5-49b4-825b-50778d04669a\") " Oct 31 00:54:08.349624 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:08.349470 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e32ce001-80c9-4be1-8a5a-46de777f11be-kube-api-access-j567t\" (UniqueName: \"kubernetes.io/projected/e32ce001-80c9-4be1-8a5a-46de777f11be-kube-api-access-j567t\") pod \"e32ce001-80c9-4be1-8a5a-46de777f11be\" (UID: \"e32ce001-80c9-4be1-8a5a-46de777f11be\") " Oct 31 00:54:08.349624 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:08.349516 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69c94b24-49c5-49b4-825b-50778d04669a-istio-token\" (UniqueName: \"kubernetes.io/projected/69c94b24-49c5-49b4-825b-50778d04669a-istio-token\") pod \"69c94b24-49c5-49b4-825b-50778d04669a\" (UID: \"69c94b24-49c5-49b4-825b-50778d04669a\") " Oct 31 00:54:08.349624 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:08.349563 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69c94b24-49c5-49b4-825b-50778d04669a-kube-api-access-pl4vw\" (UniqueName: \"kubernetes.io/projected/69c94b24-49c5-49b4-825b-50778d04669a-kube-api-access-pl4vw\") pod \"69c94b24-49c5-49b4-825b-50778d04669a\" (UID: \"69c94b24-49c5-49b4-825b-50778d04669a\") " Oct 31 00:54:08.349624 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:08.349603 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e32ce001-80c9-4be1-8a5a-46de777f11be-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e32ce001-80c9-4be1-8a5a-46de777f11be-workload-certs\") pod \"e32ce001-80c9-4be1-8a5a-46de777f11be\" (UID: \"e32ce001-80c9-4be1-8a5a-46de777f11be\") " Oct 31 00:54:08.349879 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:08.349654 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cb14bbdd-960a-457c-900c-7de064657e5e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb14bbdd-960a-457c-900c-7de064657e5e-istio-podinfo\") pod \"cb14bbdd-960a-457c-900c-7de064657e5e\" (UID: \"cb14bbdd-960a-457c-900c-7de064657e5e\") " Oct 31 00:54:08.349879 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:08.349693 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb14bbdd-960a-457c-900c-7de064657e5e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb14bbdd-960a-457c-900c-7de064657e5e-credential-socket\") pod \"cb14bbdd-960a-457c-900c-7de064657e5e\" (UID: \"cb14bbdd-960a-457c-900c-7de064657e5e\") " Oct 31 00:54:08.349879 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:08.349734 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e32ce001-80c9-4be1-8a5a-46de777f11be-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e32ce001-80c9-4be1-8a5a-46de777f11be-istio-envoy\") pod \"e32ce001-80c9-4be1-8a5a-46de777f11be\" (UID: \"e32ce001-80c9-4be1-8a5a-46de777f11be\") " Oct 31 00:54:08.349879 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:08.349773 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb14bbdd-960a-457c-900c-7de064657e5e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb14bbdd-960a-457c-900c-7de064657e5e-istio-data\") pod \"cb14bbdd-960a-457c-900c-7de064657e5e\" (UID: \"cb14bbdd-960a-457c-900c-7de064657e5e\") " Oct 31 00:54:08.349879 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:08.349809 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb14bbdd-960a-457c-900c-7de064657e5e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb14bbdd-960a-457c-900c-7de064657e5e-istio-envoy\") pod \"cb14bbdd-960a-457c-900c-7de064657e5e\" (UID: \"cb14bbdd-960a-457c-900c-7de064657e5e\") " Oct 31 00:54:08.349879 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:08.349844 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb14bbdd-960a-457c-900c-7de064657e5e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb14bbdd-960a-457c-900c-7de064657e5e-workload-socket\") pod \"cb14bbdd-960a-457c-900c-7de064657e5e\" (UID: \"cb14bbdd-960a-457c-900c-7de064657e5e\") " Oct 31 00:54:08.350404 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:08.349885 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69c94b24-49c5-49b4-825b-50778d04669a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69c94b24-49c5-49b4-825b-50778d04669a-workload-certs\") pod \"69c94b24-49c5-49b4-825b-50778d04669a\" (UID: \"69c94b24-49c5-49b4-825b-50778d04669a\") " Oct 31 00:54:08.350404 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:08.349954 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb14bbdd-960a-457c-900c-7de064657e5e-istio-token\" (UniqueName: \"kubernetes.io/projected/cb14bbdd-960a-457c-900c-7de064657e5e-istio-token\") pod \"cb14bbdd-960a-457c-900c-7de064657e5e\" (UID: \"cb14bbdd-960a-457c-900c-7de064657e5e\") " Oct 31 00:54:08.350404 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:08.349991 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e32ce001-80c9-4be1-8a5a-46de777f11be-istio-token\" (UniqueName: \"kubernetes.io/projected/e32ce001-80c9-4be1-8a5a-46de777f11be-istio-token\") pod \"e32ce001-80c9-4be1-8a5a-46de777f11be\" (UID: \"e32ce001-80c9-4be1-8a5a-46de777f11be\") " Oct 31 00:54:08.350404 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:08.350031 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb14bbdd-960a-457c-900c-7de064657e5e-kube-api-access-c5gbs\" (UniqueName: \"kubernetes.io/projected/cb14bbdd-960a-457c-900c-7de064657e5e-kube-api-access-c5gbs\") pod \"cb14bbdd-960a-457c-900c-7de064657e5e\" (UID: \"cb14bbdd-960a-457c-900c-7de064657e5e\") " Oct 31 00:54:08.350404 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:08.350091 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e32ce001-80c9-4be1-8a5a-46de777f11be-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e32ce001-80c9-4be1-8a5a-46de777f11be-istiod-ca-cert\") pod \"e32ce001-80c9-4be1-8a5a-46de777f11be\" (UID: \"e32ce001-80c9-4be1-8a5a-46de777f11be\") " Oct 31 00:54:08.350404 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:08.350134 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb14bbdd-960a-457c-900c-7de064657e5e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb14bbdd-960a-457c-900c-7de064657e5e-workload-certs\") pod \"cb14bbdd-960a-457c-900c-7de064657e5e\" (UID: \"cb14bbdd-960a-457c-900c-7de064657e5e\") " Oct 31 00:54:08.350404 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:08.350175 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e32ce001-80c9-4be1-8a5a-46de777f11be-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e32ce001-80c9-4be1-8a5a-46de777f11be-workload-socket\") pod \"e32ce001-80c9-4be1-8a5a-46de777f11be\" (UID: \"e32ce001-80c9-4be1-8a5a-46de777f11be\") " Oct 31 00:54:08.350404 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:08.350210 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69c94b24-49c5-49b4-825b-50778d04669a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69c94b24-49c5-49b4-825b-50778d04669a-istio-data\") pod \"69c94b24-49c5-49b4-825b-50778d04669a\" (UID: \"69c94b24-49c5-49b4-825b-50778d04669a\") " Oct 31 00:54:08.350404 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:08.350237 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e32ce001-80c9-4be1-8a5a-46de777f11be-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e32ce001-80c9-4be1-8a5a-46de777f11be-credential-socket\") pod \"e32ce001-80c9-4be1-8a5a-46de777f11be\" (UID: \"e32ce001-80c9-4be1-8a5a-46de777f11be\") " Oct 31 00:54:08.350895 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:08.350436 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69c94b24-49c5-49b4-825b-50778d04669a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69c94b24-49c5-49b4-825b-50778d04669a-istio-envoy\") pod \"69c94b24-49c5-49b4-825b-50778d04669a\" (UID: \"69c94b24-49c5-49b4-825b-50778d04669a\") " Oct 31 00:54:08.350895 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:08.350478 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb14bbdd-960a-457c-900c-7de064657e5e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb14bbdd-960a-457c-900c-7de064657e5e-istiod-ca-cert\") pod \"cb14bbdd-960a-457c-900c-7de064657e5e\" (UID: \"cb14bbdd-960a-457c-900c-7de064657e5e\") " Oct 31 00:54:08.350895 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:08.350511 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e32ce001-80c9-4be1-8a5a-46de777f11be-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e32ce001-80c9-4be1-8a5a-46de777f11be-istio-data\") pod \"e32ce001-80c9-4be1-8a5a-46de777f11be\" (UID: \"e32ce001-80c9-4be1-8a5a-46de777f11be\") " Oct 31 00:54:08.350895 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:08.350543 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69c94b24-49c5-49b4-825b-50778d04669a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69c94b24-49c5-49b4-825b-50778d04669a-credential-socket\") pod \"69c94b24-49c5-49b4-825b-50778d04669a\" (UID: \"69c94b24-49c5-49b4-825b-50778d04669a\") " Oct 31 00:54:08.350895 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:08.350597 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69c94b24-49c5-49b4-825b-50778d04669a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69c94b24-49c5-49b4-825b-50778d04669a-istiod-ca-cert\") pod \"69c94b24-49c5-49b4-825b-50778d04669a\" (UID: \"69c94b24-49c5-49b4-825b-50778d04669a\") " Oct 31 00:54:08.350895 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:08.350752 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69c94b24-49c5-49b4-825b-50778d04669a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69c94b24-49c5-49b4-825b-50778d04669a-workload-socket\") pod \"69c94b24-49c5-49b4-825b-50778d04669a\" (UID: \"69c94b24-49c5-49b4-825b-50778d04669a\") " Oct 31 00:54:08.350895 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:08.350806 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e32ce001-80c9-4be1-8a5a-46de777f11be-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e32ce001-80c9-4be1-8a5a-46de777f11be-istio-podinfo\") pod \"e32ce001-80c9-4be1-8a5a-46de777f11be\" (UID: \"e32ce001-80c9-4be1-8a5a-46de777f11be\") " Oct 31 00:54:08.352221 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:08.351998 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb14bbdd-960a-457c-900c-7de064657e5e-istio-data" pod "cb14bbdd-960a-457c-900c-7de064657e5e" (UID: "cb14bbdd-960a-457c-900c-7de064657e5e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:54:08.352343 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:08.352236 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb14bbdd-960a-457c-900c-7de064657e5e-credential-socket" pod "cb14bbdd-960a-457c-900c-7de064657e5e" (UID: "cb14bbdd-960a-457c-900c-7de064657e5e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:54:08.353392 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:08.352927 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb14bbdd-960a-457c-900c-7de064657e5e-istiod-ca-cert" pod "cb14bbdd-960a-457c-900c-7de064657e5e" (UID: "cb14bbdd-960a-457c-900c-7de064657e5e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:54:08.353392 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:08.352934 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/69c94b24-49c5-49b4-825b-50778d04669a-istio-podinfo" pod "69c94b24-49c5-49b4-825b-50778d04669a" (UID: "69c94b24-49c5-49b4-825b-50778d04669a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:54:08.353392 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:08.353167 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69c94b24-49c5-49b4-825b-50778d04669a-istio-token" pod "69c94b24-49c5-49b4-825b-50778d04669a" (UID: "69c94b24-49c5-49b4-825b-50778d04669a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:54:08.353658 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:08.353493 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e32ce001-80c9-4be1-8a5a-46de777f11be-workload-socket" pod "e32ce001-80c9-4be1-8a5a-46de777f11be" (UID: "e32ce001-80c9-4be1-8a5a-46de777f11be"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:54:08.353797 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:08.353681 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb14bbdd-960a-457c-900c-7de064657e5e-workload-certs" pod "cb14bbdd-960a-457c-900c-7de064657e5e" (UID: "cb14bbdd-960a-457c-900c-7de064657e5e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:54:08.353948 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:08.353931 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e32ce001-80c9-4be1-8a5a-46de777f11be-istiod-ca-cert" pod "e32ce001-80c9-4be1-8a5a-46de777f11be" (UID: "e32ce001-80c9-4be1-8a5a-46de777f11be"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:54:08.354223 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:08.354171 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69c94b24-49c5-49b4-825b-50778d04669a-workload-certs" pod "69c94b24-49c5-49b4-825b-50778d04669a" (UID: "69c94b24-49c5-49b4-825b-50778d04669a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:54:08.354824 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:08.354359 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69c94b24-49c5-49b4-825b-50778d04669a-credential-socket" pod "69c94b24-49c5-49b4-825b-50778d04669a" (UID: "69c94b24-49c5-49b4-825b-50778d04669a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:54:08.354824 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:08.354387 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69c94b24-49c5-49b4-825b-50778d04669a-istio-data" pod "69c94b24-49c5-49b4-825b-50778d04669a" (UID: "69c94b24-49c5-49b4-825b-50778d04669a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:54:08.354824 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:08.354561 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e32ce001-80c9-4be1-8a5a-46de777f11be-istio-data" pod "e32ce001-80c9-4be1-8a5a-46de777f11be" (UID: "e32ce001-80c9-4be1-8a5a-46de777f11be"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:54:08.354824 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:08.354735 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e32ce001-80c9-4be1-8a5a-46de777f11be-workload-certs" pod "e32ce001-80c9-4be1-8a5a-46de777f11be" (UID: "e32ce001-80c9-4be1-8a5a-46de777f11be"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:54:08.354824 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:08.354812 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e32ce001-80c9-4be1-8a5a-46de777f11be-credential-socket" pod "e32ce001-80c9-4be1-8a5a-46de777f11be" (UID: "e32ce001-80c9-4be1-8a5a-46de777f11be"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:54:08.355232 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:08.354830 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb14bbdd-960a-457c-900c-7de064657e5e-workload-socket" pod "cb14bbdd-960a-457c-900c-7de064657e5e" (UID: "cb14bbdd-960a-457c-900c-7de064657e5e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:54:08.355232 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:08.354862 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69c94b24-49c5-49b4-825b-50778d04669a-istiod-ca-cert" pod "69c94b24-49c5-49b4-825b-50778d04669a" (UID: "69c94b24-49c5-49b4-825b-50778d04669a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:54:08.355232 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:08.355009 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69c94b24-49c5-49b4-825b-50778d04669a-workload-socket" pod "69c94b24-49c5-49b4-825b-50778d04669a" (UID: "69c94b24-49c5-49b4-825b-50778d04669a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:54:08.359610 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:08.359582 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e32ce001-80c9-4be1-8a5a-46de777f11be-kube-api-access-j567t" pod "e32ce001-80c9-4be1-8a5a-46de777f11be" (UID: "e32ce001-80c9-4be1-8a5a-46de777f11be"). InnerVolumeSpecName "kube-api-access-j567t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:54:08.359725 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:08.359698 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb14bbdd-960a-457c-900c-7de064657e5e-kube-api-access-c5gbs" pod "cb14bbdd-960a-457c-900c-7de064657e5e" (UID: "cb14bbdd-960a-457c-900c-7de064657e5e"). InnerVolumeSpecName "kube-api-access-c5gbs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:54:08.359784 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:08.359721 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e32ce001-80c9-4be1-8a5a-46de777f11be-istio-token" pod "e32ce001-80c9-4be1-8a5a-46de777f11be" (UID: "e32ce001-80c9-4be1-8a5a-46de777f11be"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:54:08.359784 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:08.359722 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69c94b24-49c5-49b4-825b-50778d04669a-istio-envoy" pod "69c94b24-49c5-49b4-825b-50778d04669a" (UID: "69c94b24-49c5-49b4-825b-50778d04669a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:54:08.359855 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:08.359832 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e32ce001-80c9-4be1-8a5a-46de777f11be-istio-podinfo" pod "e32ce001-80c9-4be1-8a5a-46de777f11be" (UID: "e32ce001-80c9-4be1-8a5a-46de777f11be"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:54:08.360103 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:08.360084 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb14bbdd-960a-457c-900c-7de064657e5e-istio-envoy" pod "cb14bbdd-960a-457c-900c-7de064657e5e" (UID: "cb14bbdd-960a-457c-900c-7de064657e5e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:54:08.360166 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:08.360141 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e32ce001-80c9-4be1-8a5a-46de777f11be-istio-envoy" pod "e32ce001-80c9-4be1-8a5a-46de777f11be" (UID: "e32ce001-80c9-4be1-8a5a-46de777f11be"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:54:08.360246 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:08.360227 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69c94b24-49c5-49b4-825b-50778d04669a-kube-api-access-pl4vw" pod "69c94b24-49c5-49b4-825b-50778d04669a" (UID: "69c94b24-49c5-49b4-825b-50778d04669a"). InnerVolumeSpecName "kube-api-access-pl4vw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:54:08.360522 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:08.360505 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb14bbdd-960a-457c-900c-7de064657e5e-istio-token" pod "cb14bbdd-960a-457c-900c-7de064657e5e" (UID: "cb14bbdd-960a-457c-900c-7de064657e5e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:54:08.360584 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:08.360537 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cb14bbdd-960a-457c-900c-7de064657e5e-istio-podinfo" pod "cb14bbdd-960a-457c-900c-7de064657e5e" (UID: "cb14bbdd-960a-457c-900c-7de064657e5e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:54:08.453016 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:08.452924 5222 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69c94b24-49c5-49b4-825b-50778d04669a-workload-socket\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:08.453016 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:08.452957 5222 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e32ce001-80c9-4be1-8a5a-46de777f11be-istio-podinfo\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:08.453016 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:08.452972 5222 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69c94b24-49c5-49b4-825b-50778d04669a-istio-podinfo\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:08.453016 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:08.452985 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j567t\" (UniqueName: \"kubernetes.io/projected/e32ce001-80c9-4be1-8a5a-46de777f11be-kube-api-access-j567t\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:08.453016 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:08.452996 5222 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69c94b24-49c5-49b4-825b-50778d04669a-istio-token\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:08.453016 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:08.453008 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pl4vw\" (UniqueName: \"kubernetes.io/projected/69c94b24-49c5-49b4-825b-50778d04669a-kube-api-access-pl4vw\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:08.453432 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:08.453021 5222 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e32ce001-80c9-4be1-8a5a-46de777f11be-workload-certs\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:08.453432 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:08.453035 5222 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb14bbdd-960a-457c-900c-7de064657e5e-istio-podinfo\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:08.453432 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:08.453050 5222 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb14bbdd-960a-457c-900c-7de064657e5e-credential-socket\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:08.453432 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:08.453061 5222 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e32ce001-80c9-4be1-8a5a-46de777f11be-istio-envoy\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:08.453432 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:08.453072 5222 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb14bbdd-960a-457c-900c-7de064657e5e-istio-data\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:08.453432 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:08.453083 5222 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb14bbdd-960a-457c-900c-7de064657e5e-istio-envoy\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:08.453432 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:08.453096 5222 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb14bbdd-960a-457c-900c-7de064657e5e-workload-socket\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:08.453432 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:08.453107 5222 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69c94b24-49c5-49b4-825b-50778d04669a-workload-certs\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:08.453432 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:08.453118 5222 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb14bbdd-960a-457c-900c-7de064657e5e-istio-token\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:08.453432 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:08.453131 5222 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e32ce001-80c9-4be1-8a5a-46de777f11be-istio-token\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:08.453432 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:08.453144 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5gbs\" (UniqueName: \"kubernetes.io/projected/cb14bbdd-960a-457c-900c-7de064657e5e-kube-api-access-c5gbs\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:08.453432 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:08.453156 5222 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e32ce001-80c9-4be1-8a5a-46de777f11be-istiod-ca-cert\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:08.453432 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:08.453167 5222 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb14bbdd-960a-457c-900c-7de064657e5e-workload-certs\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:08.453432 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:08.453179 5222 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e32ce001-80c9-4be1-8a5a-46de777f11be-workload-socket\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:08.453432 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:08.453191 5222 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69c94b24-49c5-49b4-825b-50778d04669a-istio-data\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:08.453432 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:08.453202 5222 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e32ce001-80c9-4be1-8a5a-46de777f11be-credential-socket\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:08.453432 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:08.453215 5222 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69c94b24-49c5-49b4-825b-50778d04669a-istio-envoy\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:08.453432 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:08.453228 5222 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb14bbdd-960a-457c-900c-7de064657e5e-istiod-ca-cert\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:08.453432 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:08.453241 5222 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e32ce001-80c9-4be1-8a5a-46de777f11be-istio-data\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:08.453432 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:08.453255 5222 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69c94b24-49c5-49b4-825b-50778d04669a-credential-socket\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:08.453432 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:08.453269 5222 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69c94b24-49c5-49b4-825b-50778d04669a-istiod-ca-cert\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:08.791234 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:08.791198 5222 generic.go:356] "Generic (PLEG): container finished" podID="e32ce001-80c9-4be1-8a5a-46de777f11be" containerID="18004b16438c027ac33f5ecff4c92cc19fb4e0676a8890c022f2c889481e6540" exitCode=0 Oct 31 00:54:08.791455 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:08.791242 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-mwd5g" Oct 31 00:54:08.791455 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:08.791283 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-mwd5g" event={"ID":"e32ce001-80c9-4be1-8a5a-46de777f11be","Type":"ContainerDied","Data":"18004b16438c027ac33f5ecff4c92cc19fb4e0676a8890c022f2c889481e6540"} Oct 31 00:54:08.791455 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:08.791338 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-mwd5g" event={"ID":"e32ce001-80c9-4be1-8a5a-46de777f11be","Type":"ContainerDied","Data":"ff84d7df07c31dcc480d926de626ee54bd59c4a24bdbde3c0d672dd3d8e52fc2"} Oct 31 00:54:08.791455 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:08.791362 5222 scope.go:122] "RemoveContainer" containerID="18004b16438c027ac33f5ecff4c92cc19fb4e0676a8890c022f2c889481e6540" Oct 31 00:54:08.793129 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:08.793102 5222 generic.go:356] "Generic (PLEG): container finished" podID="69c94b24-49c5-49b4-825b-50778d04669a" containerID="1759f6dde775c2a3e85842b471291eb25eeec97618130cd89261cc2a9abdf738" exitCode=0 Oct 31 00:54:08.793297 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:08.793158 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-2rtnt" Oct 31 00:54:08.793297 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:08.793186 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-2rtnt" event={"ID":"69c94b24-49c5-49b4-825b-50778d04669a","Type":"ContainerDied","Data":"1759f6dde775c2a3e85842b471291eb25eeec97618130cd89261cc2a9abdf738"} Oct 31 00:54:08.793297 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:08.793242 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-2rtnt" event={"ID":"69c94b24-49c5-49b4-825b-50778d04669a","Type":"ContainerDied","Data":"5b8b1a33f821a2b3071308ea0879b7b2276c2fdfa073fea74fbfef6766a31885"} Oct 31 00:54:08.794935 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:08.794909 5222 generic.go:356] "Generic (PLEG): container finished" podID="cb14bbdd-960a-457c-900c-7de064657e5e" containerID="218749fd9f675eb9807f080764790e06e6f3cc6f089833c55695c269c84bb890" exitCode=0 Oct 31 00:54:08.795093 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:08.794947 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-hpj9w" Oct 31 00:54:08.795093 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:08.794963 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-hpj9w" event={"ID":"cb14bbdd-960a-457c-900c-7de064657e5e","Type":"ContainerDied","Data":"218749fd9f675eb9807f080764790e06e6f3cc6f089833c55695c269c84bb890"} Oct 31 00:54:08.795093 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:08.795002 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-hpj9w" event={"ID":"cb14bbdd-960a-457c-900c-7de064657e5e","Type":"ContainerDied","Data":"88a789530b09b9285299a5e93bc99a711c8733f7f2cbd9b247358a3add2a60f5"} Oct 31 00:54:08.804159 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:08.804135 5222 scope.go:122] "RemoveContainer" containerID="18004b16438c027ac33f5ecff4c92cc19fb4e0676a8890c022f2c889481e6540" Oct 31 00:54:08.804568 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:54:08.804541 5222 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18004b16438c027ac33f5ecff4c92cc19fb4e0676a8890c022f2c889481e6540\": container with ID starting with 18004b16438c027ac33f5ecff4c92cc19fb4e0676a8890c022f2c889481e6540 not found: ID does not exist" containerID="18004b16438c027ac33f5ecff4c92cc19fb4e0676a8890c022f2c889481e6540" Oct 31 00:54:08.804681 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:08.804577 5222 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18004b16438c027ac33f5ecff4c92cc19fb4e0676a8890c022f2c889481e6540"} err="failed to get container status \"18004b16438c027ac33f5ecff4c92cc19fb4e0676a8890c022f2c889481e6540\": rpc error: code = NotFound desc = could not find container \"18004b16438c027ac33f5ecff4c92cc19fb4e0676a8890c022f2c889481e6540\": container with ID starting with 18004b16438c027ac33f5ecff4c92cc19fb4e0676a8890c022f2c889481e6540 not found: ID does not exist" Oct 31 00:54:08.804681 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:08.804602 5222 scope.go:122] "RemoveContainer" containerID="1759f6dde775c2a3e85842b471291eb25eeec97618130cd89261cc2a9abdf738" Oct 31 00:54:08.819210 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:08.819183 5222 scope.go:122] "RemoveContainer" containerID="1759f6dde775c2a3e85842b471291eb25eeec97618130cd89261cc2a9abdf738" Oct 31 00:54:08.819787 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:54:08.819761 5222 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1759f6dde775c2a3e85842b471291eb25eeec97618130cd89261cc2a9abdf738\": container with ID starting with 1759f6dde775c2a3e85842b471291eb25eeec97618130cd89261cc2a9abdf738 not found: ID does not exist" containerID="1759f6dde775c2a3e85842b471291eb25eeec97618130cd89261cc2a9abdf738" Oct 31 00:54:08.819885 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:08.819799 5222 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1759f6dde775c2a3e85842b471291eb25eeec97618130cd89261cc2a9abdf738"} err="failed to get container status \"1759f6dde775c2a3e85842b471291eb25eeec97618130cd89261cc2a9abdf738\": rpc error: code = NotFound desc = could not find container \"1759f6dde775c2a3e85842b471291eb25eeec97618130cd89261cc2a9abdf738\": container with ID starting with 1759f6dde775c2a3e85842b471291eb25eeec97618130cd89261cc2a9abdf738 not found: ID does not exist" Oct 31 00:54:08.819885 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:08.819824 5222 scope.go:122] "RemoveContainer" containerID="218749fd9f675eb9807f080764790e06e6f3cc6f089833c55695c269c84bb890" Oct 31 00:54:08.833314 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:08.833291 5222 scope.go:122] "RemoveContainer" containerID="218749fd9f675eb9807f080764790e06e6f3cc6f089833c55695c269c84bb890" Oct 31 00:54:08.833727 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:54:08.833703 5222 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"218749fd9f675eb9807f080764790e06e6f3cc6f089833c55695c269c84bb890\": container with ID starting with 218749fd9f675eb9807f080764790e06e6f3cc6f089833c55695c269c84bb890 not found: ID does not exist" containerID="218749fd9f675eb9807f080764790e06e6f3cc6f089833c55695c269c84bb890" Oct 31 00:54:08.833829 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:08.833738 5222 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"218749fd9f675eb9807f080764790e06e6f3cc6f089833c55695c269c84bb890"} err="failed to get container status \"218749fd9f675eb9807f080764790e06e6f3cc6f089833c55695c269c84bb890\": rpc error: code = NotFound desc = could not find container \"218749fd9f675eb9807f080764790e06e6f3cc6f089833c55695c269c84bb890\": container with ID starting with 218749fd9f675eb9807f080764790e06e6f3cc6f089833c55695c269c84bb890 not found: ID does not exist" Oct 31 00:54:08.961816 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:08.961784 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-mwd5g"] Oct 31 00:54:09.012881 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:09.012847 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-mwd5g"] Oct 31 00:54:09.078239 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:09.078133 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-2rtnt"] Oct 31 00:54:09.085813 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:09.085779 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-2rtnt"] Oct 31 00:54:09.198234 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:09.198202 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-hpj9w"] Oct 31 00:54:09.203812 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:09.203782 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-hpj9w"] Oct 31 00:54:09.657914 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:09.657880 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69c94b24-49c5-49b4-825b-50778d04669a" path="/var/lib/kubelet/pods/69c94b24-49c5-49b4-825b-50778d04669a/volumes" Oct 31 00:54:09.658381 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:09.658352 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb14bbdd-960a-457c-900c-7de064657e5e" path="/var/lib/kubelet/pods/cb14bbdd-960a-457c-900c-7de064657e5e/volumes" Oct 31 00:54:09.658849 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:09.658832 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e32ce001-80c9-4be1-8a5a-46de777f11be" path="/var/lib/kubelet/pods/e32ce001-80c9-4be1-8a5a-46de777f11be/volumes" Oct 31 00:54:09.673353 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:09.673325 5222 scope.go:122] "RemoveContainer" containerID="b9575b0f0e60615f1565c41ee5035718819a039d3a6f9f6b82aa1309effe4818" Oct 31 00:54:09.684250 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:09.684220 5222 scope.go:122] "RemoveContainer" containerID="f9dc36877834ec53cf93518d59a7ae1f923c22d665dc2aa77b62d7c8c6c8f9e7" Oct 31 00:54:09.698790 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:09.698768 5222 scope.go:122] "RemoveContainer" containerID="a1e6ade7838d5c818c690a3fb64bc36b8f67c1d3876d415d0f2f853d77edcb9c" Oct 31 00:54:10.592572 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:10.592537 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-50a19d-openshift-default-68c6d7d74c-24fwv"] Oct 31 00:54:10.593089 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:10.592841 5222 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-50a19d-openshift-default-68c6d7d74c-24fwv" podUID="1f2eb643-034b-4b9a-bd2f-6ddb034008b8" containerName="istio-proxy" containerID="cri-o://5c11287883f26b13ebd3c0fbefb7ffe9f512dd4615040c56d0e120124faccc7b" gracePeriod=30 Oct 31 00:54:11.693230 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:11.693190 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-1e5e75-l5vzz"] Oct 31 00:54:11.694013 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:11.693993 5222 cpu_manager.go:414] "containerMap: removing container" podUID="69c94b24-49c5-49b4-825b-50778d04669a" containerName="istio-proxy" Oct 31 00:54:11.694013 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:11.694013 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69c94b24-49c5-49b4-825b-50778d04669a" containerName="istio-proxy" Oct 31 00:54:11.694138 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:11.694034 5222 cpu_manager.go:414] "containerMap: removing container" podUID="f85c9ba2-c597-4786-93a1-930609ff510a" containerName="istio-proxy" Oct 31 00:54:11.694138 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:11.694042 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f85c9ba2-c597-4786-93a1-930609ff510a" containerName="istio-proxy" Oct 31 00:54:11.694138 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:11.694069 5222 cpu_manager.go:414] "containerMap: removing container" podUID="602dd633-f02d-4891-a476-cc693443d0a9" containerName="revoke-keys" Oct 31 00:54:11.694138 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:11.694077 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="602dd633-f02d-4891-a476-cc693443d0a9" containerName="revoke-keys" Oct 31 00:54:11.694138 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:11.694124 5222 cpu_manager.go:414] "containerMap: removing container" podUID="cb14bbdd-960a-457c-900c-7de064657e5e" containerName="istio-proxy" Oct 31 00:54:11.694138 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:11.694131 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb14bbdd-960a-457c-900c-7de064657e5e" containerName="istio-proxy" Oct 31 00:54:11.694404 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:11.694147 5222 cpu_manager.go:414] "containerMap: removing container" podUID="e32ce001-80c9-4be1-8a5a-46de777f11be" containerName="istio-proxy" Oct 31 00:54:11.694404 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:11.694155 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e32ce001-80c9-4be1-8a5a-46de777f11be" containerName="istio-proxy" Oct 31 00:54:11.694404 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:11.694262 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb14bbdd-960a-457c-900c-7de064657e5e" containerName="istio-proxy" Oct 31 00:54:11.694404 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:11.694277 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="e32ce001-80c9-4be1-8a5a-46de777f11be" containerName="istio-proxy" Oct 31 00:54:11.694404 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:11.694288 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="69c94b24-49c5-49b4-825b-50778d04669a" containerName="istio-proxy" Oct 31 00:54:11.694404 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:11.694300 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="602dd633-f02d-4891-a476-cc693443d0a9" containerName="revoke-keys" Oct 31 00:54:11.699999 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:11.699971 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-1e5e75-l5vzz" Oct 31 00:54:11.703026 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:11.702996 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fde6a27f-2b5e-4e9e-9b39-e53c294597b7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-1e5e75-l5vzz\" (UID: \"fde6a27f-2b5e-4e9e-9b39-e53c294597b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-1e5e75-l5vzz" Oct 31 00:54:11.703178 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:11.703116 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82xfn\" (UniqueName: \"kubernetes.io/projected/fde6a27f-2b5e-4e9e-9b39-e53c294597b7-kube-api-access-82xfn\") pod \"maas-api-revoke-keys-e2e-worker-w3-1e5e75-l5vzz\" (UID: \"fde6a27f-2b5e-4e9e-9b39-e53c294597b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-1e5e75-l5vzz" Oct 31 00:54:11.703271 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:11.703255 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fde6a27f-2b5e-4e9e-9b39-e53c294597b7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-1e5e75-l5vzz\" (UID: \"fde6a27f-2b5e-4e9e-9b39-e53c294597b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-1e5e75-l5vzz" Oct 31 00:54:11.749332 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:11.749292 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-1e5e75-l5vzz"] Oct 31 00:54:11.797203 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:11.797163 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:54:11.797203 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:11.797189 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfz9z\"" Oct 31 00:54:11.797203 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:11.797188 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:54:11.797505 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:11.797175 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:54:11.804219 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:11.804185 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fde6a27f-2b5e-4e9e-9b39-e53c294597b7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-1e5e75-l5vzz\" (UID: \"fde6a27f-2b5e-4e9e-9b39-e53c294597b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-1e5e75-l5vzz" Oct 31 00:54:11.804352 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:11.804245 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82xfn\" (UniqueName: \"kubernetes.io/projected/fde6a27f-2b5e-4e9e-9b39-e53c294597b7-kube-api-access-82xfn\") pod \"maas-api-revoke-keys-e2e-worker-w3-1e5e75-l5vzz\" (UID: \"fde6a27f-2b5e-4e9e-9b39-e53c294597b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-1e5e75-l5vzz" Oct 31 00:54:11.804352 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:11.804319 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fde6a27f-2b5e-4e9e-9b39-e53c294597b7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-1e5e75-l5vzz\" (UID: \"fde6a27f-2b5e-4e9e-9b39-e53c294597b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-1e5e75-l5vzz" Oct 31 00:54:11.804955 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:11.804930 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fde6a27f-2b5e-4e9e-9b39-e53c294597b7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-1e5e75-l5vzz\" (UID: \"fde6a27f-2b5e-4e9e-9b39-e53c294597b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-1e5e75-l5vzz" Oct 31 00:54:11.806871 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:11.806844 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fde6a27f-2b5e-4e9e-9b39-e53c294597b7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-1e5e75-l5vzz\" (UID: \"fde6a27f-2b5e-4e9e-9b39-e53c294597b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-1e5e75-l5vzz" Oct 31 00:54:11.867751 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:11.867717 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82xfn\" (UniqueName: \"kubernetes.io/projected/fde6a27f-2b5e-4e9e-9b39-e53c294597b7-kube-api-access-82xfn\") pod \"maas-api-revoke-keys-e2e-worker-w3-1e5e75-l5vzz\" (UID: \"fde6a27f-2b5e-4e9e-9b39-e53c294597b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-1e5e75-l5vzz" Oct 31 00:54:12.016063 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:12.016032 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-1e5e75-l5vzz" Oct 31 00:54:12.390105 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:54:12.390054 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfde6a27f_2b5e_4e9e_9b39_e53c294597b7.slice/crio-4da7bd48e2ad625da3311e69959dfa64652e48b313f8ecbd49cdfacf89c34287 WatchSource:0}: Error finding container 4da7bd48e2ad625da3311e69959dfa64652e48b313f8ecbd49cdfacf89c34287: Status 404 returned error can't find the container with id 4da7bd48e2ad625da3311e69959dfa64652e48b313f8ecbd49cdfacf89c34287 Oct 31 00:54:12.393026 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:12.393003 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-1e5e75-l5vzz"] Oct 31 00:54:12.828328 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:12.828293 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-1e5e75-l5vzz" event={"ID":"fde6a27f-2b5e-4e9e-9b39-e53c294597b7","Type":"ContainerStarted","Data":"0b7c7f06d2559c2aef5ff0e50b8cc4fbeef1940ca2da763853d2f6bd4248e92c"} Oct 31 00:54:12.828752 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:12.828344 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-1e5e75-l5vzz" event={"ID":"fde6a27f-2b5e-4e9e-9b39-e53c294597b7","Type":"ContainerStarted","Data":"4da7bd48e2ad625da3311e69959dfa64652e48b313f8ecbd49cdfacf89c34287"} Oct 31 00:54:12.947345 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:12.947224 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-1e5e75-l5vzz" podStartSLOduration=1.9472060230000001 podStartE2EDuration="1.947206023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:54:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:54:12.947115547 +0000 UTC m=+2345.418285297" watchObservedRunningTime="2026-10-31 00:54:12.947206023 +0000 UTC m=+2345.418375776" Oct 31 00:54:13.837267 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:13.837231 5222 generic.go:356] "Generic (PLEG): container finished" podID="fde6a27f-2b5e-4e9e-9b39-e53c294597b7" containerID="0b7c7f06d2559c2aef5ff0e50b8cc4fbeef1940ca2da763853d2f6bd4248e92c" exitCode=0 Oct 31 00:54:13.837747 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:13.837319 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-1e5e75-l5vzz" event={"ID":"fde6a27f-2b5e-4e9e-9b39-e53c294597b7","Type":"ContainerDied","Data":"0b7c7f06d2559c2aef5ff0e50b8cc4fbeef1940ca2da763853d2f6bd4248e92c"} Oct 31 00:54:14.989469 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:14.989438 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-1e5e75-l5vzz" Oct 31 00:54:15.034924 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:15.034884 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fde6a27f-2b5e-4e9e-9b39-e53c294597b7-kube-api-access-82xfn\" (UniqueName: \"kubernetes.io/projected/fde6a27f-2b5e-4e9e-9b39-e53c294597b7-kube-api-access-82xfn\") pod \"fde6a27f-2b5e-4e9e-9b39-e53c294597b7\" (UID: \"fde6a27f-2b5e-4e9e-9b39-e53c294597b7\") " Oct 31 00:54:15.035134 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:15.035022 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fde6a27f-2b5e-4e9e-9b39-e53c294597b7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fde6a27f-2b5e-4e9e-9b39-e53c294597b7-maas-api-openshift-ca\") pod \"fde6a27f-2b5e-4e9e-9b39-e53c294597b7\" (UID: \"fde6a27f-2b5e-4e9e-9b39-e53c294597b7\") " Oct 31 00:54:15.035134 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:15.035116 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fde6a27f-2b5e-4e9e-9b39-e53c294597b7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fde6a27f-2b5e-4e9e-9b39-e53c294597b7-maas-api-xks-ca\") pod \"fde6a27f-2b5e-4e9e-9b39-e53c294597b7\" (UID: \"fde6a27f-2b5e-4e9e-9b39-e53c294597b7\") " Oct 31 00:54:15.035463 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:15.035438 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fde6a27f-2b5e-4e9e-9b39-e53c294597b7-maas-api-openshift-ca" pod "fde6a27f-2b5e-4e9e-9b39-e53c294597b7" (UID: "fde6a27f-2b5e-4e9e-9b39-e53c294597b7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:54:15.037226 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:15.037191 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fde6a27f-2b5e-4e9e-9b39-e53c294597b7-kube-api-access-82xfn" pod "fde6a27f-2b5e-4e9e-9b39-e53c294597b7" (UID: "fde6a27f-2b5e-4e9e-9b39-e53c294597b7"). InnerVolumeSpecName "kube-api-access-82xfn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:54:15.037299 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:15.037246 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fde6a27f-2b5e-4e9e-9b39-e53c294597b7-maas-api-xks-ca" pod "fde6a27f-2b5e-4e9e-9b39-e53c294597b7" (UID: "fde6a27f-2b5e-4e9e-9b39-e53c294597b7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:54:15.136510 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:15.136409 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82xfn\" (UniqueName: \"kubernetes.io/projected/fde6a27f-2b5e-4e9e-9b39-e53c294597b7-kube-api-access-82xfn\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:15.136510 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:15.136448 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fde6a27f-2b5e-4e9e-9b39-e53c294597b7-maas-api-openshift-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:15.136510 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:15.136462 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fde6a27f-2b5e-4e9e-9b39-e53c294597b7-maas-api-xks-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:15.840680 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:15.840655 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-50a19d-openshift-default-68c6d7d74c-24fwv" Oct 31 00:54:15.842889 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:15.842866 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f2eb643-034b-4b9a-bd2f-6ddb034008b8-kube-api-access-vx7tp\" (UniqueName: \"kubernetes.io/projected/1f2eb643-034b-4b9a-bd2f-6ddb034008b8-kube-api-access-vx7tp\") pod \"1f2eb643-034b-4b9a-bd2f-6ddb034008b8\" (UID: \"1f2eb643-034b-4b9a-bd2f-6ddb034008b8\") " Oct 31 00:54:15.842973 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:15.842927 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f2eb643-034b-4b9a-bd2f-6ddb034008b8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f2eb643-034b-4b9a-bd2f-6ddb034008b8-istiod-ca-cert\") pod \"1f2eb643-034b-4b9a-bd2f-6ddb034008b8\" (UID: \"1f2eb643-034b-4b9a-bd2f-6ddb034008b8\") " Oct 31 00:54:15.842973 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:15.842966 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1f2eb643-034b-4b9a-bd2f-6ddb034008b8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f2eb643-034b-4b9a-bd2f-6ddb034008b8-istio-podinfo\") pod \"1f2eb643-034b-4b9a-bd2f-6ddb034008b8\" (UID: \"1f2eb643-034b-4b9a-bd2f-6ddb034008b8\") " Oct 31 00:54:15.843091 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:15.842992 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f2eb643-034b-4b9a-bd2f-6ddb034008b8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f2eb643-034b-4b9a-bd2f-6ddb034008b8-workload-certs\") pod \"1f2eb643-034b-4b9a-bd2f-6ddb034008b8\" (UID: \"1f2eb643-034b-4b9a-bd2f-6ddb034008b8\") " Oct 31 00:54:15.843146 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:15.843112 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f2eb643-034b-4b9a-bd2f-6ddb034008b8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f2eb643-034b-4b9a-bd2f-6ddb034008b8-credential-socket\") pod \"1f2eb643-034b-4b9a-bd2f-6ddb034008b8\" (UID: \"1f2eb643-034b-4b9a-bd2f-6ddb034008b8\") " Oct 31 00:54:15.843204 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:15.843147 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f2eb643-034b-4b9a-bd2f-6ddb034008b8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f2eb643-034b-4b9a-bd2f-6ddb034008b8-istio-envoy\") pod \"1f2eb643-034b-4b9a-bd2f-6ddb034008b8\" (UID: \"1f2eb643-034b-4b9a-bd2f-6ddb034008b8\") " Oct 31 00:54:15.843259 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:15.843215 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f2eb643-034b-4b9a-bd2f-6ddb034008b8-istio-token\" (UniqueName: \"kubernetes.io/projected/1f2eb643-034b-4b9a-bd2f-6ddb034008b8-istio-token\") pod \"1f2eb643-034b-4b9a-bd2f-6ddb034008b8\" (UID: \"1f2eb643-034b-4b9a-bd2f-6ddb034008b8\") " Oct 31 00:54:15.843320 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:15.843279 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f2eb643-034b-4b9a-bd2f-6ddb034008b8-workload-certs" pod "1f2eb643-034b-4b9a-bd2f-6ddb034008b8" (UID: "1f2eb643-034b-4b9a-bd2f-6ddb034008b8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:54:15.843320 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:15.843284 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f2eb643-034b-4b9a-bd2f-6ddb034008b8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f2eb643-034b-4b9a-bd2f-6ddb034008b8-workload-socket\") pod \"1f2eb643-034b-4b9a-bd2f-6ddb034008b8\" (UID: \"1f2eb643-034b-4b9a-bd2f-6ddb034008b8\") " Oct 31 00:54:15.843445 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:15.843334 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f2eb643-034b-4b9a-bd2f-6ddb034008b8-credential-socket" pod "1f2eb643-034b-4b9a-bd2f-6ddb034008b8" (UID: "1f2eb643-034b-4b9a-bd2f-6ddb034008b8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:54:15.843445 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:15.843343 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f2eb643-034b-4b9a-bd2f-6ddb034008b8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f2eb643-034b-4b9a-bd2f-6ddb034008b8-istio-data\") pod \"1f2eb643-034b-4b9a-bd2f-6ddb034008b8\" (UID: \"1f2eb643-034b-4b9a-bd2f-6ddb034008b8\") " Oct 31 00:54:15.843541 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:15.843427 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f2eb643-034b-4b9a-bd2f-6ddb034008b8-istiod-ca-cert" pod "1f2eb643-034b-4b9a-bd2f-6ddb034008b8" (UID: "1f2eb643-034b-4b9a-bd2f-6ddb034008b8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:54:15.843598 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:15.843527 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f2eb643-034b-4b9a-bd2f-6ddb034008b8-workload-socket" pod "1f2eb643-034b-4b9a-bd2f-6ddb034008b8" (UID: "1f2eb643-034b-4b9a-bd2f-6ddb034008b8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:54:15.843598 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:15.843555 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f2eb643-034b-4b9a-bd2f-6ddb034008b8-istio-data" pod "1f2eb643-034b-4b9a-bd2f-6ddb034008b8" (UID: "1f2eb643-034b-4b9a-bd2f-6ddb034008b8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:54:15.844268 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:15.844242 5222 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f2eb643-034b-4b9a-bd2f-6ddb034008b8-istiod-ca-cert\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:15.844412 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:15.844273 5222 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f2eb643-034b-4b9a-bd2f-6ddb034008b8-workload-certs\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:15.844412 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:15.844290 5222 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f2eb643-034b-4b9a-bd2f-6ddb034008b8-credential-socket\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:15.844412 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:15.844303 5222 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f2eb643-034b-4b9a-bd2f-6ddb034008b8-workload-socket\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:15.844412 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:15.844316 5222 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f2eb643-034b-4b9a-bd2f-6ddb034008b8-istio-data\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:15.845488 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:15.845453 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1f2eb643-034b-4b9a-bd2f-6ddb034008b8-istio-podinfo" pod "1f2eb643-034b-4b9a-bd2f-6ddb034008b8" (UID: "1f2eb643-034b-4b9a-bd2f-6ddb034008b8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:54:15.845584 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:15.845549 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f2eb643-034b-4b9a-bd2f-6ddb034008b8-istio-envoy" pod "1f2eb643-034b-4b9a-bd2f-6ddb034008b8" (UID: "1f2eb643-034b-4b9a-bd2f-6ddb034008b8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:54:15.845643 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:15.845626 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f2eb643-034b-4b9a-bd2f-6ddb034008b8-kube-api-access-vx7tp" pod "1f2eb643-034b-4b9a-bd2f-6ddb034008b8" (UID: "1f2eb643-034b-4b9a-bd2f-6ddb034008b8"). InnerVolumeSpecName "kube-api-access-vx7tp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:54:15.845746 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:15.845727 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f2eb643-034b-4b9a-bd2f-6ddb034008b8-istio-token" pod "1f2eb643-034b-4b9a-bd2f-6ddb034008b8" (UID: "1f2eb643-034b-4b9a-bd2f-6ddb034008b8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:54:15.852795 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:15.852769 5222 generic.go:356] "Generic (PLEG): container finished" podID="1f2eb643-034b-4b9a-bd2f-6ddb034008b8" containerID="5c11287883f26b13ebd3c0fbefb7ffe9f512dd4615040c56d0e120124faccc7b" exitCode=0 Oct 31 00:54:15.852906 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:15.852829 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-50a19d-openshift-default-68c6d7d74c-24fwv" Oct 31 00:54:15.852906 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:15.852857 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-50a19d-openshift-default-68c6d7d74c-24fwv" event={"ID":"1f2eb643-034b-4b9a-bd2f-6ddb034008b8","Type":"ContainerDied","Data":"5c11287883f26b13ebd3c0fbefb7ffe9f512dd4615040c56d0e120124faccc7b"} Oct 31 00:54:15.853011 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:15.852915 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-50a19d-openshift-default-68c6d7d74c-24fwv" event={"ID":"1f2eb643-034b-4b9a-bd2f-6ddb034008b8","Type":"ContainerDied","Data":"08ac91432afffe59486d44a26612b12c335cf6faf89eef4c1b069877f259c3f8"} Oct 31 00:54:15.853011 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:15.852939 5222 scope.go:122] "RemoveContainer" containerID="5c11287883f26b13ebd3c0fbefb7ffe9f512dd4615040c56d0e120124faccc7b" Oct 31 00:54:15.854729 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:15.854707 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-1e5e75-l5vzz" event={"ID":"fde6a27f-2b5e-4e9e-9b39-e53c294597b7","Type":"ContainerDied","Data":"4da7bd48e2ad625da3311e69959dfa64652e48b313f8ecbd49cdfacf89c34287"} Oct 31 00:54:15.854839 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:15.854748 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4da7bd48e2ad625da3311e69959dfa64652e48b313f8ecbd49cdfacf89c34287" Oct 31 00:54:15.854839 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:15.854747 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-1e5e75-l5vzz" Oct 31 00:54:15.864276 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:15.864258 5222 scope.go:122] "RemoveContainer" containerID="31c580e668900e489b539a14c43cd7cf2fc958b6bd1ddb1e866d31c62799d239" Oct 31 00:54:15.874221 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:15.874198 5222 scope.go:122] "RemoveContainer" containerID="5c11287883f26b13ebd3c0fbefb7ffe9f512dd4615040c56d0e120124faccc7b" Oct 31 00:54:15.874570 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:54:15.874547 5222 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c11287883f26b13ebd3c0fbefb7ffe9f512dd4615040c56d0e120124faccc7b\": container with ID starting with 5c11287883f26b13ebd3c0fbefb7ffe9f512dd4615040c56d0e120124faccc7b not found: ID does not exist" containerID="5c11287883f26b13ebd3c0fbefb7ffe9f512dd4615040c56d0e120124faccc7b" Oct 31 00:54:15.874648 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:15.874581 5222 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c11287883f26b13ebd3c0fbefb7ffe9f512dd4615040c56d0e120124faccc7b"} err="failed to get container status \"5c11287883f26b13ebd3c0fbefb7ffe9f512dd4615040c56d0e120124faccc7b\": rpc error: code = NotFound desc = could not find container \"5c11287883f26b13ebd3c0fbefb7ffe9f512dd4615040c56d0e120124faccc7b\": container with ID starting with 5c11287883f26b13ebd3c0fbefb7ffe9f512dd4615040c56d0e120124faccc7b not found: ID does not exist" Oct 31 00:54:15.874648 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:15.874600 5222 scope.go:122] "RemoveContainer" containerID="31c580e668900e489b539a14c43cd7cf2fc958b6bd1ddb1e866d31c62799d239" Oct 31 00:54:15.874878 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:54:15.874860 5222 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31c580e668900e489b539a14c43cd7cf2fc958b6bd1ddb1e866d31c62799d239\": container with ID starting with 31c580e668900e489b539a14c43cd7cf2fc958b6bd1ddb1e866d31c62799d239 not found: ID does not exist" containerID="31c580e668900e489b539a14c43cd7cf2fc958b6bd1ddb1e866d31c62799d239" Oct 31 00:54:15.874921 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:15.874884 5222 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31c580e668900e489b539a14c43cd7cf2fc958b6bd1ddb1e866d31c62799d239"} err="failed to get container status \"31c580e668900e489b539a14c43cd7cf2fc958b6bd1ddb1e866d31c62799d239\": rpc error: code = NotFound desc = could not find container \"31c580e668900e489b539a14c43cd7cf2fc958b6bd1ddb1e866d31c62799d239\": container with ID starting with 31c580e668900e489b539a14c43cd7cf2fc958b6bd1ddb1e866d31c62799d239 not found: ID does not exist" Oct 31 00:54:15.945902 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:15.945819 5222 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f2eb643-034b-4b9a-bd2f-6ddb034008b8-istio-podinfo\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:15.945902 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:15.945851 5222 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f2eb643-034b-4b9a-bd2f-6ddb034008b8-istio-envoy\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:15.945902 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:15.945860 5222 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f2eb643-034b-4b9a-bd2f-6ddb034008b8-istio-token\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:15.945902 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:15.945869 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vx7tp\" (UniqueName: \"kubernetes.io/projected/1f2eb643-034b-4b9a-bd2f-6ddb034008b8-kube-api-access-vx7tp\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:16.089730 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:16.089691 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-1e5e75/distinct-w3-1e5e75-kserve-7b55d58bbb-hchmb"] Oct 31 00:54:16.097298 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:16.097272 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-1e5e75/premium-w3-1e5e75-kserve-7865bd6955-55dbx"] Oct 31 00:54:16.108316 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:16.108282 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-1e5e75/simulator-w3-1e5e75-kserve-6cd984c46b-tjz8p"] Oct 31 00:54:16.137483 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:16.137412 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-50a19d-openshift-default-68c6d7d74c-24fwv"] Oct 31 00:54:16.143045 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:16.143017 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-50a19d-openshift-default-68c6d7d74c-24fwv"] Oct 31 00:54:17.656012 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:17.655976 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f2eb643-034b-4b9a-bd2f-6ddb034008b8" path="/var/lib/kubelet/pods/1f2eb643-034b-4b9a-bd2f-6ddb034008b8/volumes" Oct 31 00:54:23.132981 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:23.132944 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-1e5e75-ff6df9bb9-d888l"] Oct 31 00:54:23.728332 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:23.728293 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a6f4a5-59r9m"] Oct 31 00:54:23.728887 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:23.728872 5222 cpu_manager.go:414] "containerMap: removing container" podUID="fde6a27f-2b5e-4e9e-9b39-e53c294597b7" containerName="revoke-keys" Oct 31 00:54:23.728957 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:23.728889 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fde6a27f-2b5e-4e9e-9b39-e53c294597b7" containerName="revoke-keys" Oct 31 00:54:23.728957 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:23.728918 5222 cpu_manager.go:414] "containerMap: removing container" podUID="1f2eb643-034b-4b9a-bd2f-6ddb034008b8" containerName="istio-proxy" Oct 31 00:54:23.728957 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:23.728924 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f2eb643-034b-4b9a-bd2f-6ddb034008b8" containerName="istio-proxy" Oct 31 00:54:23.729059 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:23.729020 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f2eb643-034b-4b9a-bd2f-6ddb034008b8" containerName="istio-proxy" Oct 31 00:54:23.729059 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:23.729033 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="fde6a27f-2b5e-4e9e-9b39-e53c294597b7" containerName="revoke-keys" Oct 31 00:54:23.733686 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:23.733669 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a6f4a5-59r9m" Oct 31 00:54:23.782300 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:23.782259 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kz2tn\" (UniqueName: \"kubernetes.io/projected/fc45d98c-3404-458c-a1ab-9b1a2c376579-kube-api-access-kz2tn\") pod \"maas-api-revoke-keys-e2e-worker-w6-a6f4a5-59r9m\" (UID: \"fc45d98c-3404-458c-a1ab-9b1a2c376579\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a6f4a5-59r9m" Oct 31 00:54:23.782564 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:23.782541 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc45d98c-3404-458c-a1ab-9b1a2c376579-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-a6f4a5-59r9m\" (UID: \"fc45d98c-3404-458c-a1ab-9b1a2c376579\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a6f4a5-59r9m" Oct 31 00:54:23.782819 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:23.782797 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc45d98c-3404-458c-a1ab-9b1a2c376579-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-a6f4a5-59r9m\" (UID: \"fc45d98c-3404-458c-a1ab-9b1a2c376579\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a6f4a5-59r9m" Oct 31 00:54:23.786773 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:23.786744 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a6f4a5-59r9m"] Oct 31 00:54:23.787758 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:23.787728 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:54:23.787891 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:23.787835 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfz9z\"" Oct 31 00:54:23.787891 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:23.787852 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:54:23.787891 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:23.787861 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:54:23.883771 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:23.883725 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc45d98c-3404-458c-a1ab-9b1a2c376579-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-a6f4a5-59r9m\" (UID: \"fc45d98c-3404-458c-a1ab-9b1a2c376579\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a6f4a5-59r9m" Oct 31 00:54:23.883967 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:23.883860 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc45d98c-3404-458c-a1ab-9b1a2c376579-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-a6f4a5-59r9m\" (UID: \"fc45d98c-3404-458c-a1ab-9b1a2c376579\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a6f4a5-59r9m" Oct 31 00:54:23.883967 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:23.883944 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kz2tn\" (UniqueName: \"kubernetes.io/projected/fc45d98c-3404-458c-a1ab-9b1a2c376579-kube-api-access-kz2tn\") pod \"maas-api-revoke-keys-e2e-worker-w6-a6f4a5-59r9m\" (UID: \"fc45d98c-3404-458c-a1ab-9b1a2c376579\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a6f4a5-59r9m" Oct 31 00:54:23.884580 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:23.884557 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc45d98c-3404-458c-a1ab-9b1a2c376579-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-a6f4a5-59r9m\" (UID: \"fc45d98c-3404-458c-a1ab-9b1a2c376579\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a6f4a5-59r9m" Oct 31 00:54:23.886274 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:23.886244 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc45d98c-3404-458c-a1ab-9b1a2c376579-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-a6f4a5-59r9m\" (UID: \"fc45d98c-3404-458c-a1ab-9b1a2c376579\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a6f4a5-59r9m" Oct 31 00:54:23.946895 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:23.946861 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kz2tn\" (UniqueName: \"kubernetes.io/projected/fc45d98c-3404-458c-a1ab-9b1a2c376579-kube-api-access-kz2tn\") pod \"maas-api-revoke-keys-e2e-worker-w6-a6f4a5-59r9m\" (UID: \"fc45d98c-3404-458c-a1ab-9b1a2c376579\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a6f4a5-59r9m" Oct 31 00:54:24.048852 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:24.048813 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a6f4a5-59r9m" Oct 31 00:54:24.425734 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:54:24.425697 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc45d98c_3404_458c_a1ab_9b1a2c376579.slice/crio-9a76131c768135303beada7d69f4b942516b42ea220369b35488d5c0d6ecac78 WatchSource:0}: Error finding container 9a76131c768135303beada7d69f4b942516b42ea220369b35488d5c0d6ecac78: Status 404 returned error can't find the container with id 9a76131c768135303beada7d69f4b942516b42ea220369b35488d5c0d6ecac78 Oct 31 00:54:24.428754 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:24.428724 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a6f4a5-59r9m"] Oct 31 00:54:24.924982 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:24.924946 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a6f4a5-59r9m" event={"ID":"fc45d98c-3404-458c-a1ab-9b1a2c376579","Type":"ContainerStarted","Data":"9a76131c768135303beada7d69f4b942516b42ea220369b35488d5c0d6ecac78"} Oct 31 00:54:25.933594 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:25.933503 5222 generic.go:356] "Generic (PLEG): container finished" podID="fc45d98c-3404-458c-a1ab-9b1a2c376579" containerID="6cb653f3827b816e3cc84f0faf3bae10b779fd6abab6730760c99e1d589b5515" exitCode=0 Oct 31 00:54:25.933594 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:25.933580 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a6f4a5-59r9m" event={"ID":"fc45d98c-3404-458c-a1ab-9b1a2c376579","Type":"ContainerDied","Data":"6cb653f3827b816e3cc84f0faf3bae10b779fd6abab6730760c99e1d589b5515"} Oct 31 00:54:27.090685 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:27.090657 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a6f4a5-59r9m" Oct 31 00:54:27.123807 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:27.123768 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc45d98c-3404-458c-a1ab-9b1a2c376579-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc45d98c-3404-458c-a1ab-9b1a2c376579-maas-api-xks-ca\") pod \"fc45d98c-3404-458c-a1ab-9b1a2c376579\" (UID: \"fc45d98c-3404-458c-a1ab-9b1a2c376579\") " Oct 31 00:54:27.123978 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:27.123832 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc45d98c-3404-458c-a1ab-9b1a2c376579-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc45d98c-3404-458c-a1ab-9b1a2c376579-maas-api-openshift-ca\") pod \"fc45d98c-3404-458c-a1ab-9b1a2c376579\" (UID: \"fc45d98c-3404-458c-a1ab-9b1a2c376579\") " Oct 31 00:54:27.123978 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:27.123971 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc45d98c-3404-458c-a1ab-9b1a2c376579-kube-api-access-kz2tn\" (UniqueName: \"kubernetes.io/projected/fc45d98c-3404-458c-a1ab-9b1a2c376579-kube-api-access-kz2tn\") pod \"fc45d98c-3404-458c-a1ab-9b1a2c376579\" (UID: \"fc45d98c-3404-458c-a1ab-9b1a2c376579\") " Oct 31 00:54:27.124286 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:27.124263 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc45d98c-3404-458c-a1ab-9b1a2c376579-maas-api-openshift-ca" pod "fc45d98c-3404-458c-a1ab-9b1a2c376579" (UID: "fc45d98c-3404-458c-a1ab-9b1a2c376579"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:54:27.124440 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:27.124425 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc45d98c-3404-458c-a1ab-9b1a2c376579-maas-api-openshift-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:27.126143 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:27.126116 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc45d98c-3404-458c-a1ab-9b1a2c376579-maas-api-xks-ca" pod "fc45d98c-3404-458c-a1ab-9b1a2c376579" (UID: "fc45d98c-3404-458c-a1ab-9b1a2c376579"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:54:27.126256 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:27.126180 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc45d98c-3404-458c-a1ab-9b1a2c376579-kube-api-access-kz2tn" pod "fc45d98c-3404-458c-a1ab-9b1a2c376579" (UID: "fc45d98c-3404-458c-a1ab-9b1a2c376579"). InnerVolumeSpecName "kube-api-access-kz2tn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:54:27.225842 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:27.225757 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc45d98c-3404-458c-a1ab-9b1a2c376579-maas-api-xks-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:27.225842 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:27.225789 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kz2tn\" (UniqueName: \"kubernetes.io/projected/fc45d98c-3404-458c-a1ab-9b1a2c376579-kube-api-access-kz2tn\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:27.949955 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:27.949872 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a6f4a5-59r9m" Oct 31 00:54:27.950117 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:27.949878 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a6f4a5-59r9m" event={"ID":"fc45d98c-3404-458c-a1ab-9b1a2c376579","Type":"ContainerDied","Data":"9a76131c768135303beada7d69f4b942516b42ea220369b35488d5c0d6ecac78"} Oct 31 00:54:27.950117 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:27.950002 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9a76131c768135303beada7d69f4b942516b42ea220369b35488d5c0d6ecac78" Oct 31 00:54:28.060468 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:28.060434 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-a6f4a5/simulator-w6-a6f4a5-kserve-7769689d48-bpkc6"] Oct 31 00:54:28.133919 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:28.133877 5222 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-1e5e75-ff6df9bb9-d888l" podUID="795c26a3-f65f-40f0-9cd6-503ab681a866" containerName="payload-processing" containerID="cri-o://d5a9a484f5eb5ed7a74f93d682c027ec376ca05bf3215dc72e775878918b66f3" gracePeriod=25 Oct 31 00:54:29.883807 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:29.883761 5222 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-1e5e75-ff6df9bb9-d888l" podUID="795c26a3-f65f-40f0-9cd6-503ab681a866" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:54:31.112169 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:31.112104 5222 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-1e5e75/distinct-w3-1e5e75-kserve-7b55d58bbb-hchmb" podUID="79ffb21b-10c3-4174-89be-05ba9e852a98" containerName="main" containerID="cri-o://f419e42661b29d96499018f9421f7cb63bc92d160a55ca9fd87635f7892d5ba8" gracePeriod=45 Oct 31 00:54:31.119607 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:31.119546 5222 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-1e5e75/premium-w3-1e5e75-kserve-7865bd6955-55dbx" podUID="8f72e496-0ec8-44f4-a295-4d985d1afb92" containerName="main" containerID="cri-o://230a69db020e8a9fb62039a1b66c1a57252bb1377aa5dfbceab46d4b5be220a5" gracePeriod=45 Oct 31 00:54:31.129658 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:31.129609 5222 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-1e5e75/simulator-w3-1e5e75-kserve-6cd984c46b-tjz8p" podUID="a6158bb1-fa2f-4d00-8159-d1620958c7c0" containerName="main" containerID="cri-o://bc10714b685a61f0b635f904d557f09c68243916c086bdb8ecaaf41ebc808f1f" gracePeriod=45 Oct 31 00:54:31.413265 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:31.413232 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-1e5e75/premium-w3-1e5e75-kserve-7865bd6955-55dbx" Oct 31 00:54:31.452279 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:31.452251 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-1e5e75/simulator-w3-1e5e75-kserve-6cd984c46b-tjz8p" Oct 31 00:54:31.455866 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:31.455845 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-1e5e75/distinct-w3-1e5e75-kserve-7b55d58bbb-hchmb" Oct 31 00:54:31.492279 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:31.492247 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f72e496-0ec8-44f4-a295-4d985d1afb92-home\" (UniqueName: \"kubernetes.io/empty-dir/8f72e496-0ec8-44f4-a295-4d985d1afb92-home\") pod \"8f72e496-0ec8-44f4-a295-4d985d1afb92\" (UID: \"8f72e496-0ec8-44f4-a295-4d985d1afb92\") " Oct 31 00:54:31.492530 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:31.492510 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f72e496-0ec8-44f4-a295-4d985d1afb92-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f72e496-0ec8-44f4-a295-4d985d1afb92-model-cache\") pod \"8f72e496-0ec8-44f4-a295-4d985d1afb92\" (UID: \"8f72e496-0ec8-44f4-a295-4d985d1afb92\") " Oct 31 00:54:31.492623 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:31.492560 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f72e496-0ec8-44f4-a295-4d985d1afb92-kube-api-access-j2bg6\" (UniqueName: \"kubernetes.io/projected/8f72e496-0ec8-44f4-a295-4d985d1afb92-kube-api-access-j2bg6\") pod \"8f72e496-0ec8-44f4-a295-4d985d1afb92\" (UID: \"8f72e496-0ec8-44f4-a295-4d985d1afb92\") " Oct 31 00:54:31.492623 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:31.492590 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f72e496-0ec8-44f4-a295-4d985d1afb92-home" pod "8f72e496-0ec8-44f4-a295-4d985d1afb92" (UID: "8f72e496-0ec8-44f4-a295-4d985d1afb92"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:54:31.492747 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:31.492633 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f72e496-0ec8-44f4-a295-4d985d1afb92-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f72e496-0ec8-44f4-a295-4d985d1afb92-dshm\") pod \"8f72e496-0ec8-44f4-a295-4d985d1afb92\" (UID: \"8f72e496-0ec8-44f4-a295-4d985d1afb92\") " Oct 31 00:54:31.492747 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:31.492664 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f72e496-0ec8-44f4-a295-4d985d1afb92-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f72e496-0ec8-44f4-a295-4d985d1afb92-tmp-dir\") pod \"8f72e496-0ec8-44f4-a295-4d985d1afb92\" (UID: \"8f72e496-0ec8-44f4-a295-4d985d1afb92\") " Oct 31 00:54:31.492747 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:31.492695 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f72e496-0ec8-44f4-a295-4d985d1afb92-tls-certs\" (UniqueName: \"kubernetes.io/secret/8f72e496-0ec8-44f4-a295-4d985d1afb92-tls-certs\") pod \"8f72e496-0ec8-44f4-a295-4d985d1afb92\" (UID: \"8f72e496-0ec8-44f4-a295-4d985d1afb92\") " Oct 31 00:54:31.493241 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:31.493216 5222 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f72e496-0ec8-44f4-a295-4d985d1afb92-home\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:31.493572 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:31.493222 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f72e496-0ec8-44f4-a295-4d985d1afb92-tmp-dir" pod "8f72e496-0ec8-44f4-a295-4d985d1afb92" (UID: "8f72e496-0ec8-44f4-a295-4d985d1afb92"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:54:31.493695 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:31.493092 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f72e496-0ec8-44f4-a295-4d985d1afb92-model-cache" pod "8f72e496-0ec8-44f4-a295-4d985d1afb92" (UID: "8f72e496-0ec8-44f4-a295-4d985d1afb92"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:54:31.495290 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:31.495261 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f72e496-0ec8-44f4-a295-4d985d1afb92-dshm" pod "8f72e496-0ec8-44f4-a295-4d985d1afb92" (UID: "8f72e496-0ec8-44f4-a295-4d985d1afb92"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:54:31.495504 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:31.495483 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f72e496-0ec8-44f4-a295-4d985d1afb92-kube-api-access-j2bg6" pod "8f72e496-0ec8-44f4-a295-4d985d1afb92" (UID: "8f72e496-0ec8-44f4-a295-4d985d1afb92"). InnerVolumeSpecName "kube-api-access-j2bg6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:54:31.495633 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:31.495607 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f72e496-0ec8-44f4-a295-4d985d1afb92-tls-certs" pod "8f72e496-0ec8-44f4-a295-4d985d1afb92" (UID: "8f72e496-0ec8-44f4-a295-4d985d1afb92"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:54:31.594538 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:31.594506 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79ffb21b-10c3-4174-89be-05ba9e852a98-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79ffb21b-10c3-4174-89be-05ba9e852a98-tmp-dir\") pod \"79ffb21b-10c3-4174-89be-05ba9e852a98\" (UID: \"79ffb21b-10c3-4174-89be-05ba9e852a98\") " Oct 31 00:54:31.594733 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:31.594605 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79ffb21b-10c3-4174-89be-05ba9e852a98-dshm\" (UniqueName: \"kubernetes.io/empty-dir/79ffb21b-10c3-4174-89be-05ba9e852a98-dshm\") pod \"79ffb21b-10c3-4174-89be-05ba9e852a98\" (UID: \"79ffb21b-10c3-4174-89be-05ba9e852a98\") " Oct 31 00:54:31.594733 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:31.594639 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6158bb1-fa2f-4d00-8159-d1620958c7c0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6158bb1-fa2f-4d00-8159-d1620958c7c0-tmp-dir\") pod \"a6158bb1-fa2f-4d00-8159-d1620958c7c0\" (UID: \"a6158bb1-fa2f-4d00-8159-d1620958c7c0\") " Oct 31 00:54:31.594733 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:31.594698 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6158bb1-fa2f-4d00-8159-d1620958c7c0-kube-api-access-vdw2b\" (UniqueName: \"kubernetes.io/projected/a6158bb1-fa2f-4d00-8159-d1620958c7c0-kube-api-access-vdw2b\") pod \"a6158bb1-fa2f-4d00-8159-d1620958c7c0\" (UID: \"a6158bb1-fa2f-4d00-8159-d1620958c7c0\") " Oct 31 00:54:31.594733 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:31.594734 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79ffb21b-10c3-4174-89be-05ba9e852a98-tls-certs\" (UniqueName: \"kubernetes.io/secret/79ffb21b-10c3-4174-89be-05ba9e852a98-tls-certs\") pod \"79ffb21b-10c3-4174-89be-05ba9e852a98\" (UID: \"79ffb21b-10c3-4174-89be-05ba9e852a98\") " Oct 31 00:54:31.594987 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:31.594768 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6158bb1-fa2f-4d00-8159-d1620958c7c0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6158bb1-fa2f-4d00-8159-d1620958c7c0-model-cache\") pod \"a6158bb1-fa2f-4d00-8159-d1620958c7c0\" (UID: \"a6158bb1-fa2f-4d00-8159-d1620958c7c0\") " Oct 31 00:54:31.594987 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:31.594796 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79ffb21b-10c3-4174-89be-05ba9e852a98-kube-api-access-6blpj\" (UniqueName: \"kubernetes.io/projected/79ffb21b-10c3-4174-89be-05ba9e852a98-kube-api-access-6blpj\") pod \"79ffb21b-10c3-4174-89be-05ba9e852a98\" (UID: \"79ffb21b-10c3-4174-89be-05ba9e852a98\") " Oct 31 00:54:31.594987 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:31.594835 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6158bb1-fa2f-4d00-8159-d1620958c7c0-tls-certs\" (UniqueName: \"kubernetes.io/secret/a6158bb1-fa2f-4d00-8159-d1620958c7c0-tls-certs\") pod \"a6158bb1-fa2f-4d00-8159-d1620958c7c0\" (UID: \"a6158bb1-fa2f-4d00-8159-d1620958c7c0\") " Oct 31 00:54:31.594987 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:31.594831 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79ffb21b-10c3-4174-89be-05ba9e852a98-tmp-dir" pod "79ffb21b-10c3-4174-89be-05ba9e852a98" (UID: "79ffb21b-10c3-4174-89be-05ba9e852a98"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:54:31.594987 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:31.594907 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6158bb1-fa2f-4d00-8159-d1620958c7c0-tmp-dir" pod "a6158bb1-fa2f-4d00-8159-d1620958c7c0" (UID: "a6158bb1-fa2f-4d00-8159-d1620958c7c0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:54:31.594987 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:31.594918 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6158bb1-fa2f-4d00-8159-d1620958c7c0-home\" (UniqueName: \"kubernetes.io/empty-dir/a6158bb1-fa2f-4d00-8159-d1620958c7c0-home\") pod \"a6158bb1-fa2f-4d00-8159-d1620958c7c0\" (UID: \"a6158bb1-fa2f-4d00-8159-d1620958c7c0\") " Oct 31 00:54:31.595312 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:31.595005 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6158bb1-fa2f-4d00-8159-d1620958c7c0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6158bb1-fa2f-4d00-8159-d1620958c7c0-dshm\") pod \"a6158bb1-fa2f-4d00-8159-d1620958c7c0\" (UID: \"a6158bb1-fa2f-4d00-8159-d1620958c7c0\") " Oct 31 00:54:31.595312 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:31.595087 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79ffb21b-10c3-4174-89be-05ba9e852a98-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79ffb21b-10c3-4174-89be-05ba9e852a98-model-cache\") pod \"79ffb21b-10c3-4174-89be-05ba9e852a98\" (UID: \"79ffb21b-10c3-4174-89be-05ba9e852a98\") " Oct 31 00:54:31.595632 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:31.595083 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6158bb1-fa2f-4d00-8159-d1620958c7c0-model-cache" pod "a6158bb1-fa2f-4d00-8159-d1620958c7c0" (UID: "a6158bb1-fa2f-4d00-8159-d1620958c7c0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:54:31.595887 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:31.595103 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6158bb1-fa2f-4d00-8159-d1620958c7c0-home" pod "a6158bb1-fa2f-4d00-8159-d1620958c7c0" (UID: "a6158bb1-fa2f-4d00-8159-d1620958c7c0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:54:31.595887 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:31.595505 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79ffb21b-10c3-4174-89be-05ba9e852a98-model-cache" pod "79ffb21b-10c3-4174-89be-05ba9e852a98" (UID: "79ffb21b-10c3-4174-89be-05ba9e852a98"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:54:31.595887 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:31.595605 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79ffb21b-10c3-4174-89be-05ba9e852a98-home\" (UniqueName: \"kubernetes.io/empty-dir/79ffb21b-10c3-4174-89be-05ba9e852a98-home\") pod \"79ffb21b-10c3-4174-89be-05ba9e852a98\" (UID: \"79ffb21b-10c3-4174-89be-05ba9e852a98\") " Oct 31 00:54:31.595887 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:31.595809 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79ffb21b-10c3-4174-89be-05ba9e852a98-home" pod "79ffb21b-10c3-4174-89be-05ba9e852a98" (UID: "79ffb21b-10c3-4174-89be-05ba9e852a98"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:54:31.596307 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:31.596275 5222 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6158bb1-fa2f-4d00-8159-d1620958c7c0-model-cache\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:31.596519 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:31.596503 5222 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6158bb1-fa2f-4d00-8159-d1620958c7c0-home\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:31.596603 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:31.596525 5222 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79ffb21b-10c3-4174-89be-05ba9e852a98-model-cache\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:31.596603 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:31.596539 5222 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79ffb21b-10c3-4174-89be-05ba9e852a98-home\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:31.596603 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:31.596552 5222 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f72e496-0ec8-44f4-a295-4d985d1afb92-model-cache\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:31.596603 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:31.596566 5222 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79ffb21b-10c3-4174-89be-05ba9e852a98-tmp-dir\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:31.596603 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:31.596583 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2bg6\" (UniqueName: \"kubernetes.io/projected/8f72e496-0ec8-44f4-a295-4d985d1afb92-kube-api-access-j2bg6\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:31.596603 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:31.596596 5222 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f72e496-0ec8-44f4-a295-4d985d1afb92-dshm\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:31.596929 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:31.596611 5222 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f72e496-0ec8-44f4-a295-4d985d1afb92-tmp-dir\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:31.596929 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:31.596625 5222 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f72e496-0ec8-44f4-a295-4d985d1afb92-tls-certs\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:31.596929 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:31.596640 5222 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6158bb1-fa2f-4d00-8159-d1620958c7c0-tmp-dir\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:31.597614 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:31.597592 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6158bb1-fa2f-4d00-8159-d1620958c7c0-dshm" pod "a6158bb1-fa2f-4d00-8159-d1620958c7c0" (UID: "a6158bb1-fa2f-4d00-8159-d1620958c7c0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:54:31.597801 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:31.597780 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79ffb21b-10c3-4174-89be-05ba9e852a98-tls-certs" pod "79ffb21b-10c3-4174-89be-05ba9e852a98" (UID: "79ffb21b-10c3-4174-89be-05ba9e852a98"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:54:31.597854 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:31.597784 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79ffb21b-10c3-4174-89be-05ba9e852a98-dshm" pod "79ffb21b-10c3-4174-89be-05ba9e852a98" (UID: "79ffb21b-10c3-4174-89be-05ba9e852a98"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:54:31.598768 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:31.598743 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6158bb1-fa2f-4d00-8159-d1620958c7c0-kube-api-access-vdw2b" pod "a6158bb1-fa2f-4d00-8159-d1620958c7c0" (UID: "a6158bb1-fa2f-4d00-8159-d1620958c7c0"). InnerVolumeSpecName "kube-api-access-vdw2b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:54:31.598768 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:31.598753 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6158bb1-fa2f-4d00-8159-d1620958c7c0-tls-certs" pod "a6158bb1-fa2f-4d00-8159-d1620958c7c0" (UID: "a6158bb1-fa2f-4d00-8159-d1620958c7c0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:54:31.599269 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:31.599253 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79ffb21b-10c3-4174-89be-05ba9e852a98-kube-api-access-6blpj" pod "79ffb21b-10c3-4174-89be-05ba9e852a98" (UID: "79ffb21b-10c3-4174-89be-05ba9e852a98"). InnerVolumeSpecName "kube-api-access-6blpj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:54:31.700726 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:31.700685 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdw2b\" (UniqueName: \"kubernetes.io/projected/a6158bb1-fa2f-4d00-8159-d1620958c7c0-kube-api-access-vdw2b\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:31.700726 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:31.700728 5222 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79ffb21b-10c3-4174-89be-05ba9e852a98-tls-certs\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:31.700981 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:31.700744 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6blpj\" (UniqueName: \"kubernetes.io/projected/79ffb21b-10c3-4174-89be-05ba9e852a98-kube-api-access-6blpj\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:31.700981 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:31.700760 5222 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6158bb1-fa2f-4d00-8159-d1620958c7c0-tls-certs\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:31.700981 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:31.700775 5222 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6158bb1-fa2f-4d00-8159-d1620958c7c0-dshm\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:31.700981 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:31.700788 5222 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79ffb21b-10c3-4174-89be-05ba9e852a98-dshm\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:32.016951 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:32.016021 5222 generic.go:356] "Generic (PLEG): container finished" podID="a6158bb1-fa2f-4d00-8159-d1620958c7c0" containerID="bc10714b685a61f0b635f904d557f09c68243916c086bdb8ecaaf41ebc808f1f" exitCode=0 Oct 31 00:54:32.016951 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:32.016284 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-1e5e75/simulator-w3-1e5e75-kserve-6cd984c46b-tjz8p" event={"ID":"a6158bb1-fa2f-4d00-8159-d1620958c7c0","Type":"ContainerDied","Data":"bc10714b685a61f0b635f904d557f09c68243916c086bdb8ecaaf41ebc808f1f"} Oct 31 00:54:32.016951 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:32.016332 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-1e5e75/simulator-w3-1e5e75-kserve-6cd984c46b-tjz8p" event={"ID":"a6158bb1-fa2f-4d00-8159-d1620958c7c0","Type":"ContainerDied","Data":"a2401b91de8f7e3fc737242074ecc205e2db0cde73808518f890a1c3256bd21d"} Oct 31 00:54:32.016951 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:32.016547 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-1e5e75/simulator-w3-1e5e75-kserve-6cd984c46b-tjz8p" Oct 31 00:54:32.025045 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:32.025013 5222 scope.go:122] "RemoveContainer" containerID="bc10714b685a61f0b635f904d557f09c68243916c086bdb8ecaaf41ebc808f1f" Oct 31 00:54:32.033611 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:32.033573 5222 generic.go:356] "Generic (PLEG): container finished" podID="8f72e496-0ec8-44f4-a295-4d985d1afb92" containerID="230a69db020e8a9fb62039a1b66c1a57252bb1377aa5dfbceab46d4b5be220a5" exitCode=0 Oct 31 00:54:32.033799 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:32.033687 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-1e5e75/premium-w3-1e5e75-kserve-7865bd6955-55dbx" event={"ID":"8f72e496-0ec8-44f4-a295-4d985d1afb92","Type":"ContainerDied","Data":"230a69db020e8a9fb62039a1b66c1a57252bb1377aa5dfbceab46d4b5be220a5"} Oct 31 00:54:32.033799 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:32.033724 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-1e5e75/premium-w3-1e5e75-kserve-7865bd6955-55dbx" event={"ID":"8f72e496-0ec8-44f4-a295-4d985d1afb92","Type":"ContainerDied","Data":"033b4a77e7e8bef6e4060a0a0ec85886470509a8fc0fc5bbf4fe44859ec8f023"} Oct 31 00:54:32.040946 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:32.037571 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-1e5e75/premium-w3-1e5e75-kserve-7865bd6955-55dbx" Oct 31 00:54:32.067010 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:32.066974 5222 generic.go:356] "Generic (PLEG): container finished" podID="79ffb21b-10c3-4174-89be-05ba9e852a98" containerID="f419e42661b29d96499018f9421f7cb63bc92d160a55ca9fd87635f7892d5ba8" exitCode=0 Oct 31 00:54:32.067316 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:32.067024 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-1e5e75/distinct-w3-1e5e75-kserve-7b55d58bbb-hchmb" Oct 31 00:54:32.067618 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:32.067046 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-1e5e75/distinct-w3-1e5e75-kserve-7b55d58bbb-hchmb" event={"ID":"79ffb21b-10c3-4174-89be-05ba9e852a98","Type":"ContainerDied","Data":"f419e42661b29d96499018f9421f7cb63bc92d160a55ca9fd87635f7892d5ba8"} Oct 31 00:54:32.067726 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:32.067649 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-1e5e75/distinct-w3-1e5e75-kserve-7b55d58bbb-hchmb" event={"ID":"79ffb21b-10c3-4174-89be-05ba9e852a98","Type":"ContainerDied","Data":"a39bf91a09bf8326ddb64e46e0cc3afabd1edd214d0a415b941bb924d0f96567"} Oct 31 00:54:32.073742 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:32.073718 5222 scope.go:122] "RemoveContainer" containerID="bc10714b685a61f0b635f904d557f09c68243916c086bdb8ecaaf41ebc808f1f" Oct 31 00:54:32.074414 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:54:32.074386 5222 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc10714b685a61f0b635f904d557f09c68243916c086bdb8ecaaf41ebc808f1f\": container with ID starting with bc10714b685a61f0b635f904d557f09c68243916c086bdb8ecaaf41ebc808f1f not found: ID does not exist" containerID="bc10714b685a61f0b635f904d557f09c68243916c086bdb8ecaaf41ebc808f1f" Oct 31 00:54:32.074587 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:32.074560 5222 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc10714b685a61f0b635f904d557f09c68243916c086bdb8ecaaf41ebc808f1f"} err="failed to get container status \"bc10714b685a61f0b635f904d557f09c68243916c086bdb8ecaaf41ebc808f1f\": rpc error: code = NotFound desc = could not find container \"bc10714b685a61f0b635f904d557f09c68243916c086bdb8ecaaf41ebc808f1f\": container with ID starting with bc10714b685a61f0b635f904d557f09c68243916c086bdb8ecaaf41ebc808f1f not found: ID does not exist" Oct 31 00:54:32.074725 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:32.074713 5222 scope.go:122] "RemoveContainer" containerID="230a69db020e8a9fb62039a1b66c1a57252bb1377aa5dfbceab46d4b5be220a5" Oct 31 00:54:32.088848 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:32.088824 5222 scope.go:122] "RemoveContainer" containerID="230a69db020e8a9fb62039a1b66c1a57252bb1377aa5dfbceab46d4b5be220a5" Oct 31 00:54:32.089233 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:54:32.089201 5222 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"230a69db020e8a9fb62039a1b66c1a57252bb1377aa5dfbceab46d4b5be220a5\": container with ID starting with 230a69db020e8a9fb62039a1b66c1a57252bb1377aa5dfbceab46d4b5be220a5 not found: ID does not exist" containerID="230a69db020e8a9fb62039a1b66c1a57252bb1377aa5dfbceab46d4b5be220a5" Oct 31 00:54:32.089337 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:32.089241 5222 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"230a69db020e8a9fb62039a1b66c1a57252bb1377aa5dfbceab46d4b5be220a5"} err="failed to get container status \"230a69db020e8a9fb62039a1b66c1a57252bb1377aa5dfbceab46d4b5be220a5\": rpc error: code = NotFound desc = could not find container \"230a69db020e8a9fb62039a1b66c1a57252bb1377aa5dfbceab46d4b5be220a5\": container with ID starting with 230a69db020e8a9fb62039a1b66c1a57252bb1377aa5dfbceab46d4b5be220a5 not found: ID does not exist" Oct 31 00:54:32.089337 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:32.089272 5222 scope.go:122] "RemoveContainer" containerID="f419e42661b29d96499018f9421f7cb63bc92d160a55ca9fd87635f7892d5ba8" Oct 31 00:54:32.101948 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:32.101899 5222 scope.go:122] "RemoveContainer" containerID="f419e42661b29d96499018f9421f7cb63bc92d160a55ca9fd87635f7892d5ba8" Oct 31 00:54:32.102329 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:54:32.102297 5222 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f419e42661b29d96499018f9421f7cb63bc92d160a55ca9fd87635f7892d5ba8\": container with ID starting with f419e42661b29d96499018f9421f7cb63bc92d160a55ca9fd87635f7892d5ba8 not found: ID does not exist" containerID="f419e42661b29d96499018f9421f7cb63bc92d160a55ca9fd87635f7892d5ba8" Oct 31 00:54:32.102513 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:32.102339 5222 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f419e42661b29d96499018f9421f7cb63bc92d160a55ca9fd87635f7892d5ba8"} err="failed to get container status \"f419e42661b29d96499018f9421f7cb63bc92d160a55ca9fd87635f7892d5ba8\": rpc error: code = NotFound desc = could not find container \"f419e42661b29d96499018f9421f7cb63bc92d160a55ca9fd87635f7892d5ba8\": container with ID starting with f419e42661b29d96499018f9421f7cb63bc92d160a55ca9fd87635f7892d5ba8 not found: ID does not exist" Oct 31 00:54:32.189127 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:32.189088 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-1e5e75/premium-w3-1e5e75-kserve-7865bd6955-55dbx"] Oct 31 00:54:32.238387 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:32.238347 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-1e5e75/premium-w3-1e5e75-kserve-7865bd6955-55dbx"] Oct 31 00:54:32.400184 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:32.400095 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-1e5e75/simulator-w3-1e5e75-kserve-6cd984c46b-tjz8p"] Oct 31 00:54:32.416098 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:32.416044 5222 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1e5e75/simulator-w3-1e5e75-kserve-6cd984c46b-tjz8p" podUID="a6158bb1-fa2f-4d00-8159-d1620958c7c0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Oct 31 00:54:32.420579 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:32.420547 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-1e5e75/simulator-w3-1e5e75-kserve-6cd984c46b-tjz8p"] Oct 31 00:54:32.556134 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:32.556099 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-1e5e75/distinct-w3-1e5e75-kserve-7b55d58bbb-hchmb"] Oct 31 00:54:32.561394 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:32.560061 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-1e5e75/distinct-w3-1e5e75-kserve-7b55d58bbb-hchmb"] Oct 31 00:54:33.688395 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:33.682511 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79ffb21b-10c3-4174-89be-05ba9e852a98" path="/var/lib/kubelet/pods/79ffb21b-10c3-4174-89be-05ba9e852a98/volumes" Oct 31 00:54:33.688395 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:33.683191 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f72e496-0ec8-44f4-a295-4d985d1afb92" path="/var/lib/kubelet/pods/8f72e496-0ec8-44f4-a295-4d985d1afb92/volumes" Oct 31 00:54:33.688395 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:33.687256 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6158bb1-fa2f-4d00-8159-d1620958c7c0" path="/var/lib/kubelet/pods/a6158bb1-fa2f-4d00-8159-d1620958c7c0/volumes" Oct 31 00:54:35.305718 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:35.305682 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-a6f4a5-7b4d7b4dcc-v2hts"] Oct 31 00:54:35.418816 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:35.418781 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-a6f4a5-7666bb86c-glmj2"] Oct 31 00:54:35.890407 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:54:35.890353 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 00:54:39.532994 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:39.532955 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-432acaff-l87vl"] Oct 31 00:54:39.533554 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:39.533539 5222 cpu_manager.go:414] "containerMap: removing container" podUID="fc45d98c-3404-458c-a1ab-9b1a2c376579" containerName="revoke-keys" Oct 31 00:54:39.533554 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:39.533553 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc45d98c-3404-458c-a1ab-9b1a2c376579" containerName="revoke-keys" Oct 31 00:54:39.533649 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:39.533579 5222 cpu_manager.go:414] "containerMap: removing container" podUID="a6158bb1-fa2f-4d00-8159-d1620958c7c0" containerName="main" Oct 31 00:54:39.533649 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:39.533583 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6158bb1-fa2f-4d00-8159-d1620958c7c0" containerName="main" Oct 31 00:54:39.533649 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:39.533593 5222 cpu_manager.go:414] "containerMap: removing container" podUID="79ffb21b-10c3-4174-89be-05ba9e852a98" containerName="main" Oct 31 00:54:39.533649 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:39.533598 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79ffb21b-10c3-4174-89be-05ba9e852a98" containerName="main" Oct 31 00:54:39.533649 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:39.533606 5222 cpu_manager.go:414] "containerMap: removing container" podUID="1f2eb643-034b-4b9a-bd2f-6ddb034008b8" containerName="istio-proxy" Oct 31 00:54:39.533649 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:39.533611 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f2eb643-034b-4b9a-bd2f-6ddb034008b8" containerName="istio-proxy" Oct 31 00:54:39.533649 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:39.533625 5222 cpu_manager.go:414] "containerMap: removing container" podUID="8f72e496-0ec8-44f4-a295-4d985d1afb92" containerName="main" Oct 31 00:54:39.533649 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:39.533630 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f72e496-0ec8-44f4-a295-4d985d1afb92" containerName="main" Oct 31 00:54:39.533909 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:39.533713 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6158bb1-fa2f-4d00-8159-d1620958c7c0" containerName="main" Oct 31 00:54:39.533909 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:39.533720 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="79ffb21b-10c3-4174-89be-05ba9e852a98" containerName="main" Oct 31 00:54:39.533909 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:39.533728 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f2eb643-034b-4b9a-bd2f-6ddb034008b8" containerName="istio-proxy" Oct 31 00:54:39.533909 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:39.533735 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f72e496-0ec8-44f4-a295-4d985d1afb92" containerName="main" Oct 31 00:54:39.533909 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:39.533744 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc45d98c-3404-458c-a1ab-9b1a2c376579" containerName="revoke-keys" Oct 31 00:54:39.538650 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:39.538625 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-432acaff-l87vl" Oct 31 00:54:39.590063 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:39.590026 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-432acaff-l87vl"] Oct 31 00:54:39.609431 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:39.599387 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9a35539-tvbkt"] Oct 31 00:54:39.609431 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:39.600962 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ce45316-6e99-4cf1-83b4-b24f3822a4f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-432acaff-l87vl\" (UID: \"4ce45316-6e99-4cf1-83b4-b24f3822a4f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-432acaff-l87vl" Oct 31 00:54:39.609431 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:39.601260 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mjsk\" (UniqueName: \"kubernetes.io/projected/4ce45316-6e99-4cf1-83b4-b24f3822a4f2-kube-api-access-4mjsk\") pod \"maas-api-revoke-sub-432acaff-l87vl\" (UID: \"4ce45316-6e99-4cf1-83b4-b24f3822a4f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-432acaff-l87vl" Oct 31 00:54:39.609431 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:39.601426 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ce45316-6e99-4cf1-83b4-b24f3822a4f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-432acaff-l87vl\" (UID: \"4ce45316-6e99-4cf1-83b4-b24f3822a4f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-432acaff-l87vl" Oct 31 00:54:39.611124 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:39.611094 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9a35539-tvbkt" Oct 31 00:54:39.638402 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:39.638353 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfz9z\"" Oct 31 00:54:39.640820 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:39.640807 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:54:39.640889 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:39.640815 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:54:39.690256 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:39.690226 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9a35539-tvbkt"] Oct 31 00:54:39.692880 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:39.692862 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:54:39.704546 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:39.704514 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef06038e-a7b5-4d47-a9a7-32129308ccc5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c9a35539-tvbkt\" (UID: \"ef06038e-a7b5-4d47-a9a7-32129308ccc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9a35539-tvbkt" Oct 31 00:54:39.704753 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:39.704575 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ce45316-6e99-4cf1-83b4-b24f3822a4f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-432acaff-l87vl\" (UID: \"4ce45316-6e99-4cf1-83b4-b24f3822a4f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-432acaff-l87vl" Oct 31 00:54:39.704822 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:39.704746 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ce45316-6e99-4cf1-83b4-b24f3822a4f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-432acaff-l87vl\" (UID: \"4ce45316-6e99-4cf1-83b4-b24f3822a4f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-432acaff-l87vl" Oct 31 00:54:39.704880 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:39.704839 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlj6z\" (UniqueName: \"kubernetes.io/projected/ef06038e-a7b5-4d47-a9a7-32129308ccc5-kube-api-access-jlj6z\") pod \"maas-api-revoke-sub-c9a35539-tvbkt\" (UID: \"ef06038e-a7b5-4d47-a9a7-32129308ccc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9a35539-tvbkt" Oct 31 00:54:39.704961 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:39.704942 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mjsk\" (UniqueName: \"kubernetes.io/projected/4ce45316-6e99-4cf1-83b4-b24f3822a4f2-kube-api-access-4mjsk\") pod \"maas-api-revoke-sub-432acaff-l87vl\" (UID: \"4ce45316-6e99-4cf1-83b4-b24f3822a4f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-432acaff-l87vl" Oct 31 00:54:39.705023 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:39.704979 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef06038e-a7b5-4d47-a9a7-32129308ccc5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c9a35539-tvbkt\" (UID: \"ef06038e-a7b5-4d47-a9a7-32129308ccc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9a35539-tvbkt" Oct 31 00:54:39.706509 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:39.706481 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ce45316-6e99-4cf1-83b4-b24f3822a4f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-432acaff-l87vl\" (UID: \"4ce45316-6e99-4cf1-83b4-b24f3822a4f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-432acaff-l87vl" Oct 31 00:54:39.707706 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:39.707683 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ce45316-6e99-4cf1-83b4-b24f3822a4f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-432acaff-l87vl\" (UID: \"4ce45316-6e99-4cf1-83b4-b24f3822a4f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-432acaff-l87vl" Oct 31 00:54:39.767916 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:39.767879 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mjsk\" (UniqueName: \"kubernetes.io/projected/4ce45316-6e99-4cf1-83b4-b24f3822a4f2-kube-api-access-4mjsk\") pod \"maas-api-revoke-sub-432acaff-l87vl\" (UID: \"4ce45316-6e99-4cf1-83b4-b24f3822a4f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-432acaff-l87vl" Oct 31 00:54:39.806257 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:39.806176 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef06038e-a7b5-4d47-a9a7-32129308ccc5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c9a35539-tvbkt\" (UID: \"ef06038e-a7b5-4d47-a9a7-32129308ccc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9a35539-tvbkt" Oct 31 00:54:39.806460 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:39.806286 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlj6z\" (UniqueName: \"kubernetes.io/projected/ef06038e-a7b5-4d47-a9a7-32129308ccc5-kube-api-access-jlj6z\") pod \"maas-api-revoke-sub-c9a35539-tvbkt\" (UID: \"ef06038e-a7b5-4d47-a9a7-32129308ccc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9a35539-tvbkt" Oct 31 00:54:39.806460 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:39.806327 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef06038e-a7b5-4d47-a9a7-32129308ccc5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c9a35539-tvbkt\" (UID: \"ef06038e-a7b5-4d47-a9a7-32129308ccc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9a35539-tvbkt" Oct 31 00:54:39.806927 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:39.806905 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef06038e-a7b5-4d47-a9a7-32129308ccc5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c9a35539-tvbkt\" (UID: \"ef06038e-a7b5-4d47-a9a7-32129308ccc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9a35539-tvbkt" Oct 31 00:54:39.808887 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:39.808866 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef06038e-a7b5-4d47-a9a7-32129308ccc5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c9a35539-tvbkt\" (UID: \"ef06038e-a7b5-4d47-a9a7-32129308ccc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9a35539-tvbkt" Oct 31 00:54:39.852410 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:39.852358 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-432acaff-l87vl" Oct 31 00:54:39.868810 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:39.868778 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlj6z\" (UniqueName: \"kubernetes.io/projected/ef06038e-a7b5-4d47-a9a7-32129308ccc5-kube-api-access-jlj6z\") pod \"maas-api-revoke-sub-c9a35539-tvbkt\" (UID: \"ef06038e-a7b5-4d47-a9a7-32129308ccc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9a35539-tvbkt" Oct 31 00:54:39.868945 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:39.868859 5222 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-1e5e75-ff6df9bb9-d888l" podUID="795c26a3-f65f-40f0-9cd6-503ab681a866" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:54:39.929113 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:39.929081 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9a35539-tvbkt" Oct 31 00:54:40.229315 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:54:40.229277 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ce45316_6e99_4cf1_83b4_b24f3822a4f2.slice/crio-7f3641945fe9e073f5c8a73b28a0d14187b64f1a896876ec6944713cf656bfae WatchSource:0}: Error finding container 7f3641945fe9e073f5c8a73b28a0d14187b64f1a896876ec6944713cf656bfae: Status 404 returned error can't find the container with id 7f3641945fe9e073f5c8a73b28a0d14187b64f1a896876ec6944713cf656bfae Oct 31 00:54:40.232109 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:40.231959 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-432acaff-l87vl"] Oct 31 00:54:40.306783 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:40.306712 5222 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-a6f4a5-7b4d7b4dcc-v2hts" podUID="37865b42-ea0f-4744-ab63-9dbc9b7f74b1" containerName="payload-processing" containerID="cri-o://0c2a1b91b0df790654027c7e739688f777a630888871a77c41e3fa155ef1407f" gracePeriod=25 Oct 31 00:54:40.339054 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:40.339018 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9a35539-tvbkt"] Oct 31 00:54:40.342895 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:54:40.342863 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef06038e_a7b5_4d47_a9a7_32129308ccc5.slice/crio-cfb6475900a3d1f56a0e3ecd0810b0b3c3e7ab8340a24965fa57d957f3d1aa89 WatchSource:0}: Error finding container cfb6475900a3d1f56a0e3ecd0810b0b3c3e7ab8340a24965fa57d957f3d1aa89: Status 404 returned error can't find the container with id cfb6475900a3d1f56a0e3ecd0810b0b3c3e7ab8340a24965fa57d957f3d1aa89 Oct 31 00:54:40.419867 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:40.419818 5222 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a6f4a5-7666bb86c-glmj2" podUID="91cb96ab-e55e-463f-9fe0-059e65184a2c" containerName="payload-pre-processing" containerID="cri-o://1383fa36b66f6c16b4a2c7b462b5504bf6c56f6de4fa8d25b95849d2ff1a4800" gracePeriod=25 Oct 31 00:54:41.171478 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:41.171431 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9a35539-tvbkt" event={"ID":"ef06038e-a7b5-4d47-a9a7-32129308ccc5","Type":"ContainerStarted","Data":"d3315a0346a141c03cca81210f6cad2a30129a54ea8319f656e9b6bad015926b"} Oct 31 00:54:41.171941 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:41.171508 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9a35539-tvbkt" event={"ID":"ef06038e-a7b5-4d47-a9a7-32129308ccc5","Type":"ContainerStarted","Data":"cfb6475900a3d1f56a0e3ecd0810b0b3c3e7ab8340a24965fa57d957f3d1aa89"} Oct 31 00:54:41.173060 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:41.173034 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-432acaff-l87vl" event={"ID":"4ce45316-6e99-4cf1-83b4-b24f3822a4f2","Type":"ContainerStarted","Data":"97d4dbc0448918ee3baa3b83612217f0bfbb7521c5699506f523ae1c9bed5db8"} Oct 31 00:54:41.173187 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:41.173081 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-432acaff-l87vl" event={"ID":"4ce45316-6e99-4cf1-83b4-b24f3822a4f2","Type":"ContainerStarted","Data":"7f3641945fe9e073f5c8a73b28a0d14187b64f1a896876ec6944713cf656bfae"} Oct 31 00:54:41.293911 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:41.293833 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9a35539-tvbkt" podStartSLOduration=2.293814069 podStartE2EDuration="2.293814069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:54:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:54:41.290513408 +0000 UTC m=+2373.761683157" watchObservedRunningTime="2026-10-31 00:54:41.293814069 +0000 UTC m=+2373.764983819" Oct 31 00:54:41.409943 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:41.409886 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-432acaff-l87vl" podStartSLOduration=2.4098719060000002 podStartE2EDuration="2.409871906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:54:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:54:41.409057198 +0000 UTC m=+2373.880226949" watchObservedRunningTime="2026-10-31 00:54:41.409871906 +0000 UTC m=+2373.881041655" Oct 31 00:54:41.888157 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:41.888121 5222 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-a6f4a5-7b4d7b4dcc-v2hts" podUID="37865b42-ea0f-4744-ab63-9dbc9b7f74b1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:54:41.888385 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:41.888129 5222 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a6f4a5-7666bb86c-glmj2" podUID="91cb96ab-e55e-463f-9fe0-059e65184a2c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:54:42.181771 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:42.181668 5222 generic.go:356] "Generic (PLEG): container finished" podID="ef06038e-a7b5-4d47-a9a7-32129308ccc5" containerID="d3315a0346a141c03cca81210f6cad2a30129a54ea8319f656e9b6bad015926b" exitCode=0 Oct 31 00:54:42.181771 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:42.181761 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9a35539-tvbkt" event={"ID":"ef06038e-a7b5-4d47-a9a7-32129308ccc5","Type":"ContainerDied","Data":"d3315a0346a141c03cca81210f6cad2a30129a54ea8319f656e9b6bad015926b"} Oct 31 00:54:42.183535 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:42.183506 5222 generic.go:356] "Generic (PLEG): container finished" podID="4ce45316-6e99-4cf1-83b4-b24f3822a4f2" containerID="97d4dbc0448918ee3baa3b83612217f0bfbb7521c5699506f523ae1c9bed5db8" exitCode=0 Oct 31 00:54:42.183689 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:42.183595 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-432acaff-l87vl" event={"ID":"4ce45316-6e99-4cf1-83b4-b24f3822a4f2","Type":"ContainerDied","Data":"97d4dbc0448918ee3baa3b83612217f0bfbb7521c5699506f523ae1c9bed5db8"} Oct 31 00:54:43.083262 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:43.083201 5222 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-a6f4a5/simulator-w6-a6f4a5-kserve-7769689d48-bpkc6" podUID="24079bcb-658a-4109-9b14-a167a38cfac7" containerName="main" containerID="cri-o://98c8701665839fb9b5694c7d1cb27b0fe5491f47e419c2d4032a60210b5adf5f" gracePeriod=45 Oct 31 00:54:43.366591 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:43.366561 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9a35539-tvbkt" Oct 31 00:54:43.415737 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:43.415709 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-432acaff-l87vl" Oct 31 00:54:43.419173 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:43.419148 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-a6f4a5/simulator-w6-a6f4a5-kserve-7769689d48-bpkc6" Oct 31 00:54:43.456060 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:43.454936 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ce45316-6e99-4cf1-83b4-b24f3822a4f2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ce45316-6e99-4cf1-83b4-b24f3822a4f2-maas-api-xks-ca\") pod \"4ce45316-6e99-4cf1-83b4-b24f3822a4f2\" (UID: \"4ce45316-6e99-4cf1-83b4-b24f3822a4f2\") " Oct 31 00:54:43.456060 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:43.455015 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ce45316-6e99-4cf1-83b4-b24f3822a4f2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ce45316-6e99-4cf1-83b4-b24f3822a4f2-maas-api-openshift-ca\") pod \"4ce45316-6e99-4cf1-83b4-b24f3822a4f2\" (UID: \"4ce45316-6e99-4cf1-83b4-b24f3822a4f2\") " Oct 31 00:54:43.456060 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:43.455068 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ce45316-6e99-4cf1-83b4-b24f3822a4f2-kube-api-access-4mjsk\" (UniqueName: \"kubernetes.io/projected/4ce45316-6e99-4cf1-83b4-b24f3822a4f2-kube-api-access-4mjsk\") pod \"4ce45316-6e99-4cf1-83b4-b24f3822a4f2\" (UID: \"4ce45316-6e99-4cf1-83b4-b24f3822a4f2\") " Oct 31 00:54:43.456060 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:43.455177 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef06038e-a7b5-4d47-a9a7-32129308ccc5-kube-api-access-jlj6z\" (UniqueName: \"kubernetes.io/projected/ef06038e-a7b5-4d47-a9a7-32129308ccc5-kube-api-access-jlj6z\") pod \"ef06038e-a7b5-4d47-a9a7-32129308ccc5\" (UID: \"ef06038e-a7b5-4d47-a9a7-32129308ccc5\") " Oct 31 00:54:43.456060 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:43.455245 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef06038e-a7b5-4d47-a9a7-32129308ccc5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef06038e-a7b5-4d47-a9a7-32129308ccc5-maas-api-xks-ca\") pod \"ef06038e-a7b5-4d47-a9a7-32129308ccc5\" (UID: \"ef06038e-a7b5-4d47-a9a7-32129308ccc5\") " Oct 31 00:54:43.456060 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:43.455355 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef06038e-a7b5-4d47-a9a7-32129308ccc5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef06038e-a7b5-4d47-a9a7-32129308ccc5-maas-api-openshift-ca\") pod \"ef06038e-a7b5-4d47-a9a7-32129308ccc5\" (UID: \"ef06038e-a7b5-4d47-a9a7-32129308ccc5\") " Oct 31 00:54:43.456600 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:43.455995 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ce45316-6e99-4cf1-83b4-b24f3822a4f2-maas-api-openshift-ca" pod "4ce45316-6e99-4cf1-83b4-b24f3822a4f2" (UID: "4ce45316-6e99-4cf1-83b4-b24f3822a4f2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:54:43.456924 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:43.456888 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef06038e-a7b5-4d47-a9a7-32129308ccc5-maas-api-openshift-ca" pod "ef06038e-a7b5-4d47-a9a7-32129308ccc5" (UID: "ef06038e-a7b5-4d47-a9a7-32129308ccc5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:54:43.458260 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:43.458233 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ce45316-6e99-4cf1-83b4-b24f3822a4f2-maas-api-xks-ca" pod "4ce45316-6e99-4cf1-83b4-b24f3822a4f2" (UID: "4ce45316-6e99-4cf1-83b4-b24f3822a4f2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:54:43.458463 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:43.458234 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef06038e-a7b5-4d47-a9a7-32129308ccc5-kube-api-access-jlj6z" pod "ef06038e-a7b5-4d47-a9a7-32129308ccc5" (UID: "ef06038e-a7b5-4d47-a9a7-32129308ccc5"). InnerVolumeSpecName "kube-api-access-jlj6z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:54:43.458819 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:43.458797 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ce45316-6e99-4cf1-83b4-b24f3822a4f2-kube-api-access-4mjsk" pod "4ce45316-6e99-4cf1-83b4-b24f3822a4f2" (UID: "4ce45316-6e99-4cf1-83b4-b24f3822a4f2"). InnerVolumeSpecName "kube-api-access-4mjsk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:54:43.459654 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:43.459631 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef06038e-a7b5-4d47-a9a7-32129308ccc5-maas-api-xks-ca" pod "ef06038e-a7b5-4d47-a9a7-32129308ccc5" (UID: "ef06038e-a7b5-4d47-a9a7-32129308ccc5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:54:43.556921 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:43.556889 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24079bcb-658a-4109-9b14-a167a38cfac7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24079bcb-658a-4109-9b14-a167a38cfac7-model-cache\") pod \"24079bcb-658a-4109-9b14-a167a38cfac7\" (UID: \"24079bcb-658a-4109-9b14-a167a38cfac7\") " Oct 31 00:54:43.557121 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:43.557008 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24079bcb-658a-4109-9b14-a167a38cfac7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24079bcb-658a-4109-9b14-a167a38cfac7-tmp-dir\") pod \"24079bcb-658a-4109-9b14-a167a38cfac7\" (UID: \"24079bcb-658a-4109-9b14-a167a38cfac7\") " Oct 31 00:54:43.557121 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:43.557057 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24079bcb-658a-4109-9b14-a167a38cfac7-home\" (UniqueName: \"kubernetes.io/empty-dir/24079bcb-658a-4109-9b14-a167a38cfac7-home\") pod \"24079bcb-658a-4109-9b14-a167a38cfac7\" (UID: \"24079bcb-658a-4109-9b14-a167a38cfac7\") " Oct 31 00:54:43.557242 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:43.557148 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24079bcb-658a-4109-9b14-a167a38cfac7-kube-api-access-49zss\" (UniqueName: \"kubernetes.io/projected/24079bcb-658a-4109-9b14-a167a38cfac7-kube-api-access-49zss\") pod \"24079bcb-658a-4109-9b14-a167a38cfac7\" (UID: \"24079bcb-658a-4109-9b14-a167a38cfac7\") " Oct 31 00:54:43.557242 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:43.557162 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24079bcb-658a-4109-9b14-a167a38cfac7-model-cache" pod "24079bcb-658a-4109-9b14-a167a38cfac7" (UID: "24079bcb-658a-4109-9b14-a167a38cfac7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:54:43.557242 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:43.557220 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24079bcb-658a-4109-9b14-a167a38cfac7-tmp-dir" pod "24079bcb-658a-4109-9b14-a167a38cfac7" (UID: "24079bcb-658a-4109-9b14-a167a38cfac7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:54:43.557242 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:43.557236 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24079bcb-658a-4109-9b14-a167a38cfac7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/24079bcb-658a-4109-9b14-a167a38cfac7-dshm\") pod \"24079bcb-658a-4109-9b14-a167a38cfac7\" (UID: \"24079bcb-658a-4109-9b14-a167a38cfac7\") " Oct 31 00:54:43.557502 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:43.557299 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24079bcb-658a-4109-9b14-a167a38cfac7-tls-certs\" (UniqueName: \"kubernetes.io/secret/24079bcb-658a-4109-9b14-a167a38cfac7-tls-certs\") pod \"24079bcb-658a-4109-9b14-a167a38cfac7\" (UID: \"24079bcb-658a-4109-9b14-a167a38cfac7\") " Oct 31 00:54:43.557704 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:43.557676 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24079bcb-658a-4109-9b14-a167a38cfac7-home" pod "24079bcb-658a-4109-9b14-a167a38cfac7" (UID: "24079bcb-658a-4109-9b14-a167a38cfac7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:54:43.557829 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:43.557813 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mjsk\" (UniqueName: \"kubernetes.io/projected/4ce45316-6e99-4cf1-83b4-b24f3822a4f2-kube-api-access-4mjsk\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:43.557904 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:43.557836 5222 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24079bcb-658a-4109-9b14-a167a38cfac7-model-cache\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:43.557904 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:43.557851 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlj6z\" (UniqueName: \"kubernetes.io/projected/ef06038e-a7b5-4d47-a9a7-32129308ccc5-kube-api-access-jlj6z\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:43.557904 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:43.557866 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef06038e-a7b5-4d47-a9a7-32129308ccc5-maas-api-xks-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:43.558061 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:43.557904 5222 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24079bcb-658a-4109-9b14-a167a38cfac7-tmp-dir\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:43.558061 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:43.557918 5222 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24079bcb-658a-4109-9b14-a167a38cfac7-home\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:43.558061 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:43.557932 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef06038e-a7b5-4d47-a9a7-32129308ccc5-maas-api-openshift-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:43.558061 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:43.557949 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ce45316-6e99-4cf1-83b4-b24f3822a4f2-maas-api-xks-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:43.558061 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:43.557964 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ce45316-6e99-4cf1-83b4-b24f3822a4f2-maas-api-openshift-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:43.559644 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:43.559618 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24079bcb-658a-4109-9b14-a167a38cfac7-dshm" pod "24079bcb-658a-4109-9b14-a167a38cfac7" (UID: "24079bcb-658a-4109-9b14-a167a38cfac7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:54:43.560161 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:43.560137 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24079bcb-658a-4109-9b14-a167a38cfac7-tls-certs" pod "24079bcb-658a-4109-9b14-a167a38cfac7" (UID: "24079bcb-658a-4109-9b14-a167a38cfac7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:54:43.560215 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:43.560179 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24079bcb-658a-4109-9b14-a167a38cfac7-kube-api-access-49zss" pod "24079bcb-658a-4109-9b14-a167a38cfac7" (UID: "24079bcb-658a-4109-9b14-a167a38cfac7"). InnerVolumeSpecName "kube-api-access-49zss". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:54:43.659151 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:43.659072 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49zss\" (UniqueName: \"kubernetes.io/projected/24079bcb-658a-4109-9b14-a167a38cfac7-kube-api-access-49zss\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:43.659151 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:43.659110 5222 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24079bcb-658a-4109-9b14-a167a38cfac7-dshm\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:43.659151 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:43.659126 5222 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24079bcb-658a-4109-9b14-a167a38cfac7-tls-certs\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:44.200842 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:44.200801 5222 generic.go:356] "Generic (PLEG): container finished" podID="24079bcb-658a-4109-9b14-a167a38cfac7" containerID="98c8701665839fb9b5694c7d1cb27b0fe5491f47e419c2d4032a60210b5adf5f" exitCode=0 Oct 31 00:54:44.201055 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:44.200856 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-a6f4a5/simulator-w6-a6f4a5-kserve-7769689d48-bpkc6" Oct 31 00:54:44.201055 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:44.200892 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-a6f4a5/simulator-w6-a6f4a5-kserve-7769689d48-bpkc6" event={"ID":"24079bcb-658a-4109-9b14-a167a38cfac7","Type":"ContainerDied","Data":"98c8701665839fb9b5694c7d1cb27b0fe5491f47e419c2d4032a60210b5adf5f"} Oct 31 00:54:44.201055 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:44.200936 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-a6f4a5/simulator-w6-a6f4a5-kserve-7769689d48-bpkc6" event={"ID":"24079bcb-658a-4109-9b14-a167a38cfac7","Type":"ContainerDied","Data":"68b003aaf4f5c90c9ca8fa5ac14c30e7a0a3fecb255c9479a00ac4f201090478"} Oct 31 00:54:44.201055 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:44.200958 5222 scope.go:122] "RemoveContainer" containerID="98c8701665839fb9b5694c7d1cb27b0fe5491f47e419c2d4032a60210b5adf5f" Oct 31 00:54:44.202548 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:44.202514 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9a35539-tvbkt" event={"ID":"ef06038e-a7b5-4d47-a9a7-32129308ccc5","Type":"ContainerDied","Data":"cfb6475900a3d1f56a0e3ecd0810b0b3c3e7ab8340a24965fa57d957f3d1aa89"} Oct 31 00:54:44.202667 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:44.202569 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cfb6475900a3d1f56a0e3ecd0810b0b3c3e7ab8340a24965fa57d957f3d1aa89" Oct 31 00:54:44.202667 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:44.202521 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9a35539-tvbkt" Oct 31 00:54:44.204090 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:44.204066 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-432acaff-l87vl" Oct 31 00:54:44.204213 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:44.204108 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-432acaff-l87vl" event={"ID":"4ce45316-6e99-4cf1-83b4-b24f3822a4f2","Type":"ContainerDied","Data":"7f3641945fe9e073f5c8a73b28a0d14187b64f1a896876ec6944713cf656bfae"} Oct 31 00:54:44.204213 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:44.204160 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7f3641945fe9e073f5c8a73b28a0d14187b64f1a896876ec6944713cf656bfae" Oct 31 00:54:44.211989 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:44.211970 5222 scope.go:122] "RemoveContainer" containerID="98c8701665839fb9b5694c7d1cb27b0fe5491f47e419c2d4032a60210b5adf5f" Oct 31 00:54:44.212252 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:54:44.212232 5222 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98c8701665839fb9b5694c7d1cb27b0fe5491f47e419c2d4032a60210b5adf5f\": container with ID starting with 98c8701665839fb9b5694c7d1cb27b0fe5491f47e419c2d4032a60210b5adf5f not found: ID does not exist" containerID="98c8701665839fb9b5694c7d1cb27b0fe5491f47e419c2d4032a60210b5adf5f" Oct 31 00:54:44.212301 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:44.212260 5222 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98c8701665839fb9b5694c7d1cb27b0fe5491f47e419c2d4032a60210b5adf5f"} err="failed to get container status \"98c8701665839fb9b5694c7d1cb27b0fe5491f47e419c2d4032a60210b5adf5f\": rpc error: code = NotFound desc = could not find container \"98c8701665839fb9b5694c7d1cb27b0fe5491f47e419c2d4032a60210b5adf5f\": container with ID starting with 98c8701665839fb9b5694c7d1cb27b0fe5491f47e419c2d4032a60210b5adf5f not found: ID does not exist" Oct 31 00:54:44.332185 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:44.332134 5222 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6f4a5/simulator-w6-a6f4a5-kserve-7769689d48-bpkc6" podUID="24079bcb-658a-4109-9b14-a167a38cfac7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:54:44.477613 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:44.477522 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-a6f4a5/simulator-w6-a6f4a5-kserve-7769689d48-bpkc6"] Oct 31 00:54:44.481300 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:44.481275 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-a6f4a5/simulator-w6-a6f4a5-kserve-7769689d48-bpkc6"] Oct 31 00:54:45.657068 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:45.657030 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24079bcb-658a-4109-9b14-a167a38cfac7" path="/var/lib/kubelet/pods/24079bcb-658a-4109-9b14-a167a38cfac7/volumes" Oct 31 00:54:46.868958 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:46.868924 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-4ab8e5/test-model-4ab8e5-kserve-78dd57fd69-h64c4"] Oct 31 00:54:46.869542 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:46.869525 5222 cpu_manager.go:414] "containerMap: removing container" podUID="4ce45316-6e99-4cf1-83b4-b24f3822a4f2" containerName="revoke-keys" Oct 31 00:54:46.869603 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:46.869542 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ce45316-6e99-4cf1-83b4-b24f3822a4f2" containerName="revoke-keys" Oct 31 00:54:46.869603 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:46.869555 5222 cpu_manager.go:414] "containerMap: removing container" podUID="24079bcb-658a-4109-9b14-a167a38cfac7" containerName="main" Oct 31 00:54:46.869603 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:46.869560 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24079bcb-658a-4109-9b14-a167a38cfac7" containerName="main" Oct 31 00:54:46.869603 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:46.869574 5222 cpu_manager.go:414] "containerMap: removing container" podUID="ef06038e-a7b5-4d47-a9a7-32129308ccc5" containerName="revoke-keys" Oct 31 00:54:46.869603 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:46.869579 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef06038e-a7b5-4d47-a9a7-32129308ccc5" containerName="revoke-keys" Oct 31 00:54:46.869762 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:46.869693 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="24079bcb-658a-4109-9b14-a167a38cfac7" containerName="main" Oct 31 00:54:46.869762 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:46.869703 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef06038e-a7b5-4d47-a9a7-32129308ccc5" containerName="revoke-keys" Oct 31 00:54:46.869762 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:46.869717 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ce45316-6e99-4cf1-83b4-b24f3822a4f2" containerName="revoke-keys" Oct 31 00:54:46.874757 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:46.874733 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-4ab8e5/test-model-4ab8e5-kserve-78dd57fd69-h64c4" Oct 31 00:54:46.927810 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:46.927774 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-4ab8e5/test-model-4ab8e5-kserve-78dd57fd69-h64c4"] Oct 31 00:54:46.975446 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:46.975416 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-4ab8e5\"/\"openshift-service-ca.crt\"" Oct 31 00:54:46.975641 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:46.975490 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-4ab8e5\"/\"test-model-4ab8e5-kserve-self-signed-certs\"" Oct 31 00:54:46.980066 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:46.980036 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-4ab8e5\"/\"default-dockercfg-rnv4t\"" Oct 31 00:54:46.980066 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:46.980061 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-4ab8e5\"/\"kube-root-ca.crt\"" Oct 31 00:54:47.001978 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:47.001945 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c05360e3-dc37-4ee8-8589-9d32cb58a814-dshm\") pod \"test-model-4ab8e5-kserve-78dd57fd69-h64c4\" (UID: \"c05360e3-dc37-4ee8-8589-9d32cb58a814\") " pod="ai-tenant-e2e-inf-a-4ab8e5/test-model-4ab8e5-kserve-78dd57fd69-h64c4" Oct 31 00:54:47.002094 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:47.001990 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c05360e3-dc37-4ee8-8589-9d32cb58a814-tmp-dir\") pod \"test-model-4ab8e5-kserve-78dd57fd69-h64c4\" (UID: \"c05360e3-dc37-4ee8-8589-9d32cb58a814\") " pod="ai-tenant-e2e-inf-a-4ab8e5/test-model-4ab8e5-kserve-78dd57fd69-h64c4" Oct 31 00:54:47.002181 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:47.002162 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c05360e3-dc37-4ee8-8589-9d32cb58a814-tls-certs\") pod \"test-model-4ab8e5-kserve-78dd57fd69-h64c4\" (UID: \"c05360e3-dc37-4ee8-8589-9d32cb58a814\") " pod="ai-tenant-e2e-inf-a-4ab8e5/test-model-4ab8e5-kserve-78dd57fd69-h64c4" Oct 31 00:54:47.002224 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:47.002198 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9krk\" (UniqueName: \"kubernetes.io/projected/c05360e3-dc37-4ee8-8589-9d32cb58a814-kube-api-access-k9krk\") pod \"test-model-4ab8e5-kserve-78dd57fd69-h64c4\" (UID: \"c05360e3-dc37-4ee8-8589-9d32cb58a814\") " pod="ai-tenant-e2e-inf-a-4ab8e5/test-model-4ab8e5-kserve-78dd57fd69-h64c4" Oct 31 00:54:47.002330 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:47.002314 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c05360e3-dc37-4ee8-8589-9d32cb58a814-home\") pod \"test-model-4ab8e5-kserve-78dd57fd69-h64c4\" (UID: \"c05360e3-dc37-4ee8-8589-9d32cb58a814\") " pod="ai-tenant-e2e-inf-a-4ab8e5/test-model-4ab8e5-kserve-78dd57fd69-h64c4" Oct 31 00:54:47.002414 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:47.002353 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c05360e3-dc37-4ee8-8589-9d32cb58a814-model-cache\") pod \"test-model-4ab8e5-kserve-78dd57fd69-h64c4\" (UID: \"c05360e3-dc37-4ee8-8589-9d32cb58a814\") " pod="ai-tenant-e2e-inf-a-4ab8e5/test-model-4ab8e5-kserve-78dd57fd69-h64c4" Oct 31 00:54:47.103607 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:47.103562 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c05360e3-dc37-4ee8-8589-9d32cb58a814-tmp-dir\") pod \"test-model-4ab8e5-kserve-78dd57fd69-h64c4\" (UID: \"c05360e3-dc37-4ee8-8589-9d32cb58a814\") " pod="ai-tenant-e2e-inf-a-4ab8e5/test-model-4ab8e5-kserve-78dd57fd69-h64c4" Oct 31 00:54:47.103826 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:47.103706 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c05360e3-dc37-4ee8-8589-9d32cb58a814-tls-certs\") pod \"test-model-4ab8e5-kserve-78dd57fd69-h64c4\" (UID: \"c05360e3-dc37-4ee8-8589-9d32cb58a814\") " pod="ai-tenant-e2e-inf-a-4ab8e5/test-model-4ab8e5-kserve-78dd57fd69-h64c4" Oct 31 00:54:47.103826 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:47.103746 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9krk\" (UniqueName: \"kubernetes.io/projected/c05360e3-dc37-4ee8-8589-9d32cb58a814-kube-api-access-k9krk\") pod \"test-model-4ab8e5-kserve-78dd57fd69-h64c4\" (UID: \"c05360e3-dc37-4ee8-8589-9d32cb58a814\") " pod="ai-tenant-e2e-inf-a-4ab8e5/test-model-4ab8e5-kserve-78dd57fd69-h64c4" Oct 31 00:54:47.103969 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:47.103845 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c05360e3-dc37-4ee8-8589-9d32cb58a814-home\") pod \"test-model-4ab8e5-kserve-78dd57fd69-h64c4\" (UID: \"c05360e3-dc37-4ee8-8589-9d32cb58a814\") " pod="ai-tenant-e2e-inf-a-4ab8e5/test-model-4ab8e5-kserve-78dd57fd69-h64c4" Oct 31 00:54:47.103969 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:47.103892 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c05360e3-dc37-4ee8-8589-9d32cb58a814-model-cache\") pod \"test-model-4ab8e5-kserve-78dd57fd69-h64c4\" (UID: \"c05360e3-dc37-4ee8-8589-9d32cb58a814\") " pod="ai-tenant-e2e-inf-a-4ab8e5/test-model-4ab8e5-kserve-78dd57fd69-h64c4" Oct 31 00:54:47.103969 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:47.103934 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c05360e3-dc37-4ee8-8589-9d32cb58a814-dshm\") pod \"test-model-4ab8e5-kserve-78dd57fd69-h64c4\" (UID: \"c05360e3-dc37-4ee8-8589-9d32cb58a814\") " pod="ai-tenant-e2e-inf-a-4ab8e5/test-model-4ab8e5-kserve-78dd57fd69-h64c4" Oct 31 00:54:47.104128 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:47.103977 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c05360e3-dc37-4ee8-8589-9d32cb58a814-tmp-dir\") pod \"test-model-4ab8e5-kserve-78dd57fd69-h64c4\" (UID: \"c05360e3-dc37-4ee8-8589-9d32cb58a814\") " pod="ai-tenant-e2e-inf-a-4ab8e5/test-model-4ab8e5-kserve-78dd57fd69-h64c4" Oct 31 00:54:47.104213 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:47.104192 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c05360e3-dc37-4ee8-8589-9d32cb58a814-home\") pod \"test-model-4ab8e5-kserve-78dd57fd69-h64c4\" (UID: \"c05360e3-dc37-4ee8-8589-9d32cb58a814\") " pod="ai-tenant-e2e-inf-a-4ab8e5/test-model-4ab8e5-kserve-78dd57fd69-h64c4" Oct 31 00:54:47.104213 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:47.104202 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c05360e3-dc37-4ee8-8589-9d32cb58a814-model-cache\") pod \"test-model-4ab8e5-kserve-78dd57fd69-h64c4\" (UID: \"c05360e3-dc37-4ee8-8589-9d32cb58a814\") " pod="ai-tenant-e2e-inf-a-4ab8e5/test-model-4ab8e5-kserve-78dd57fd69-h64c4" Oct 31 00:54:47.106230 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:47.106201 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c05360e3-dc37-4ee8-8589-9d32cb58a814-dshm\") pod \"test-model-4ab8e5-kserve-78dd57fd69-h64c4\" (UID: \"c05360e3-dc37-4ee8-8589-9d32cb58a814\") " pod="ai-tenant-e2e-inf-a-4ab8e5/test-model-4ab8e5-kserve-78dd57fd69-h64c4" Oct 31 00:54:47.106551 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:47.106527 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c05360e3-dc37-4ee8-8589-9d32cb58a814-tls-certs\") pod \"test-model-4ab8e5-kserve-78dd57fd69-h64c4\" (UID: \"c05360e3-dc37-4ee8-8589-9d32cb58a814\") " pod="ai-tenant-e2e-inf-a-4ab8e5/test-model-4ab8e5-kserve-78dd57fd69-h64c4" Oct 31 00:54:47.166654 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:47.166584 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9krk\" (UniqueName: \"kubernetes.io/projected/c05360e3-dc37-4ee8-8589-9d32cb58a814-kube-api-access-k9krk\") pod \"test-model-4ab8e5-kserve-78dd57fd69-h64c4\" (UID: \"c05360e3-dc37-4ee8-8589-9d32cb58a814\") " pod="ai-tenant-e2e-inf-a-4ab8e5/test-model-4ab8e5-kserve-78dd57fd69-h64c4" Oct 31 00:54:47.187538 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:47.187494 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-4ab8e5/test-model-4ab8e5-kserve-78dd57fd69-h64c4" Oct 31 00:54:47.564942 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:54:47.564914 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc05360e3_dc37_4ee8_8589_9d32cb58a814.slice/crio-bd91276dfecd28a8b6f7d78650a22273023d752d2d2608eb0d3cf12a261aecfe WatchSource:0}: Error finding container bd91276dfecd28a8b6f7d78650a22273023d752d2d2608eb0d3cf12a261aecfe: Status 404 returned error can't find the container with id bd91276dfecd28a8b6f7d78650a22273023d752d2d2608eb0d3cf12a261aecfe Oct 31 00:54:47.567176 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:47.567151 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-4ab8e5/test-model-4ab8e5-kserve-78dd57fd69-h64c4"] Oct 31 00:54:48.234148 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:48.234112 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-4ab8e5/test-model-4ab8e5-kserve-78dd57fd69-h64c4" event={"ID":"c05360e3-dc37-4ee8-8589-9d32cb58a814","Type":"ContainerStarted","Data":"d5917ea2f01324648c6e288197658969633db19fc68c4e24a8eaaa08d3d61a65"} Oct 31 00:54:48.234648 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:48.234157 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-4ab8e5/test-model-4ab8e5-kserve-78dd57fd69-h64c4" event={"ID":"c05360e3-dc37-4ee8-8589-9d32cb58a814","Type":"ContainerStarted","Data":"bd91276dfecd28a8b6f7d78650a22273023d752d2d2608eb0d3cf12a261aecfe"} Oct 31 00:54:48.357798 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:48.357740 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-4ab8e5/test-model-4ab8e5-kserve-78dd57fd69-h64c4" podStartSLOduration=2.35772573 podStartE2EDuration="2.35772573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:54:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:54:48.355699333 +0000 UTC m=+2380.826869082" watchObservedRunningTime="2026-10-31 00:54:48.35772573 +0000 UTC m=+2380.828895479" Oct 31 00:54:48.416543 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:48.416511 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-1e5e75-ff6df9bb9-d888l" Oct 31 00:54:48.534151 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:48.534114 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/795c26a3-f65f-40f0-9cd6-503ab681a866-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/795c26a3-f65f-40f0-9cd6-503ab681a866-plugins-config-volume\") pod \"795c26a3-f65f-40f0-9cd6-503ab681a866\" (UID: \"795c26a3-f65f-40f0-9cd6-503ab681a866\") " Oct 31 00:54:48.534336 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:48.534187 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/795c26a3-f65f-40f0-9cd6-503ab681a866-kube-api-access-jcv6x\" (UniqueName: \"kubernetes.io/projected/795c26a3-f65f-40f0-9cd6-503ab681a866-kube-api-access-jcv6x\") pod \"795c26a3-f65f-40f0-9cd6-503ab681a866\" (UID: \"795c26a3-f65f-40f0-9cd6-503ab681a866\") " Oct 31 00:54:48.534481 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:48.534451 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/795c26a3-f65f-40f0-9cd6-503ab681a866-plugins-config-volume" pod "795c26a3-f65f-40f0-9cd6-503ab681a866" (UID: "795c26a3-f65f-40f0-9cd6-503ab681a866"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:54:48.534797 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:48.534773 5222 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/795c26a3-f65f-40f0-9cd6-503ab681a866-plugins-config-volume\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:48.536939 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:48.536912 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/795c26a3-f65f-40f0-9cd6-503ab681a866-kube-api-access-jcv6x" pod "795c26a3-f65f-40f0-9cd6-503ab681a866" (UID: "795c26a3-f65f-40f0-9cd6-503ab681a866"). InnerVolumeSpecName "kube-api-access-jcv6x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:54:48.636240 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:48.636205 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcv6x\" (UniqueName: \"kubernetes.io/projected/795c26a3-f65f-40f0-9cd6-503ab681a866-kube-api-access-jcv6x\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:49.188291 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:49.188251 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-4ab8e5/test-model-4ab8e5-kserve-78dd57fd69-h64c4" Oct 31 00:54:49.242341 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:49.242308 5222 generic.go:356] "Generic (PLEG): container finished" podID="795c26a3-f65f-40f0-9cd6-503ab681a866" containerID="d5a9a484f5eb5ed7a74f93d682c027ec376ca05bf3215dc72e775878918b66f3" exitCode=0 Oct 31 00:54:49.242796 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:49.242358 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-1e5e75-ff6df9bb9-d888l" Oct 31 00:54:49.242796 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:49.242399 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-1e5e75-ff6df9bb9-d888l" event={"ID":"795c26a3-f65f-40f0-9cd6-503ab681a866","Type":"ContainerDied","Data":"d5a9a484f5eb5ed7a74f93d682c027ec376ca05bf3215dc72e775878918b66f3"} Oct 31 00:54:49.242796 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:49.242537 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-1e5e75-ff6df9bb9-d888l" event={"ID":"795c26a3-f65f-40f0-9cd6-503ab681a866","Type":"ContainerDied","Data":"6828b5ac8b242aa183bdc0c2415b2de10679848aa9e5f1ad54b055690a7b1be3"} Oct 31 00:54:49.242796 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:49.242566 5222 scope.go:122] "RemoveContainer" containerID="d5a9a484f5eb5ed7a74f93d682c027ec376ca05bf3215dc72e775878918b66f3" Oct 31 00:54:49.252723 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:49.252705 5222 scope.go:122] "RemoveContainer" containerID="d5a9a484f5eb5ed7a74f93d682c027ec376ca05bf3215dc72e775878918b66f3" Oct 31 00:54:49.253012 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:54:49.252992 5222 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5a9a484f5eb5ed7a74f93d682c027ec376ca05bf3215dc72e775878918b66f3\": container with ID starting with d5a9a484f5eb5ed7a74f93d682c027ec376ca05bf3215dc72e775878918b66f3 not found: ID does not exist" containerID="d5a9a484f5eb5ed7a74f93d682c027ec376ca05bf3215dc72e775878918b66f3" Oct 31 00:54:49.253058 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:49.253020 5222 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5a9a484f5eb5ed7a74f93d682c027ec376ca05bf3215dc72e775878918b66f3"} err="failed to get container status \"d5a9a484f5eb5ed7a74f93d682c027ec376ca05bf3215dc72e775878918b66f3\": rpc error: code = NotFound desc = could not find container \"d5a9a484f5eb5ed7a74f93d682c027ec376ca05bf3215dc72e775878918b66f3\": container with ID starting with d5a9a484f5eb5ed7a74f93d682c027ec376ca05bf3215dc72e775878918b66f3 not found: ID does not exist" Oct 31 00:54:49.422769 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:49.422724 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-1e5e75-ff6df9bb9-d888l"] Oct 31 00:54:49.426827 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:49.426799 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-1e5e75-ff6df9bb9-d888l"] Oct 31 00:54:49.655622 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:49.655590 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="795c26a3-f65f-40f0-9cd6-503ab681a866" path="/var/lib/kubelet/pods/795c26a3-f65f-40f0-9cd6-503ab681a866/volumes" Oct 31 00:54:50.993766 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:50.993731 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sf5lc"] Oct 31 00:54:50.994266 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:50.993959 5222 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sf5lc" podUID="6039aff2-c967-413b-865a-d933b6c0c9ba" containerName="istio-proxy" containerID="cri-o://e80d62d0507fb6c42657188e5402d5a0f774b71d46f47fa2b20847b935c462ce" gracePeriod=30 Oct 31 00:54:51.911742 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:51.911694 5222 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-a6f4a5-7b4d7b4dcc-v2hts" podUID="37865b42-ea0f-4744-ab63-9dbc9b7f74b1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:54:51.911925 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:51.911852 5222 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a6f4a5-7666bb86c-glmj2" podUID="91cb96ab-e55e-463f-9fe0-059e65184a2c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:54:54.684080 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:54.684042 5222 patch_prober.go:29] interesting pod/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sf5lc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:54:54.684583 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:54.684101 5222 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sf5lc" podUID="6039aff2-c967-413b-865a-d933b6c0c9ba" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:54:55.925282 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:55.925242 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-79tk8"] Oct 31 00:54:56.260579 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:56.260554 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sf5lc" Oct 31 00:54:56.312940 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:56.312907 5222 generic.go:356] "Generic (PLEG): container finished" podID="6039aff2-c967-413b-865a-d933b6c0c9ba" containerID="e80d62d0507fb6c42657188e5402d5a0f774b71d46f47fa2b20847b935c462ce" exitCode=0 Oct 31 00:54:56.313139 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:56.312979 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sf5lc" Oct 31 00:54:56.313139 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:56.312997 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sf5lc" event={"ID":"6039aff2-c967-413b-865a-d933b6c0c9ba","Type":"ContainerDied","Data":"e80d62d0507fb6c42657188e5402d5a0f774b71d46f47fa2b20847b935c462ce"} Oct 31 00:54:56.313139 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:56.313056 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sf5lc" event={"ID":"6039aff2-c967-413b-865a-d933b6c0c9ba","Type":"ContainerDied","Data":"1d3484ed1ef5ced6e5ec9da187ca5252b57767fc95c4295553de2040ab2b2e6c"} Oct 31 00:54:56.313139 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:56.313083 5222 scope.go:122] "RemoveContainer" containerID="e80d62d0507fb6c42657188e5402d5a0f774b71d46f47fa2b20847b935c462ce" Oct 31 00:54:56.323536 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:56.323512 5222 scope.go:122] "RemoveContainer" containerID="e80d62d0507fb6c42657188e5402d5a0f774b71d46f47fa2b20847b935c462ce" Oct 31 00:54:56.323958 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:54:56.323933 5222 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e80d62d0507fb6c42657188e5402d5a0f774b71d46f47fa2b20847b935c462ce\": container with ID starting with e80d62d0507fb6c42657188e5402d5a0f774b71d46f47fa2b20847b935c462ce not found: ID does not exist" containerID="e80d62d0507fb6c42657188e5402d5a0f774b71d46f47fa2b20847b935c462ce" Oct 31 00:54:56.324053 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:56.323970 5222 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e80d62d0507fb6c42657188e5402d5a0f774b71d46f47fa2b20847b935c462ce"} err="failed to get container status \"e80d62d0507fb6c42657188e5402d5a0f774b71d46f47fa2b20847b935c462ce\": rpc error: code = NotFound desc = could not find container \"e80d62d0507fb6c42657188e5402d5a0f774b71d46f47fa2b20847b935c462ce\": container with ID starting with e80d62d0507fb6c42657188e5402d5a0f774b71d46f47fa2b20847b935c462ce not found: ID does not exist" Oct 31 00:54:56.356874 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:56.356844 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6039aff2-c967-413b-865a-d933b6c0c9ba-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6039aff2-c967-413b-865a-d933b6c0c9ba-workload-certs\") pod \"6039aff2-c967-413b-865a-d933b6c0c9ba\" (UID: \"6039aff2-c967-413b-865a-d933b6c0c9ba\") " Oct 31 00:54:56.357046 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:56.356893 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6039aff2-c967-413b-865a-d933b6c0c9ba-istio-token\" (UniqueName: \"kubernetes.io/projected/6039aff2-c967-413b-865a-d933b6c0c9ba-istio-token\") pod \"6039aff2-c967-413b-865a-d933b6c0c9ba\" (UID: \"6039aff2-c967-413b-865a-d933b6c0c9ba\") " Oct 31 00:54:56.357046 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:56.356916 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6039aff2-c967-413b-865a-d933b6c0c9ba-kube-api-access-zdkfd\" (UniqueName: \"kubernetes.io/projected/6039aff2-c967-413b-865a-d933b6c0c9ba-kube-api-access-zdkfd\") pod \"6039aff2-c967-413b-865a-d933b6c0c9ba\" (UID: \"6039aff2-c967-413b-865a-d933b6c0c9ba\") " Oct 31 00:54:56.357046 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:56.356942 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6039aff2-c967-413b-865a-d933b6c0c9ba-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6039aff2-c967-413b-865a-d933b6c0c9ba-istio-data\") pod \"6039aff2-c967-413b-865a-d933b6c0c9ba\" (UID: \"6039aff2-c967-413b-865a-d933b6c0c9ba\") " Oct 31 00:54:56.357046 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:56.357025 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6039aff2-c967-413b-865a-d933b6c0c9ba-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6039aff2-c967-413b-865a-d933b6c0c9ba-credential-socket\") pod \"6039aff2-c967-413b-865a-d933b6c0c9ba\" (UID: \"6039aff2-c967-413b-865a-d933b6c0c9ba\") " Oct 31 00:54:56.357281 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:56.357074 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6039aff2-c967-413b-865a-d933b6c0c9ba-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6039aff2-c967-413b-865a-d933b6c0c9ba-istio-envoy\") pod \"6039aff2-c967-413b-865a-d933b6c0c9ba\" (UID: \"6039aff2-c967-413b-865a-d933b6c0c9ba\") " Oct 31 00:54:56.357281 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:56.357109 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6039aff2-c967-413b-865a-d933b6c0c9ba-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6039aff2-c967-413b-865a-d933b6c0c9ba-istiod-ca-cert\") pod \"6039aff2-c967-413b-865a-d933b6c0c9ba\" (UID: \"6039aff2-c967-413b-865a-d933b6c0c9ba\") " Oct 31 00:54:56.357281 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:56.357193 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6039aff2-c967-413b-865a-d933b6c0c9ba-workload-certs" pod "6039aff2-c967-413b-865a-d933b6c0c9ba" (UID: "6039aff2-c967-413b-865a-d933b6c0c9ba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:54:56.357281 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:56.357226 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6039aff2-c967-413b-865a-d933b6c0c9ba-istio-data" pod "6039aff2-c967-413b-865a-d933b6c0c9ba" (UID: "6039aff2-c967-413b-865a-d933b6c0c9ba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:54:56.357534 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:56.357301 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6039aff2-c967-413b-865a-d933b6c0c9ba-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6039aff2-c967-413b-865a-d933b6c0c9ba-istio-podinfo\") pod \"6039aff2-c967-413b-865a-d933b6c0c9ba\" (UID: \"6039aff2-c967-413b-865a-d933b6c0c9ba\") " Oct 31 00:54:56.357534 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:56.357325 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6039aff2-c967-413b-865a-d933b6c0c9ba-credential-socket" pod "6039aff2-c967-413b-865a-d933b6c0c9ba" (UID: "6039aff2-c967-413b-865a-d933b6c0c9ba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:54:56.357534 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:56.357335 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6039aff2-c967-413b-865a-d933b6c0c9ba-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6039aff2-c967-413b-865a-d933b6c0c9ba-workload-socket\") pod \"6039aff2-c967-413b-865a-d933b6c0c9ba\" (UID: \"6039aff2-c967-413b-865a-d933b6c0c9ba\") " Oct 31 00:54:56.357688 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:56.357555 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6039aff2-c967-413b-865a-d933b6c0c9ba-workload-socket" pod "6039aff2-c967-413b-865a-d933b6c0c9ba" (UID: "6039aff2-c967-413b-865a-d933b6c0c9ba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:54:56.357688 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:56.357599 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6039aff2-c967-413b-865a-d933b6c0c9ba-istiod-ca-cert" pod "6039aff2-c967-413b-865a-d933b6c0c9ba" (UID: "6039aff2-c967-413b-865a-d933b6c0c9ba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:54:56.358133 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:56.358106 5222 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6039aff2-c967-413b-865a-d933b6c0c9ba-istio-data\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:56.358193 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:56.358140 5222 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6039aff2-c967-413b-865a-d933b6c0c9ba-credential-socket\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:56.358193 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:56.358156 5222 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6039aff2-c967-413b-865a-d933b6c0c9ba-istiod-ca-cert\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:56.358193 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:56.358170 5222 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6039aff2-c967-413b-865a-d933b6c0c9ba-workload-socket\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:56.358193 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:56.358185 5222 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6039aff2-c967-413b-865a-d933b6c0c9ba-workload-certs\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:56.359344 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:56.359320 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6039aff2-c967-413b-865a-d933b6c0c9ba-kube-api-access-zdkfd" pod "6039aff2-c967-413b-865a-d933b6c0c9ba" (UID: "6039aff2-c967-413b-865a-d933b6c0c9ba"). InnerVolumeSpecName "kube-api-access-zdkfd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:54:56.359551 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:56.359527 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6039aff2-c967-413b-865a-d933b6c0c9ba-istio-podinfo" pod "6039aff2-c967-413b-865a-d933b6c0c9ba" (UID: "6039aff2-c967-413b-865a-d933b6c0c9ba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:54:56.359833 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:56.359813 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6039aff2-c967-413b-865a-d933b6c0c9ba-istio-token" pod "6039aff2-c967-413b-865a-d933b6c0c9ba" (UID: "6039aff2-c967-413b-865a-d933b6c0c9ba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:54:56.359888 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:56.359826 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6039aff2-c967-413b-865a-d933b6c0c9ba-istio-envoy" pod "6039aff2-c967-413b-865a-d933b6c0c9ba" (UID: "6039aff2-c967-413b-865a-d933b6c0c9ba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:54:56.459831 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:56.459787 5222 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6039aff2-c967-413b-865a-d933b6c0c9ba-istio-token\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:56.459831 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:56.459834 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdkfd\" (UniqueName: \"kubernetes.io/projected/6039aff2-c967-413b-865a-d933b6c0c9ba-kube-api-access-zdkfd\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:56.460109 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:56.459852 5222 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6039aff2-c967-413b-865a-d933b6c0c9ba-istio-envoy\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:56.460109 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:56.459865 5222 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6039aff2-c967-413b-865a-d933b6c0c9ba-istio-podinfo\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:56.767537 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:56.767454 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sf5lc"] Oct 31 00:54:56.777951 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:56.777898 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-sf5lc"] Oct 31 00:54:57.188168 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:57.188130 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-4ab8e5/test-model-4ab8e5-kserve-78dd57fd69-h64c4" Oct 31 00:54:57.202068 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:57.202040 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-4ab8e5/test-model-4ab8e5-kserve-78dd57fd69-h64c4" Oct 31 00:54:57.334171 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:57.334139 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-4ab8e5/test-model-4ab8e5-kserve-78dd57fd69-h64c4" Oct 31 00:54:57.656984 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:57.656949 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6039aff2-c967-413b-865a-d933b6c0c9ba" path="/var/lib/kubelet/pods/6039aff2-c967-413b-865a-d933b6c0c9ba/volumes" Oct 31 00:54:59.916405 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:59.916355 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54259ec6-rsllx"] Oct 31 00:54:59.917165 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:59.917143 5222 cpu_manager.go:414] "containerMap: removing container" podUID="6039aff2-c967-413b-865a-d933b6c0c9ba" containerName="istio-proxy" Oct 31 00:54:59.917165 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:59.917166 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6039aff2-c967-413b-865a-d933b6c0c9ba" containerName="istio-proxy" Oct 31 00:54:59.917331 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:59.917185 5222 cpu_manager.go:414] "containerMap: removing container" podUID="795c26a3-f65f-40f0-9cd6-503ab681a866" containerName="payload-processing" Oct 31 00:54:59.917331 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:59.917192 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="795c26a3-f65f-40f0-9cd6-503ab681a866" containerName="payload-processing" Oct 31 00:54:59.917491 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:59.917355 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="6039aff2-c967-413b-865a-d933b6c0c9ba" containerName="istio-proxy" Oct 31 00:54:59.917491 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:59.917387 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="795c26a3-f65f-40f0-9cd6-503ab681a866" containerName="payload-processing" Oct 31 00:54:59.920784 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:59.920762 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54259ec6-rsllx" Oct 31 00:54:59.975404 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:59.975332 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54259ec6-rsllx"] Oct 31 00:54:59.977726 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:59.977700 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e83b6db1-gmc99"] Oct 31 00:54:59.981634 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:54:59.981611 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e83b6db1-gmc99" Oct 31 00:55:00.009149 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:00.009116 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10bf9d7a-5f72-4d13-a167-38cc2b204282-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e83b6db1-gmc99\" (UID: \"10bf9d7a-5f72-4d13-a167-38cc2b204282\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e83b6db1-gmc99" Oct 31 00:55:00.009320 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:00.009160 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgscs\" (UniqueName: \"kubernetes.io/projected/328b5ab8-81eb-4f94-8210-a0f390052a86-kube-api-access-hgscs\") pod \"maas-api-revoke-sub-54259ec6-rsllx\" (UID: \"328b5ab8-81eb-4f94-8210-a0f390052a86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54259ec6-rsllx" Oct 31 00:55:00.009427 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:00.009401 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/328b5ab8-81eb-4f94-8210-a0f390052a86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54259ec6-rsllx\" (UID: \"328b5ab8-81eb-4f94-8210-a0f390052a86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54259ec6-rsllx" Oct 31 00:55:00.009490 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:00.009463 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10bf9d7a-5f72-4d13-a167-38cc2b204282-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e83b6db1-gmc99\" (UID: \"10bf9d7a-5f72-4d13-a167-38cc2b204282\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e83b6db1-gmc99" Oct 31 00:55:00.009593 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:00.009574 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxslb\" (UniqueName: \"kubernetes.io/projected/10bf9d7a-5f72-4d13-a167-38cc2b204282-kube-api-access-rxslb\") pod \"maas-api-revoke-sub-e83b6db1-gmc99\" (UID: \"10bf9d7a-5f72-4d13-a167-38cc2b204282\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e83b6db1-gmc99" Oct 31 00:55:00.009673 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:00.009653 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/328b5ab8-81eb-4f94-8210-a0f390052a86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54259ec6-rsllx\" (UID: \"328b5ab8-81eb-4f94-8210-a0f390052a86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54259ec6-rsllx" Oct 31 00:55:00.022028 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:00.022001 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:55:00.025945 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:00.025923 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfz9z\"" Oct 31 00:55:00.025945 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:00.025935 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:55:00.026140 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:00.025950 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:55:00.093296 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:00.093260 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e83b6db1-gmc99"] Oct 31 00:55:00.111317 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:00.111287 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/328b5ab8-81eb-4f94-8210-a0f390052a86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54259ec6-rsllx\" (UID: \"328b5ab8-81eb-4f94-8210-a0f390052a86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54259ec6-rsllx" Oct 31 00:55:00.111515 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:00.111339 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10bf9d7a-5f72-4d13-a167-38cc2b204282-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e83b6db1-gmc99\" (UID: \"10bf9d7a-5f72-4d13-a167-38cc2b204282\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e83b6db1-gmc99" Oct 31 00:55:00.111515 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:00.111405 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxslb\" (UniqueName: \"kubernetes.io/projected/10bf9d7a-5f72-4d13-a167-38cc2b204282-kube-api-access-rxslb\") pod \"maas-api-revoke-sub-e83b6db1-gmc99\" (UID: \"10bf9d7a-5f72-4d13-a167-38cc2b204282\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e83b6db1-gmc99" Oct 31 00:55:00.111515 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:00.111446 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/328b5ab8-81eb-4f94-8210-a0f390052a86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54259ec6-rsllx\" (UID: \"328b5ab8-81eb-4f94-8210-a0f390052a86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54259ec6-rsllx" Oct 31 00:55:00.111515 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:00.111485 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10bf9d7a-5f72-4d13-a167-38cc2b204282-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e83b6db1-gmc99\" (UID: \"10bf9d7a-5f72-4d13-a167-38cc2b204282\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e83b6db1-gmc99" Oct 31 00:55:00.111733 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:00.111519 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgscs\" (UniqueName: \"kubernetes.io/projected/328b5ab8-81eb-4f94-8210-a0f390052a86-kube-api-access-hgscs\") pod \"maas-api-revoke-sub-54259ec6-rsllx\" (UID: \"328b5ab8-81eb-4f94-8210-a0f390052a86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54259ec6-rsllx" Oct 31 00:55:00.112345 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:00.112299 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/328b5ab8-81eb-4f94-8210-a0f390052a86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54259ec6-rsllx\" (UID: \"328b5ab8-81eb-4f94-8210-a0f390052a86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54259ec6-rsllx" Oct 31 00:55:00.112714 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:00.112688 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10bf9d7a-5f72-4d13-a167-38cc2b204282-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e83b6db1-gmc99\" (UID: \"10bf9d7a-5f72-4d13-a167-38cc2b204282\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e83b6db1-gmc99" Oct 31 00:55:00.114089 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:00.114062 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10bf9d7a-5f72-4d13-a167-38cc2b204282-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e83b6db1-gmc99\" (UID: \"10bf9d7a-5f72-4d13-a167-38cc2b204282\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e83b6db1-gmc99" Oct 31 00:55:00.114089 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:00.114075 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/328b5ab8-81eb-4f94-8210-a0f390052a86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54259ec6-rsllx\" (UID: \"328b5ab8-81eb-4f94-8210-a0f390052a86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54259ec6-rsllx" Oct 31 00:55:00.191957 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:00.191873 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgscs\" (UniqueName: \"kubernetes.io/projected/328b5ab8-81eb-4f94-8210-a0f390052a86-kube-api-access-hgscs\") pod \"maas-api-revoke-sub-54259ec6-rsllx\" (UID: \"328b5ab8-81eb-4f94-8210-a0f390052a86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54259ec6-rsllx" Oct 31 00:55:00.191957 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:00.191903 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxslb\" (UniqueName: \"kubernetes.io/projected/10bf9d7a-5f72-4d13-a167-38cc2b204282-kube-api-access-rxslb\") pod \"maas-api-revoke-sub-e83b6db1-gmc99\" (UID: \"10bf9d7a-5f72-4d13-a167-38cc2b204282\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e83b6db1-gmc99" Oct 31 00:55:00.233926 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:00.233728 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54259ec6-rsllx" Oct 31 00:55:00.296107 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:00.295647 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e83b6db1-gmc99" Oct 31 00:55:00.550006 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:00.549976 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-a6f4a5-7b4d7b4dcc-v2hts" Oct 31 00:55:00.609488 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:55:00.609340 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod328b5ab8_81eb_4f94_8210_a0f390052a86.slice/crio-e6a5515ec5f05c923fc2e1a89507d6b9cdd4e6c4d5a9a31fe7c065f7d0d907d4 WatchSource:0}: Error finding container e6a5515ec5f05c923fc2e1a89507d6b9cdd4e6c4d5a9a31fe7c065f7d0d907d4: Status 404 returned error can't find the container with id e6a5515ec5f05c923fc2e1a89507d6b9cdd4e6c4d5a9a31fe7c065f7d0d907d4 Oct 31 00:55:00.611246 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:00.611221 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54259ec6-rsllx"] Oct 31 00:55:00.618999 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:00.618972 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37865b42-ea0f-4744-ab63-9dbc9b7f74b1-kube-api-access-9bcm2\" (UniqueName: \"kubernetes.io/projected/37865b42-ea0f-4744-ab63-9dbc9b7f74b1-kube-api-access-9bcm2\") pod \"37865b42-ea0f-4744-ab63-9dbc9b7f74b1\" (UID: \"37865b42-ea0f-4744-ab63-9dbc9b7f74b1\") " Oct 31 00:55:00.619146 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:00.619121 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37865b42-ea0f-4744-ab63-9dbc9b7f74b1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37865b42-ea0f-4744-ab63-9dbc9b7f74b1-plugins-config-volume\") pod \"37865b42-ea0f-4744-ab63-9dbc9b7f74b1\" (UID: \"37865b42-ea0f-4744-ab63-9dbc9b7f74b1\") " Oct 31 00:55:00.619527 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:00.619502 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37865b42-ea0f-4744-ab63-9dbc9b7f74b1-plugins-config-volume" pod "37865b42-ea0f-4744-ab63-9dbc9b7f74b1" (UID: "37865b42-ea0f-4744-ab63-9dbc9b7f74b1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:55:00.619697 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:00.619677 5222 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37865b42-ea0f-4744-ab63-9dbc9b7f74b1-plugins-config-volume\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:00.621835 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:00.621807 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37865b42-ea0f-4744-ab63-9dbc9b7f74b1-kube-api-access-9bcm2" pod "37865b42-ea0f-4744-ab63-9dbc9b7f74b1" (UID: "37865b42-ea0f-4744-ab63-9dbc9b7f74b1"). InnerVolumeSpecName "kube-api-access-9bcm2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:55:00.677078 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:00.677048 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a6f4a5-7666bb86c-glmj2" Oct 31 00:55:00.711812 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:00.711777 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e83b6db1-gmc99"] Oct 31 00:55:00.720971 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:00.720942 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91cb96ab-e55e-463f-9fe0-059e65184a2c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91cb96ab-e55e-463f-9fe0-059e65184a2c-plugins-config-volume\") pod \"91cb96ab-e55e-463f-9fe0-059e65184a2c\" (UID: \"91cb96ab-e55e-463f-9fe0-059e65184a2c\") " Oct 31 00:55:00.721217 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:00.721200 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91cb96ab-e55e-463f-9fe0-059e65184a2c-kube-api-access-9tvjl\" (UniqueName: \"kubernetes.io/projected/91cb96ab-e55e-463f-9fe0-059e65184a2c-kube-api-access-9tvjl\") pod \"91cb96ab-e55e-463f-9fe0-059e65184a2c\" (UID: \"91cb96ab-e55e-463f-9fe0-059e65184a2c\") " Oct 31 00:55:00.721292 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:00.721265 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91cb96ab-e55e-463f-9fe0-059e65184a2c-plugins-config-volume" pod "91cb96ab-e55e-463f-9fe0-059e65184a2c" (UID: "91cb96ab-e55e-463f-9fe0-059e65184a2c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:55:00.721812 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:00.721753 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bcm2\" (UniqueName: \"kubernetes.io/projected/37865b42-ea0f-4744-ab63-9dbc9b7f74b1-kube-api-access-9bcm2\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:00.721812 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:00.721781 5222 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91cb96ab-e55e-463f-9fe0-059e65184a2c-plugins-config-volume\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:00.723533 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:00.723509 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91cb96ab-e55e-463f-9fe0-059e65184a2c-kube-api-access-9tvjl" pod "91cb96ab-e55e-463f-9fe0-059e65184a2c" (UID: "91cb96ab-e55e-463f-9fe0-059e65184a2c"). InnerVolumeSpecName "kube-api-access-9tvjl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:55:00.827823 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:00.824315 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tvjl\" (UniqueName: \"kubernetes.io/projected/91cb96ab-e55e-463f-9fe0-059e65184a2c-kube-api-access-9tvjl\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:01.374915 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:01.374818 5222 generic.go:356] "Generic (PLEG): container finished" podID="91cb96ab-e55e-463f-9fe0-059e65184a2c" containerID="1383fa36b66f6c16b4a2c7b462b5504bf6c56f6de4fa8d25b95849d2ff1a4800" exitCode=0 Oct 31 00:55:01.374915 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:01.374872 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a6f4a5-7666bb86c-glmj2" Oct 31 00:55:01.375525 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:01.374913 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a6f4a5-7666bb86c-glmj2" event={"ID":"91cb96ab-e55e-463f-9fe0-059e65184a2c","Type":"ContainerDied","Data":"1383fa36b66f6c16b4a2c7b462b5504bf6c56f6de4fa8d25b95849d2ff1a4800"} Oct 31 00:55:01.375525 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:01.374985 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a6f4a5-7666bb86c-glmj2" event={"ID":"91cb96ab-e55e-463f-9fe0-059e65184a2c","Type":"ContainerDied","Data":"ddf9fe68f8abf47d3cb3cc8d5a48b70d927a8426631e4d8f62b221bb9d5a3342"} Oct 31 00:55:01.375525 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:01.375017 5222 scope.go:122] "RemoveContainer" containerID="1383fa36b66f6c16b4a2c7b462b5504bf6c56f6de4fa8d25b95849d2ff1a4800" Oct 31 00:55:01.376736 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:01.376703 5222 generic.go:356] "Generic (PLEG): container finished" podID="37865b42-ea0f-4744-ab63-9dbc9b7f74b1" containerID="0c2a1b91b0df790654027c7e739688f777a630888871a77c41e3fa155ef1407f" exitCode=0 Oct 31 00:55:01.376857 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:01.376753 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-a6f4a5-7b4d7b4dcc-v2hts" Oct 31 00:55:01.376857 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:01.376787 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-a6f4a5-7b4d7b4dcc-v2hts" event={"ID":"37865b42-ea0f-4744-ab63-9dbc9b7f74b1","Type":"ContainerDied","Data":"0c2a1b91b0df790654027c7e739688f777a630888871a77c41e3fa155ef1407f"} Oct 31 00:55:01.376857 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:01.376836 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-a6f4a5-7b4d7b4dcc-v2hts" event={"ID":"37865b42-ea0f-4744-ab63-9dbc9b7f74b1","Type":"ContainerDied","Data":"626e7bdc6144bc68cbfd611456fbebd0124d0f86806d10239747ed076fd151b4"} Oct 31 00:55:01.378719 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:01.378690 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54259ec6-rsllx" event={"ID":"328b5ab8-81eb-4f94-8210-a0f390052a86","Type":"ContainerStarted","Data":"217be6d7ac777b9d93f152f122e0c627cb6028ba22975c4b5a4be8420441a3b8"} Oct 31 00:55:01.378862 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:01.378747 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54259ec6-rsllx" event={"ID":"328b5ab8-81eb-4f94-8210-a0f390052a86","Type":"ContainerStarted","Data":"e6a5515ec5f05c923fc2e1a89507d6b9cdd4e6c4d5a9a31fe7c065f7d0d907d4"} Oct 31 00:55:01.380441 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:01.380406 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e83b6db1-gmc99" event={"ID":"10bf9d7a-5f72-4d13-a167-38cc2b204282","Type":"ContainerStarted","Data":"212a202fb631f69b536129a01a56c9fe3979c7ab86f69391a1e95212c31f65d7"} Oct 31 00:55:01.380546 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:01.380470 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e83b6db1-gmc99" event={"ID":"10bf9d7a-5f72-4d13-a167-38cc2b204282","Type":"ContainerStarted","Data":"5ca5e15f1ee6c0cf8a2214e2f7dca1f7fa97bec1daf8a0be9f93d77080241361"} Oct 31 00:55:01.392522 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:01.392502 5222 scope.go:122] "RemoveContainer" containerID="1383fa36b66f6c16b4a2c7b462b5504bf6c56f6de4fa8d25b95849d2ff1a4800" Oct 31 00:55:01.392858 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:55:01.392839 5222 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1383fa36b66f6c16b4a2c7b462b5504bf6c56f6de4fa8d25b95849d2ff1a4800\": container with ID starting with 1383fa36b66f6c16b4a2c7b462b5504bf6c56f6de4fa8d25b95849d2ff1a4800 not found: ID does not exist" containerID="1383fa36b66f6c16b4a2c7b462b5504bf6c56f6de4fa8d25b95849d2ff1a4800" Oct 31 00:55:01.392910 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:01.392870 5222 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1383fa36b66f6c16b4a2c7b462b5504bf6c56f6de4fa8d25b95849d2ff1a4800"} err="failed to get container status \"1383fa36b66f6c16b4a2c7b462b5504bf6c56f6de4fa8d25b95849d2ff1a4800\": rpc error: code = NotFound desc = could not find container \"1383fa36b66f6c16b4a2c7b462b5504bf6c56f6de4fa8d25b95849d2ff1a4800\": container with ID starting with 1383fa36b66f6c16b4a2c7b462b5504bf6c56f6de4fa8d25b95849d2ff1a4800 not found: ID does not exist" Oct 31 00:55:01.392910 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:01.392889 5222 scope.go:122] "RemoveContainer" containerID="0c2a1b91b0df790654027c7e739688f777a630888871a77c41e3fa155ef1407f" Oct 31 00:55:01.402750 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:01.402733 5222 scope.go:122] "RemoveContainer" containerID="0c2a1b91b0df790654027c7e739688f777a630888871a77c41e3fa155ef1407f" Oct 31 00:55:01.403055 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:55:01.403032 5222 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c2a1b91b0df790654027c7e739688f777a630888871a77c41e3fa155ef1407f\": container with ID starting with 0c2a1b91b0df790654027c7e739688f777a630888871a77c41e3fa155ef1407f not found: ID does not exist" containerID="0c2a1b91b0df790654027c7e739688f777a630888871a77c41e3fa155ef1407f" Oct 31 00:55:01.403105 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:01.403063 5222 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c2a1b91b0df790654027c7e739688f777a630888871a77c41e3fa155ef1407f"} err="failed to get container status \"0c2a1b91b0df790654027c7e739688f777a630888871a77c41e3fa155ef1407f\": rpc error: code = NotFound desc = could not find container \"0c2a1b91b0df790654027c7e739688f777a630888871a77c41e3fa155ef1407f\": container with ID starting with 0c2a1b91b0df790654027c7e739688f777a630888871a77c41e3fa155ef1407f not found: ID does not exist" Oct 31 00:55:01.548317 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:01.548258 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54259ec6-rsllx" podStartSLOduration=2.548244669 podStartE2EDuration="2.548244669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:54:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:55:01.546796296 +0000 UTC m=+2394.017966045" watchObservedRunningTime="2026-10-31 00:55:01.548244669 +0000 UTC m=+2394.019414417" Oct 31 00:55:01.662269 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:01.662191 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-a6f4a5-7666bb86c-glmj2"] Oct 31 00:55:01.666314 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:01.666281 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-a6f4a5-7666bb86c-glmj2"] Oct 31 00:55:01.781242 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:01.781192 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e83b6db1-gmc99" podStartSLOduration=2.7811780539999997 podStartE2EDuration="2.781178054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:54:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:55:01.780930394 +0000 UTC m=+2394.252100145" watchObservedRunningTime="2026-10-31 00:55:01.781178054 +0000 UTC m=+2394.252347803" Oct 31 00:55:01.900651 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:01.900573 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-a6f4a5-7b4d7b4dcc-v2hts"] Oct 31 00:55:01.903845 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:01.903821 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-a6f4a5-7b4d7b4dcc-v2hts"] Oct 31 00:55:02.388865 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:02.388833 5222 generic.go:356] "Generic (PLEG): container finished" podID="10bf9d7a-5f72-4d13-a167-38cc2b204282" containerID="212a202fb631f69b536129a01a56c9fe3979c7ab86f69391a1e95212c31f65d7" exitCode=0 Oct 31 00:55:02.389363 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:02.388927 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e83b6db1-gmc99" event={"ID":"10bf9d7a-5f72-4d13-a167-38cc2b204282","Type":"ContainerDied","Data":"212a202fb631f69b536129a01a56c9fe3979c7ab86f69391a1e95212c31f65d7"} Oct 31 00:55:02.392693 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:02.392661 5222 generic.go:356] "Generic (PLEG): container finished" podID="328b5ab8-81eb-4f94-8210-a0f390052a86" containerID="217be6d7ac777b9d93f152f122e0c627cb6028ba22975c4b5a4be8420441a3b8" exitCode=0 Oct 31 00:55:02.392833 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:02.392732 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54259ec6-rsllx" event={"ID":"328b5ab8-81eb-4f94-8210-a0f390052a86","Type":"ContainerDied","Data":"217be6d7ac777b9d93f152f122e0c627cb6028ba22975c4b5a4be8420441a3b8"} Oct 31 00:55:03.545817 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:03.545791 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54259ec6-rsllx" Oct 31 00:55:03.589679 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:03.589651 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e83b6db1-gmc99" Oct 31 00:55:03.656354 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:03.656275 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37865b42-ea0f-4744-ab63-9dbc9b7f74b1" path="/var/lib/kubelet/pods/37865b42-ea0f-4744-ab63-9dbc9b7f74b1/volumes" Oct 31 00:55:03.656739 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:03.656724 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91cb96ab-e55e-463f-9fe0-059e65184a2c" path="/var/lib/kubelet/pods/91cb96ab-e55e-463f-9fe0-059e65184a2c/volumes" Oct 31 00:55:03.657429 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:03.657405 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/328b5ab8-81eb-4f94-8210-a0f390052a86-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/328b5ab8-81eb-4f94-8210-a0f390052a86-maas-api-openshift-ca\") pod \"328b5ab8-81eb-4f94-8210-a0f390052a86\" (UID: \"328b5ab8-81eb-4f94-8210-a0f390052a86\") " Oct 31 00:55:03.657564 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:03.657517 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10bf9d7a-5f72-4d13-a167-38cc2b204282-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10bf9d7a-5f72-4d13-a167-38cc2b204282-maas-api-xks-ca\") pod \"10bf9d7a-5f72-4d13-a167-38cc2b204282\" (UID: \"10bf9d7a-5f72-4d13-a167-38cc2b204282\") " Oct 31 00:55:03.657564 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:03.657557 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10bf9d7a-5f72-4d13-a167-38cc2b204282-kube-api-access-rxslb\" (UniqueName: \"kubernetes.io/projected/10bf9d7a-5f72-4d13-a167-38cc2b204282-kube-api-access-rxslb\") pod \"10bf9d7a-5f72-4d13-a167-38cc2b204282\" (UID: \"10bf9d7a-5f72-4d13-a167-38cc2b204282\") " Oct 31 00:55:03.657681 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:03.657575 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/328b5ab8-81eb-4f94-8210-a0f390052a86-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/328b5ab8-81eb-4f94-8210-a0f390052a86-maas-api-xks-ca\") pod \"328b5ab8-81eb-4f94-8210-a0f390052a86\" (UID: \"328b5ab8-81eb-4f94-8210-a0f390052a86\") " Oct 31 00:55:03.657681 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:03.657622 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/328b5ab8-81eb-4f94-8210-a0f390052a86-kube-api-access-hgscs\" (UniqueName: \"kubernetes.io/projected/328b5ab8-81eb-4f94-8210-a0f390052a86-kube-api-access-hgscs\") pod \"328b5ab8-81eb-4f94-8210-a0f390052a86\" (UID: \"328b5ab8-81eb-4f94-8210-a0f390052a86\") " Oct 31 00:55:03.657681 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:03.657639 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10bf9d7a-5f72-4d13-a167-38cc2b204282-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10bf9d7a-5f72-4d13-a167-38cc2b204282-maas-api-openshift-ca\") pod \"10bf9d7a-5f72-4d13-a167-38cc2b204282\" (UID: \"10bf9d7a-5f72-4d13-a167-38cc2b204282\") " Oct 31 00:55:03.657842 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:03.657822 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/328b5ab8-81eb-4f94-8210-a0f390052a86-maas-api-openshift-ca" pod "328b5ab8-81eb-4f94-8210-a0f390052a86" (UID: "328b5ab8-81eb-4f94-8210-a0f390052a86"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:55:03.658232 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:03.658206 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/328b5ab8-81eb-4f94-8210-a0f390052a86-maas-api-openshift-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:03.659277 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:03.659237 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10bf9d7a-5f72-4d13-a167-38cc2b204282-maas-api-openshift-ca" pod "10bf9d7a-5f72-4d13-a167-38cc2b204282" (UID: "10bf9d7a-5f72-4d13-a167-38cc2b204282"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:55:03.660467 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:03.660439 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/328b5ab8-81eb-4f94-8210-a0f390052a86-maas-api-xks-ca" pod "328b5ab8-81eb-4f94-8210-a0f390052a86" (UID: "328b5ab8-81eb-4f94-8210-a0f390052a86"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:55:03.660588 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:03.660524 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/328b5ab8-81eb-4f94-8210-a0f390052a86-kube-api-access-hgscs" pod "328b5ab8-81eb-4f94-8210-a0f390052a86" (UID: "328b5ab8-81eb-4f94-8210-a0f390052a86"). InnerVolumeSpecName "kube-api-access-hgscs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:55:03.661063 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:03.661040 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10bf9d7a-5f72-4d13-a167-38cc2b204282-kube-api-access-rxslb" pod "10bf9d7a-5f72-4d13-a167-38cc2b204282" (UID: "10bf9d7a-5f72-4d13-a167-38cc2b204282"). InnerVolumeSpecName "kube-api-access-rxslb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:55:03.661146 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:03.661091 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10bf9d7a-5f72-4d13-a167-38cc2b204282-maas-api-xks-ca" pod "10bf9d7a-5f72-4d13-a167-38cc2b204282" (UID: "10bf9d7a-5f72-4d13-a167-38cc2b204282"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:55:03.759238 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:03.759193 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10bf9d7a-5f72-4d13-a167-38cc2b204282-maas-api-xks-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:03.759238 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:03.759225 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxslb\" (UniqueName: \"kubernetes.io/projected/10bf9d7a-5f72-4d13-a167-38cc2b204282-kube-api-access-rxslb\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:03.759238 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:03.759236 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/328b5ab8-81eb-4f94-8210-a0f390052a86-maas-api-xks-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:03.759238 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:03.759247 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgscs\" (UniqueName: \"kubernetes.io/projected/328b5ab8-81eb-4f94-8210-a0f390052a86-kube-api-access-hgscs\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:03.759564 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:03.759255 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10bf9d7a-5f72-4d13-a167-38cc2b204282-maas-api-openshift-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:04.416074 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:04.416037 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54259ec6-rsllx" Oct 31 00:55:04.416287 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:04.416062 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54259ec6-rsllx" event={"ID":"328b5ab8-81eb-4f94-8210-a0f390052a86","Type":"ContainerDied","Data":"e6a5515ec5f05c923fc2e1a89507d6b9cdd4e6c4d5a9a31fe7c065f7d0d907d4"} Oct 31 00:55:04.416287 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:04.416157 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e6a5515ec5f05c923fc2e1a89507d6b9cdd4e6c4d5a9a31fe7c065f7d0d907d4" Oct 31 00:55:04.417664 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:04.417638 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e83b6db1-gmc99" event={"ID":"10bf9d7a-5f72-4d13-a167-38cc2b204282","Type":"ContainerDied","Data":"5ca5e15f1ee6c0cf8a2214e2f7dca1f7fa97bec1daf8a0be9f93d77080241361"} Oct 31 00:55:04.417664 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:04.417656 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e83b6db1-gmc99" Oct 31 00:55:04.417843 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:04.417675 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5ca5e15f1ee6c0cf8a2214e2f7dca1f7fa97bec1daf8a0be9f93d77080241361" Oct 31 00:55:05.890842 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:55:05.890804 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 00:55:08.052046 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:08.052014 5222 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h9hsf_0f8c3c03-a1fc-4376-98c6-feeeefa08068/ovn-acl-logging/0.log" Oct 31 00:55:08.060505 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:08.060483 5222 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h9hsf_0f8c3c03-a1fc-4376-98c6-feeeefa08068/ovn-acl-logging/0.log" Oct 31 00:55:09.788861 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:09.788826 5222 scope.go:122] "RemoveContainer" containerID="c6dfa239dbf3b9a61e5fba7509c5c1050ad1b5b51d04c39111f44f5a75cf7405" Oct 31 00:55:11.722993 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:11.722960 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32633ca9-nmg42"] Oct 31 00:55:11.723531 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:11.723515 5222 cpu_manager.go:414] "containerMap: removing container" podUID="37865b42-ea0f-4744-ab63-9dbc9b7f74b1" containerName="payload-processing" Oct 31 00:55:11.723581 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:11.723532 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37865b42-ea0f-4744-ab63-9dbc9b7f74b1" containerName="payload-processing" Oct 31 00:55:11.723581 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:11.723544 5222 cpu_manager.go:414] "containerMap: removing container" podUID="10bf9d7a-5f72-4d13-a167-38cc2b204282" containerName="revoke-keys" Oct 31 00:55:11.723581 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:11.723549 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10bf9d7a-5f72-4d13-a167-38cc2b204282" containerName="revoke-keys" Oct 31 00:55:11.723581 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:11.723563 5222 cpu_manager.go:414] "containerMap: removing container" podUID="91cb96ab-e55e-463f-9fe0-059e65184a2c" containerName="payload-pre-processing" Oct 31 00:55:11.723581 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:11.723569 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91cb96ab-e55e-463f-9fe0-059e65184a2c" containerName="payload-pre-processing" Oct 31 00:55:11.723748 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:11.723598 5222 cpu_manager.go:414] "containerMap: removing container" podUID="328b5ab8-81eb-4f94-8210-a0f390052a86" containerName="revoke-keys" Oct 31 00:55:11.723748 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:11.723603 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="328b5ab8-81eb-4f94-8210-a0f390052a86" containerName="revoke-keys" Oct 31 00:55:11.723748 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:11.723697 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="328b5ab8-81eb-4f94-8210-a0f390052a86" containerName="revoke-keys" Oct 31 00:55:11.723748 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:11.723710 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="37865b42-ea0f-4744-ab63-9dbc9b7f74b1" containerName="payload-processing" Oct 31 00:55:11.723748 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:11.723716 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="10bf9d7a-5f72-4d13-a167-38cc2b204282" containerName="revoke-keys" Oct 31 00:55:11.723748 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:11.723724 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="91cb96ab-e55e-463f-9fe0-059e65184a2c" containerName="payload-pre-processing" Oct 31 00:55:11.729114 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:11.729096 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32633ca9-nmg42" Oct 31 00:55:11.780267 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:11.780224 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32633ca9-nmg42"] Oct 31 00:55:11.828360 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:11.828315 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:55:11.828360 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:11.828319 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:55:11.828360 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:11.828349 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:55:11.828664 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:11.828491 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfz9z\"" Oct 31 00:55:11.857320 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:11.857287 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pp2cz\" (UniqueName: \"kubernetes.io/projected/23151b79-282e-4b50-9e17-0dd63053fd4d-kube-api-access-pp2cz\") pod \"maas-api-revoke-sub-32633ca9-nmg42\" (UID: \"23151b79-282e-4b50-9e17-0dd63053fd4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32633ca9-nmg42" Oct 31 00:55:11.857320 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:11.857326 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23151b79-282e-4b50-9e17-0dd63053fd4d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-32633ca9-nmg42\" (UID: \"23151b79-282e-4b50-9e17-0dd63053fd4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32633ca9-nmg42" Oct 31 00:55:11.857628 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:11.857548 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23151b79-282e-4b50-9e17-0dd63053fd4d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-32633ca9-nmg42\" (UID: \"23151b79-282e-4b50-9e17-0dd63053fd4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32633ca9-nmg42" Oct 31 00:55:11.959334 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:11.959298 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23151b79-282e-4b50-9e17-0dd63053fd4d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-32633ca9-nmg42\" (UID: \"23151b79-282e-4b50-9e17-0dd63053fd4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32633ca9-nmg42" Oct 31 00:55:11.959550 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:11.959493 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pp2cz\" (UniqueName: \"kubernetes.io/projected/23151b79-282e-4b50-9e17-0dd63053fd4d-kube-api-access-pp2cz\") pod \"maas-api-revoke-sub-32633ca9-nmg42\" (UID: \"23151b79-282e-4b50-9e17-0dd63053fd4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32633ca9-nmg42" Oct 31 00:55:11.959550 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:11.959528 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23151b79-282e-4b50-9e17-0dd63053fd4d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-32633ca9-nmg42\" (UID: \"23151b79-282e-4b50-9e17-0dd63053fd4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32633ca9-nmg42" Oct 31 00:55:11.960118 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:11.960096 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23151b79-282e-4b50-9e17-0dd63053fd4d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-32633ca9-nmg42\" (UID: \"23151b79-282e-4b50-9e17-0dd63053fd4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32633ca9-nmg42" Oct 31 00:55:11.962113 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:11.962087 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23151b79-282e-4b50-9e17-0dd63053fd4d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-32633ca9-nmg42\" (UID: \"23151b79-282e-4b50-9e17-0dd63053fd4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32633ca9-nmg42" Oct 31 00:55:12.022090 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:12.022058 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pp2cz\" (UniqueName: \"kubernetes.io/projected/23151b79-282e-4b50-9e17-0dd63053fd4d-kube-api-access-pp2cz\") pod \"maas-api-revoke-sub-32633ca9-nmg42\" (UID: \"23151b79-282e-4b50-9e17-0dd63053fd4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32633ca9-nmg42" Oct 31 00:55:12.043976 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:12.043939 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32633ca9-nmg42" Oct 31 00:55:12.422023 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:55:12.421985 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23151b79_282e_4b50_9e17_0dd63053fd4d.slice/crio-87f3e83f324b0c395a7e5829795e3f30a6de36882d2bd3a49aa2ae9790a6435f WatchSource:0}: Error finding container 87f3e83f324b0c395a7e5829795e3f30a6de36882d2bd3a49aa2ae9790a6435f: Status 404 returned error can't find the container with id 87f3e83f324b0c395a7e5829795e3f30a6de36882d2bd3a49aa2ae9790a6435f Oct 31 00:55:12.424722 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:12.424694 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32633ca9-nmg42"] Oct 31 00:55:12.481002 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:12.480949 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32633ca9-nmg42" event={"ID":"23151b79-282e-4b50-9e17-0dd63053fd4d","Type":"ContainerStarted","Data":"87f3e83f324b0c395a7e5829795e3f30a6de36882d2bd3a49aa2ae9790a6435f"} Oct 31 00:55:13.488925 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:13.488892 5222 generic.go:356] "Generic (PLEG): container finished" podID="23151b79-282e-4b50-9e17-0dd63053fd4d" containerID="3d5893debb832fbff12b529fe662172e633bda295ea59f4b3cf55cd185e9f802" exitCode=0 Oct 31 00:55:13.489424 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:13.488971 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32633ca9-nmg42" event={"ID":"23151b79-282e-4b50-9e17-0dd63053fd4d","Type":"ContainerDied","Data":"3d5893debb832fbff12b529fe662172e633bda295ea59f4b3cf55cd185e9f802"} Oct 31 00:55:14.644497 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:14.644470 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32633ca9-nmg42" Oct 31 00:55:14.794870 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:14.794838 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23151b79-282e-4b50-9e17-0dd63053fd4d-kube-api-access-pp2cz\" (UniqueName: \"kubernetes.io/projected/23151b79-282e-4b50-9e17-0dd63053fd4d-kube-api-access-pp2cz\") pod \"23151b79-282e-4b50-9e17-0dd63053fd4d\" (UID: \"23151b79-282e-4b50-9e17-0dd63053fd4d\") " Oct 31 00:55:14.795088 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:14.794954 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23151b79-282e-4b50-9e17-0dd63053fd4d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23151b79-282e-4b50-9e17-0dd63053fd4d-maas-api-xks-ca\") pod \"23151b79-282e-4b50-9e17-0dd63053fd4d\" (UID: \"23151b79-282e-4b50-9e17-0dd63053fd4d\") " Oct 31 00:55:14.795088 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:14.795023 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23151b79-282e-4b50-9e17-0dd63053fd4d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23151b79-282e-4b50-9e17-0dd63053fd4d-maas-api-openshift-ca\") pod \"23151b79-282e-4b50-9e17-0dd63053fd4d\" (UID: \"23151b79-282e-4b50-9e17-0dd63053fd4d\") " Oct 31 00:55:14.795481 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:14.795451 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23151b79-282e-4b50-9e17-0dd63053fd4d-maas-api-openshift-ca" pod "23151b79-282e-4b50-9e17-0dd63053fd4d" (UID: "23151b79-282e-4b50-9e17-0dd63053fd4d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:55:14.797198 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:14.797168 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23151b79-282e-4b50-9e17-0dd63053fd4d-maas-api-xks-ca" pod "23151b79-282e-4b50-9e17-0dd63053fd4d" (UID: "23151b79-282e-4b50-9e17-0dd63053fd4d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:55:14.797198 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:14.797178 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23151b79-282e-4b50-9e17-0dd63053fd4d-kube-api-access-pp2cz" pod "23151b79-282e-4b50-9e17-0dd63053fd4d" (UID: "23151b79-282e-4b50-9e17-0dd63053fd4d"). InnerVolumeSpecName "kube-api-access-pp2cz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:55:14.896193 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:14.896141 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23151b79-282e-4b50-9e17-0dd63053fd4d-maas-api-xks-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:14.896193 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:14.896186 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23151b79-282e-4b50-9e17-0dd63053fd4d-maas-api-openshift-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:14.896193 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:14.896199 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pp2cz\" (UniqueName: \"kubernetes.io/projected/23151b79-282e-4b50-9e17-0dd63053fd4d-kube-api-access-pp2cz\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:15.366480 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:15.366444 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-bc2453/test-model-bc2453-kserve-d4955dd5c-59x8p"] Oct 31 00:55:15.367001 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:15.366975 5222 cpu_manager.go:414] "containerMap: removing container" podUID="23151b79-282e-4b50-9e17-0dd63053fd4d" containerName="revoke-keys" Oct 31 00:55:15.367001 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:15.366998 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23151b79-282e-4b50-9e17-0dd63053fd4d" containerName="revoke-keys" Oct 31 00:55:15.367243 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:15.367122 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="23151b79-282e-4b50-9e17-0dd63053fd4d" containerName="revoke-keys" Oct 31 00:55:15.370404 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:15.370362 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-bc2453/test-model-bc2453-kserve-d4955dd5c-59x8p" Oct 31 00:55:15.425963 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:15.425929 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-bc2453/test-model-bc2453-kserve-d4955dd5c-59x8p"] Oct 31 00:55:15.473702 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:15.473660 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-bc2453\"/\"default-dockercfg-lnxxp\"" Oct 31 00:55:15.473702 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:15.473673 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-bc2453\"/\"kube-root-ca.crt\"" Oct 31 00:55:15.473927 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:15.473696 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-bc2453\"/\"test-model-bc2453-kserve-self-signed-certs\"" Oct 31 00:55:15.473927 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:15.473724 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-bc2453\"/\"openshift-service-ca.crt\"" Oct 31 00:55:15.516459 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:15.516169 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74eae8f3-ee20-417d-ada2-d300274b2038-model-cache\") pod \"test-model-bc2453-kserve-d4955dd5c-59x8p\" (UID: \"74eae8f3-ee20-417d-ada2-d300274b2038\") " pod="ai-tenant-e2e-inf-b-bc2453/test-model-bc2453-kserve-d4955dd5c-59x8p" Oct 31 00:55:15.516459 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:15.516236 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74eae8f3-ee20-417d-ada2-d300274b2038-dshm\") pod \"test-model-bc2453-kserve-d4955dd5c-59x8p\" (UID: \"74eae8f3-ee20-417d-ada2-d300274b2038\") " pod="ai-tenant-e2e-inf-b-bc2453/test-model-bc2453-kserve-d4955dd5c-59x8p" Oct 31 00:55:15.516459 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:15.516272 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6g7p\" (UniqueName: \"kubernetes.io/projected/74eae8f3-ee20-417d-ada2-d300274b2038-kube-api-access-w6g7p\") pod \"test-model-bc2453-kserve-d4955dd5c-59x8p\" (UID: \"74eae8f3-ee20-417d-ada2-d300274b2038\") " pod="ai-tenant-e2e-inf-b-bc2453/test-model-bc2453-kserve-d4955dd5c-59x8p" Oct 31 00:55:15.516459 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:15.516300 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74eae8f3-ee20-417d-ada2-d300274b2038-tmp-dir\") pod \"test-model-bc2453-kserve-d4955dd5c-59x8p\" (UID: \"74eae8f3-ee20-417d-ada2-d300274b2038\") " pod="ai-tenant-e2e-inf-b-bc2453/test-model-bc2453-kserve-d4955dd5c-59x8p" Oct 31 00:55:15.517363 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:15.516953 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74eae8f3-ee20-417d-ada2-d300274b2038-tls-certs\") pod \"test-model-bc2453-kserve-d4955dd5c-59x8p\" (UID: \"74eae8f3-ee20-417d-ada2-d300274b2038\") " pod="ai-tenant-e2e-inf-b-bc2453/test-model-bc2453-kserve-d4955dd5c-59x8p" Oct 31 00:55:15.517363 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:15.517086 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74eae8f3-ee20-417d-ada2-d300274b2038-home\") pod \"test-model-bc2453-kserve-d4955dd5c-59x8p\" (UID: \"74eae8f3-ee20-417d-ada2-d300274b2038\") " pod="ai-tenant-e2e-inf-b-bc2453/test-model-bc2453-kserve-d4955dd5c-59x8p" Oct 31 00:55:15.550829 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:15.550780 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32633ca9-nmg42" event={"ID":"23151b79-282e-4b50-9e17-0dd63053fd4d","Type":"ContainerDied","Data":"87f3e83f324b0c395a7e5829795e3f30a6de36882d2bd3a49aa2ae9790a6435f"} Oct 31 00:55:15.551215 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:15.551194 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="87f3e83f324b0c395a7e5829795e3f30a6de36882d2bd3a49aa2ae9790a6435f" Oct 31 00:55:15.551347 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:15.550819 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32633ca9-nmg42" Oct 31 00:55:15.618415 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:15.618301 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74eae8f3-ee20-417d-ada2-d300274b2038-model-cache\") pod \"test-model-bc2453-kserve-d4955dd5c-59x8p\" (UID: \"74eae8f3-ee20-417d-ada2-d300274b2038\") " pod="ai-tenant-e2e-inf-b-bc2453/test-model-bc2453-kserve-d4955dd5c-59x8p" Oct 31 00:55:15.618415 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:15.618388 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74eae8f3-ee20-417d-ada2-d300274b2038-dshm\") pod \"test-model-bc2453-kserve-d4955dd5c-59x8p\" (UID: \"74eae8f3-ee20-417d-ada2-d300274b2038\") " pod="ai-tenant-e2e-inf-b-bc2453/test-model-bc2453-kserve-d4955dd5c-59x8p" Oct 31 00:55:15.618684 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:15.618431 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6g7p\" (UniqueName: \"kubernetes.io/projected/74eae8f3-ee20-417d-ada2-d300274b2038-kube-api-access-w6g7p\") pod \"test-model-bc2453-kserve-d4955dd5c-59x8p\" (UID: \"74eae8f3-ee20-417d-ada2-d300274b2038\") " pod="ai-tenant-e2e-inf-b-bc2453/test-model-bc2453-kserve-d4955dd5c-59x8p" Oct 31 00:55:15.618684 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:15.618457 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74eae8f3-ee20-417d-ada2-d300274b2038-tmp-dir\") pod \"test-model-bc2453-kserve-d4955dd5c-59x8p\" (UID: \"74eae8f3-ee20-417d-ada2-d300274b2038\") " pod="ai-tenant-e2e-inf-b-bc2453/test-model-bc2453-kserve-d4955dd5c-59x8p" Oct 31 00:55:15.618684 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:15.618537 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74eae8f3-ee20-417d-ada2-d300274b2038-tls-certs\") pod \"test-model-bc2453-kserve-d4955dd5c-59x8p\" (UID: \"74eae8f3-ee20-417d-ada2-d300274b2038\") " pod="ai-tenant-e2e-inf-b-bc2453/test-model-bc2453-kserve-d4955dd5c-59x8p" Oct 31 00:55:15.618684 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:15.618610 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74eae8f3-ee20-417d-ada2-d300274b2038-home\") pod \"test-model-bc2453-kserve-d4955dd5c-59x8p\" (UID: \"74eae8f3-ee20-417d-ada2-d300274b2038\") " pod="ai-tenant-e2e-inf-b-bc2453/test-model-bc2453-kserve-d4955dd5c-59x8p" Oct 31 00:55:15.618903 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:15.618711 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74eae8f3-ee20-417d-ada2-d300274b2038-model-cache\") pod \"test-model-bc2453-kserve-d4955dd5c-59x8p\" (UID: \"74eae8f3-ee20-417d-ada2-d300274b2038\") " pod="ai-tenant-e2e-inf-b-bc2453/test-model-bc2453-kserve-d4955dd5c-59x8p" Oct 31 00:55:15.619027 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:15.618971 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74eae8f3-ee20-417d-ada2-d300274b2038-home\") pod \"test-model-bc2453-kserve-d4955dd5c-59x8p\" (UID: \"74eae8f3-ee20-417d-ada2-d300274b2038\") " pod="ai-tenant-e2e-inf-b-bc2453/test-model-bc2453-kserve-d4955dd5c-59x8p" Oct 31 00:55:15.619027 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:15.618998 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74eae8f3-ee20-417d-ada2-d300274b2038-tmp-dir\") pod \"test-model-bc2453-kserve-d4955dd5c-59x8p\" (UID: \"74eae8f3-ee20-417d-ada2-d300274b2038\") " pod="ai-tenant-e2e-inf-b-bc2453/test-model-bc2453-kserve-d4955dd5c-59x8p" Oct 31 00:55:15.621208 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:15.621176 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74eae8f3-ee20-417d-ada2-d300274b2038-dshm\") pod \"test-model-bc2453-kserve-d4955dd5c-59x8p\" (UID: \"74eae8f3-ee20-417d-ada2-d300274b2038\") " pod="ai-tenant-e2e-inf-b-bc2453/test-model-bc2453-kserve-d4955dd5c-59x8p" Oct 31 00:55:15.621735 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:15.621710 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74eae8f3-ee20-417d-ada2-d300274b2038-tls-certs\") pod \"test-model-bc2453-kserve-d4955dd5c-59x8p\" (UID: \"74eae8f3-ee20-417d-ada2-d300274b2038\") " pod="ai-tenant-e2e-inf-b-bc2453/test-model-bc2453-kserve-d4955dd5c-59x8p" Oct 31 00:55:15.689056 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:15.689023 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6g7p\" (UniqueName: \"kubernetes.io/projected/74eae8f3-ee20-417d-ada2-d300274b2038-kube-api-access-w6g7p\") pod \"test-model-bc2453-kserve-d4955dd5c-59x8p\" (UID: \"74eae8f3-ee20-417d-ada2-d300274b2038\") " pod="ai-tenant-e2e-inf-b-bc2453/test-model-bc2453-kserve-d4955dd5c-59x8p" Oct 31 00:55:15.983921 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:15.983830 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-bc2453/test-model-bc2453-kserve-d4955dd5c-59x8p" Oct 31 00:55:16.359648 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:55:16.359615 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod74eae8f3_ee20_417d_ada2_d300274b2038.slice/crio-f6af5b9e64c94e3927927209e16a28ee7ac8b40d0d235de2b083e8e21c5dea1f WatchSource:0}: Error finding container f6af5b9e64c94e3927927209e16a28ee7ac8b40d0d235de2b083e8e21c5dea1f: Status 404 returned error can't find the container with id f6af5b9e64c94e3927927209e16a28ee7ac8b40d0d235de2b083e8e21c5dea1f Oct 31 00:55:16.363098 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:16.363076 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-bc2453/test-model-bc2453-kserve-d4955dd5c-59x8p"] Oct 31 00:55:16.559221 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:16.559186 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-bc2453/test-model-bc2453-kserve-d4955dd5c-59x8p" event={"ID":"74eae8f3-ee20-417d-ada2-d300274b2038","Type":"ContainerStarted","Data":"f45c11c0cf2dc1d65b1e9e07d64670ae52595e09b81c462ec56f180594c0899f"} Oct 31 00:55:16.559474 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:16.559234 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-bc2453/test-model-bc2453-kserve-d4955dd5c-59x8p" event={"ID":"74eae8f3-ee20-417d-ada2-d300274b2038","Type":"ContainerStarted","Data":"f6af5b9e64c94e3927927209e16a28ee7ac8b40d0d235de2b083e8e21c5dea1f"} Oct 31 00:55:16.681482 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:16.681359 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-bc2453/test-model-bc2453-kserve-d4955dd5c-59x8p" podStartSLOduration=1.681343367 podStartE2EDuration="1.681343367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:55:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:55:16.67877154 +0000 UTC m=+2409.149941292" watchObservedRunningTime="2026-10-31 00:55:16.681343367 +0000 UTC m=+2409.152513115" Oct 31 00:55:16.984920 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:16.984817 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-bc2453/test-model-bc2453-kserve-d4955dd5c-59x8p" Oct 31 00:55:21.180240 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:21.180058 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-lzwsk"] Oct 31 00:55:21.184715 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:21.184691 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-lzwsk" Oct 31 00:55:21.193291 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:21.193257 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7747c507-e71c-45eb-8ffb-38b61253999c-credential-socket\") pod \"e2e-worker-w3-1e5e75-openshift-default-879c67bc5-lzwsk\" (UID: \"7747c507-e71c-45eb-8ffb-38b61253999c\") " pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-lzwsk" Oct 31 00:55:21.193444 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:21.193300 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4ndj\" (UniqueName: \"kubernetes.io/projected/7747c507-e71c-45eb-8ffb-38b61253999c-kube-api-access-d4ndj\") pod \"e2e-worker-w3-1e5e75-openshift-default-879c67bc5-lzwsk\" (UID: \"7747c507-e71c-45eb-8ffb-38b61253999c\") " pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-lzwsk" Oct 31 00:55:21.193444 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:21.193321 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7747c507-e71c-45eb-8ffb-38b61253999c-istio-envoy\") pod \"e2e-worker-w3-1e5e75-openshift-default-879c67bc5-lzwsk\" (UID: \"7747c507-e71c-45eb-8ffb-38b61253999c\") " pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-lzwsk" Oct 31 00:55:21.193546 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:21.193454 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7747c507-e71c-45eb-8ffb-38b61253999c-workload-socket\") pod \"e2e-worker-w3-1e5e75-openshift-default-879c67bc5-lzwsk\" (UID: \"7747c507-e71c-45eb-8ffb-38b61253999c\") " pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-lzwsk" Oct 31 00:55:21.193546 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:21.193491 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7747c507-e71c-45eb-8ffb-38b61253999c-istio-token\") pod \"e2e-worker-w3-1e5e75-openshift-default-879c67bc5-lzwsk\" (UID: \"7747c507-e71c-45eb-8ffb-38b61253999c\") " pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-lzwsk" Oct 31 00:55:21.193546 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:21.193527 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7747c507-e71c-45eb-8ffb-38b61253999c-istio-podinfo\") pod \"e2e-worker-w3-1e5e75-openshift-default-879c67bc5-lzwsk\" (UID: \"7747c507-e71c-45eb-8ffb-38b61253999c\") " pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-lzwsk" Oct 31 00:55:21.193673 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:21.193636 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7747c507-e71c-45eb-8ffb-38b61253999c-istio-data\") pod \"e2e-worker-w3-1e5e75-openshift-default-879c67bc5-lzwsk\" (UID: \"7747c507-e71c-45eb-8ffb-38b61253999c\") " pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-lzwsk" Oct 31 00:55:21.193673 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:21.193668 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7747c507-e71c-45eb-8ffb-38b61253999c-workload-certs\") pod \"e2e-worker-w3-1e5e75-openshift-default-879c67bc5-lzwsk\" (UID: \"7747c507-e71c-45eb-8ffb-38b61253999c\") " pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-lzwsk" Oct 31 00:55:21.193756 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:21.193694 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7747c507-e71c-45eb-8ffb-38b61253999c-istiod-ca-cert\") pod \"e2e-worker-w3-1e5e75-openshift-default-879c67bc5-lzwsk\" (UID: \"7747c507-e71c-45eb-8ffb-38b61253999c\") " pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-lzwsk" Oct 31 00:55:21.240059 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:21.240026 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-1e5e75-openshift-default-dockercfg-gw4mk\"" Oct 31 00:55:21.293950 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:21.293909 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-lzwsk"] Oct 31 00:55:21.294483 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:21.294460 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7747c507-e71c-45eb-8ffb-38b61253999c-credential-socket\") pod \"e2e-worker-w3-1e5e75-openshift-default-879c67bc5-lzwsk\" (UID: \"7747c507-e71c-45eb-8ffb-38b61253999c\") " pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-lzwsk" Oct 31 00:55:21.294577 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:21.294492 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4ndj\" (UniqueName: \"kubernetes.io/projected/7747c507-e71c-45eb-8ffb-38b61253999c-kube-api-access-d4ndj\") pod \"e2e-worker-w3-1e5e75-openshift-default-879c67bc5-lzwsk\" (UID: \"7747c507-e71c-45eb-8ffb-38b61253999c\") " pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-lzwsk" Oct 31 00:55:21.294577 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:21.294535 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7747c507-e71c-45eb-8ffb-38b61253999c-istio-envoy\") pod \"e2e-worker-w3-1e5e75-openshift-default-879c67bc5-lzwsk\" (UID: \"7747c507-e71c-45eb-8ffb-38b61253999c\") " pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-lzwsk" Oct 31 00:55:21.294577 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:21.294572 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7747c507-e71c-45eb-8ffb-38b61253999c-workload-socket\") pod \"e2e-worker-w3-1e5e75-openshift-default-879c67bc5-lzwsk\" (UID: \"7747c507-e71c-45eb-8ffb-38b61253999c\") " pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-lzwsk" Oct 31 00:55:21.294754 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:21.294608 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7747c507-e71c-45eb-8ffb-38b61253999c-istio-token\") pod \"e2e-worker-w3-1e5e75-openshift-default-879c67bc5-lzwsk\" (UID: \"7747c507-e71c-45eb-8ffb-38b61253999c\") " pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-lzwsk" Oct 31 00:55:21.294754 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:21.294630 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7747c507-e71c-45eb-8ffb-38b61253999c-istio-podinfo\") pod \"e2e-worker-w3-1e5e75-openshift-default-879c67bc5-lzwsk\" (UID: \"7747c507-e71c-45eb-8ffb-38b61253999c\") " pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-lzwsk" Oct 31 00:55:21.294754 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:21.294733 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7747c507-e71c-45eb-8ffb-38b61253999c-istio-data\") pod \"e2e-worker-w3-1e5e75-openshift-default-879c67bc5-lzwsk\" (UID: \"7747c507-e71c-45eb-8ffb-38b61253999c\") " pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-lzwsk" Oct 31 00:55:21.294913 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:21.294771 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7747c507-e71c-45eb-8ffb-38b61253999c-workload-certs\") pod \"e2e-worker-w3-1e5e75-openshift-default-879c67bc5-lzwsk\" (UID: \"7747c507-e71c-45eb-8ffb-38b61253999c\") " pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-lzwsk" Oct 31 00:55:21.294913 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:21.294808 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7747c507-e71c-45eb-8ffb-38b61253999c-istiod-ca-cert\") pod \"e2e-worker-w3-1e5e75-openshift-default-879c67bc5-lzwsk\" (UID: \"7747c507-e71c-45eb-8ffb-38b61253999c\") " pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-lzwsk" Oct 31 00:55:21.295464 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:21.295410 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7747c507-e71c-45eb-8ffb-38b61253999c-workload-certs\") pod \"e2e-worker-w3-1e5e75-openshift-default-879c67bc5-lzwsk\" (UID: \"7747c507-e71c-45eb-8ffb-38b61253999c\") " pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-lzwsk" Oct 31 00:55:21.295607 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:21.295460 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7747c507-e71c-45eb-8ffb-38b61253999c-istio-data\") pod \"e2e-worker-w3-1e5e75-openshift-default-879c67bc5-lzwsk\" (UID: \"7747c507-e71c-45eb-8ffb-38b61253999c\") " pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-lzwsk" Oct 31 00:55:21.295607 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:21.295547 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7747c507-e71c-45eb-8ffb-38b61253999c-workload-socket\") pod \"e2e-worker-w3-1e5e75-openshift-default-879c67bc5-lzwsk\" (UID: \"7747c507-e71c-45eb-8ffb-38b61253999c\") " pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-lzwsk" Oct 31 00:55:21.295607 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:21.295588 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7747c507-e71c-45eb-8ffb-38b61253999c-istiod-ca-cert\") pod \"e2e-worker-w3-1e5e75-openshift-default-879c67bc5-lzwsk\" (UID: \"7747c507-e71c-45eb-8ffb-38b61253999c\") " pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-lzwsk" Oct 31 00:55:21.295866 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:21.295848 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7747c507-e71c-45eb-8ffb-38b61253999c-credential-socket\") pod \"e2e-worker-w3-1e5e75-openshift-default-879c67bc5-lzwsk\" (UID: \"7747c507-e71c-45eb-8ffb-38b61253999c\") " pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-lzwsk" Oct 31 00:55:21.297274 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:21.297254 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7747c507-e71c-45eb-8ffb-38b61253999c-istio-envoy\") pod \"e2e-worker-w3-1e5e75-openshift-default-879c67bc5-lzwsk\" (UID: \"7747c507-e71c-45eb-8ffb-38b61253999c\") " pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-lzwsk" Oct 31 00:55:21.297663 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:21.297644 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7747c507-e71c-45eb-8ffb-38b61253999c-istio-podinfo\") pod \"e2e-worker-w3-1e5e75-openshift-default-879c67bc5-lzwsk\" (UID: \"7747c507-e71c-45eb-8ffb-38b61253999c\") " pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-lzwsk" Oct 31 00:55:21.357176 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:21.357133 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7747c507-e71c-45eb-8ffb-38b61253999c-istio-token\") pod \"e2e-worker-w3-1e5e75-openshift-default-879c67bc5-lzwsk\" (UID: \"7747c507-e71c-45eb-8ffb-38b61253999c\") " pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-lzwsk" Oct 31 00:55:21.357336 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:21.357252 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4ndj\" (UniqueName: \"kubernetes.io/projected/7747c507-e71c-45eb-8ffb-38b61253999c-kube-api-access-d4ndj\") pod \"e2e-worker-w3-1e5e75-openshift-default-879c67bc5-lzwsk\" (UID: \"7747c507-e71c-45eb-8ffb-38b61253999c\") " pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-lzwsk" Oct 31 00:55:21.499500 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:21.499411 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-lzwsk" Oct 31 00:55:21.943824 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:55:21.943777 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7747c507_e71c_45eb_8ffb_38b61253999c.slice/crio-43a4b6654b906d12133e2cf080e6e65bf9e13fe0280a4ae6ca42e0e40832ed08 WatchSource:0}: Error finding container 43a4b6654b906d12133e2cf080e6e65bf9e13fe0280a4ae6ca42e0e40832ed08: Status 404 returned error can't find the container with id 43a4b6654b906d12133e2cf080e6e65bf9e13fe0280a4ae6ca42e0e40832ed08 Oct 31 00:55:21.946440 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:21.946416 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-lzwsk"] Oct 31 00:55:21.946440 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:21.946415 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:55:21.946561 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:21.946479 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:55:21.946561 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:21.946511 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:55:22.608405 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:22.608352 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-lzwsk" event={"ID":"7747c507-e71c-45eb-8ffb-38b61253999c","Type":"ContainerStarted","Data":"5d501b5bd3b7acfe5169d0990e01ea56fb56fb6a8dca96b6d2f6a87e170b14c8"} Oct 31 00:55:22.608861 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:22.608427 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-lzwsk" event={"ID":"7747c507-e71c-45eb-8ffb-38b61253999c","Type":"ContainerStarted","Data":"43a4b6654b906d12133e2cf080e6e65bf9e13fe0280a4ae6ca42e0e40832ed08"} Oct 31 00:55:22.785211 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:22.785150 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-lzwsk" podStartSLOduration=1.7851349 podStartE2EDuration="1.7851349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:55:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:55:22.782162201 +0000 UTC m=+2415.253331950" watchObservedRunningTime="2026-10-31 00:55:22.7851349 +0000 UTC m=+2415.256304649" Oct 31 00:55:23.018880 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:23.018844 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-412743-hdbhc"] Oct 31 00:55:23.022938 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:23.022917 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-412743-hdbhc" Oct 31 00:55:23.076528 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:23.076488 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-412743-hdbhc"] Oct 31 00:55:23.076775 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:23.076757 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfz9z\"" Oct 31 00:55:23.076870 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:23.076778 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:55:23.113097 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:23.113057 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjqlf\" (UniqueName: \"kubernetes.io/projected/2721a0cc-1697-4f49-8c2b-b72f84186b8b-kube-api-access-rjqlf\") pod \"maas-api-revoke-keys-e2e-worker-w5-412743-hdbhc\" (UID: \"2721a0cc-1697-4f49-8c2b-b72f84186b8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-412743-hdbhc" Oct 31 00:55:23.113276 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:23.113202 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2721a0cc-1697-4f49-8c2b-b72f84186b8b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-412743-hdbhc\" (UID: \"2721a0cc-1697-4f49-8c2b-b72f84186b8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-412743-hdbhc" Oct 31 00:55:23.113347 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:23.113331 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2721a0cc-1697-4f49-8c2b-b72f84186b8b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-412743-hdbhc\" (UID: \"2721a0cc-1697-4f49-8c2b-b72f84186b8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-412743-hdbhc" Oct 31 00:55:23.126866 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:23.126831 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:55:23.126866 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:23.126857 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:55:23.214880 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:23.214843 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2721a0cc-1697-4f49-8c2b-b72f84186b8b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-412743-hdbhc\" (UID: \"2721a0cc-1697-4f49-8c2b-b72f84186b8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-412743-hdbhc" Oct 31 00:55:23.215070 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:23.214948 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2721a0cc-1697-4f49-8c2b-b72f84186b8b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-412743-hdbhc\" (UID: \"2721a0cc-1697-4f49-8c2b-b72f84186b8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-412743-hdbhc" Oct 31 00:55:23.215070 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:23.214985 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjqlf\" (UniqueName: \"kubernetes.io/projected/2721a0cc-1697-4f49-8c2b-b72f84186b8b-kube-api-access-rjqlf\") pod \"maas-api-revoke-keys-e2e-worker-w5-412743-hdbhc\" (UID: \"2721a0cc-1697-4f49-8c2b-b72f84186b8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-412743-hdbhc" Oct 31 00:55:23.215596 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:23.215573 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2721a0cc-1697-4f49-8c2b-b72f84186b8b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-412743-hdbhc\" (UID: \"2721a0cc-1697-4f49-8c2b-b72f84186b8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-412743-hdbhc" Oct 31 00:55:23.217566 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:23.217544 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2721a0cc-1697-4f49-8c2b-b72f84186b8b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-412743-hdbhc\" (UID: \"2721a0cc-1697-4f49-8c2b-b72f84186b8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-412743-hdbhc" Oct 31 00:55:23.278390 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:23.278291 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjqlf\" (UniqueName: \"kubernetes.io/projected/2721a0cc-1697-4f49-8c2b-b72f84186b8b-kube-api-access-rjqlf\") pod \"maas-api-revoke-keys-e2e-worker-w5-412743-hdbhc\" (UID: \"2721a0cc-1697-4f49-8c2b-b72f84186b8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-412743-hdbhc" Oct 31 00:55:23.336629 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:23.336599 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-412743-hdbhc" Oct 31 00:55:23.499999 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:23.499967 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-lzwsk" Oct 31 00:55:23.501336 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:23.501310 5222 patch_prober.go:29] interesting pod/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-lzwsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" start-of-body= Oct 31 00:55:23.501442 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:23.501363 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-lzwsk" podUID="7747c507-e71c-45eb-8ffb-38b61253999c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" Oct 31 00:55:23.767637 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:23.766482 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-412743-hdbhc"] Oct 31 00:55:24.500842 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:24.500759 5222 patch_prober.go:29] interesting pod/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-lzwsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" start-of-body= Oct 31 00:55:24.500842 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:24.500812 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-lzwsk" podUID="7747c507-e71c-45eb-8ffb-38b61253999c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" Oct 31 00:55:24.625259 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:24.625224 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-412743-hdbhc" event={"ID":"2721a0cc-1697-4f49-8c2b-b72f84186b8b","Type":"ContainerStarted","Data":"29c14a3f4780eaafdc8305f58522d2dbc7ab57689dda24210abaf7059642a199"} Oct 31 00:55:24.625475 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:24.625283 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-412743-hdbhc" event={"ID":"2721a0cc-1697-4f49-8c2b-b72f84186b8b","Type":"ContainerStarted","Data":"b3809468638c8fc998ad85b502ffd38fb9f9c3a4b5d2ff4765f77bcec879b8c0"} Oct 31 00:55:24.761455 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:24.755938 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-412743-hdbhc" podStartSLOduration=2.755916429 podStartE2EDuration="2.755916429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:55:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:55:24.75074196 +0000 UTC m=+2417.221911709" watchObservedRunningTime="2026-10-31 00:55:24.755916429 +0000 UTC m=+2417.227086178" Oct 31 00:55:25.500540 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:25.500505 5222 patch_prober.go:29] interesting pod/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-lzwsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" start-of-body= Oct 31 00:55:25.500976 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:25.500560 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-lzwsk" podUID="7747c507-e71c-45eb-8ffb-38b61253999c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" Oct 31 00:55:25.662516 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:25.660915 5222 generic.go:356] "Generic (PLEG): container finished" podID="2721a0cc-1697-4f49-8c2b-b72f84186b8b" containerID="29c14a3f4780eaafdc8305f58522d2dbc7ab57689dda24210abaf7059642a199" exitCode=0 Oct 31 00:55:25.670617 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:25.670579 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-412743-hdbhc" event={"ID":"2721a0cc-1697-4f49-8c2b-b72f84186b8b","Type":"ContainerDied","Data":"29c14a3f4780eaafdc8305f58522d2dbc7ab57689dda24210abaf7059642a199"} Oct 31 00:55:25.989604 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:25.987094 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-bc2453/test-model-bc2453-kserve-d4955dd5c-59x8p" Oct 31 00:55:26.003135 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:26.003104 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-bc2453/test-model-bc2453-kserve-d4955dd5c-59x8p" Oct 31 00:55:26.501388 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:26.500842 5222 patch_prober.go:29] interesting pod/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-lzwsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" start-of-body= Oct 31 00:55:26.501388 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:26.500904 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-lzwsk" podUID="7747c507-e71c-45eb-8ffb-38b61253999c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" Oct 31 00:55:26.681201 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:26.681158 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-bc2453/test-model-bc2453-kserve-d4955dd5c-59x8p" Oct 31 00:55:26.865529 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:26.865489 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-412743-hdbhc" Oct 31 00:55:26.921847 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:26.921818 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2721a0cc-1697-4f49-8c2b-b72f84186b8b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2721a0cc-1697-4f49-8c2b-b72f84186b8b-maas-api-openshift-ca\") pod \"2721a0cc-1697-4f49-8c2b-b72f84186b8b\" (UID: \"2721a0cc-1697-4f49-8c2b-b72f84186b8b\") " Oct 31 00:55:26.922047 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:26.921995 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2721a0cc-1697-4f49-8c2b-b72f84186b8b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2721a0cc-1697-4f49-8c2b-b72f84186b8b-maas-api-xks-ca\") pod \"2721a0cc-1697-4f49-8c2b-b72f84186b8b\" (UID: \"2721a0cc-1697-4f49-8c2b-b72f84186b8b\") " Oct 31 00:55:26.922154 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:26.922139 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2721a0cc-1697-4f49-8c2b-b72f84186b8b-kube-api-access-rjqlf\" (UniqueName: \"kubernetes.io/projected/2721a0cc-1697-4f49-8c2b-b72f84186b8b-kube-api-access-rjqlf\") pod \"2721a0cc-1697-4f49-8c2b-b72f84186b8b\" (UID: \"2721a0cc-1697-4f49-8c2b-b72f84186b8b\") " Oct 31 00:55:26.922222 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:26.922179 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2721a0cc-1697-4f49-8c2b-b72f84186b8b-maas-api-openshift-ca" pod "2721a0cc-1697-4f49-8c2b-b72f84186b8b" (UID: "2721a0cc-1697-4f49-8c2b-b72f84186b8b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:55:26.922582 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:26.922563 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2721a0cc-1697-4f49-8c2b-b72f84186b8b-maas-api-openshift-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:26.924316 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:26.924293 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2721a0cc-1697-4f49-8c2b-b72f84186b8b-maas-api-xks-ca" pod "2721a0cc-1697-4f49-8c2b-b72f84186b8b" (UID: "2721a0cc-1697-4f49-8c2b-b72f84186b8b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:55:26.924435 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:26.924323 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2721a0cc-1697-4f49-8c2b-b72f84186b8b-kube-api-access-rjqlf" pod "2721a0cc-1697-4f49-8c2b-b72f84186b8b" (UID: "2721a0cc-1697-4f49-8c2b-b72f84186b8b"). InnerVolumeSpecName "kube-api-access-rjqlf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:55:27.023478 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:27.023444 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjqlf\" (UniqueName: \"kubernetes.io/projected/2721a0cc-1697-4f49-8c2b-b72f84186b8b-kube-api-access-rjqlf\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:27.023478 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:27.023475 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2721a0cc-1697-4f49-8c2b-b72f84186b8b-maas-api-xks-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:27.470146 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:27.470101 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-412743/distinct-2-w5-412743-kserve-5b679b6b66-59prz"] Oct 31 00:55:27.473562 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:27.473525 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-412743/distinct-w5-412743-kserve-64d65f7d8f-c4tfs"] Oct 31 00:55:27.500672 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:27.500631 5222 patch_prober.go:29] interesting pod/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-lzwsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" start-of-body= Oct 31 00:55:27.500852 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:27.500703 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-lzwsk" podUID="7747c507-e71c-45eb-8ffb-38b61253999c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" Oct 31 00:55:27.519118 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:27.519085 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-412743/simulator-w5-412743-kserve-6f897dd967-5bkq5"] Oct 31 00:55:27.677162 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:27.677139 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-412743-hdbhc" Oct 31 00:55:27.677328 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:27.677178 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-412743-hdbhc" event={"ID":"2721a0cc-1697-4f49-8c2b-b72f84186b8b","Type":"ContainerDied","Data":"b3809468638c8fc998ad85b502ffd38fb9f9c3a4b5d2ff4765f77bcec879b8c0"} Oct 31 00:55:27.677328 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:27.677225 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b3809468638c8fc998ad85b502ffd38fb9f9c3a4b5d2ff4765f77bcec879b8c0" Oct 31 00:55:27.765420 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:27.765386 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-1e5e75-l5vzz"] Oct 31 00:55:27.768798 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:27.768771 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-1e5e75-l5vzz"] Oct 31 00:55:28.500603 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:28.500565 5222 patch_prober.go:29] interesting pod/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-lzwsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" start-of-body= Oct 31 00:55:28.500814 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:28.500628 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-lzwsk" podUID="7747c507-e71c-45eb-8ffb-38b61253999c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" Oct 31 00:55:29.658780 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:29.658736 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fde6a27f-2b5e-4e9e-9b39-e53c294597b7" path="/var/lib/kubelet/pods/fde6a27f-2b5e-4e9e-9b39-e53c294597b7/volumes" Oct 31 00:55:29.755436 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:29.755310 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-lzwsk" Oct 31 00:55:29.755836 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:29.755804 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-lzwsk" Oct 31 00:55:29.757016 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:29.756992 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-lzwsk" Oct 31 00:55:34.496821 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:34.496783 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-412743-688d66585f-vk5r7"] Oct 31 00:55:34.605084 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:34.605054 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-412743-6d5cb5bd7f-pzb7l"] Oct 31 00:55:34.845070 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:34.845032 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-lzwsk"] Oct 31 00:55:34.845330 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:34.845300 5222 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-lzwsk" podUID="7747c507-e71c-45eb-8ffb-38b61253999c" containerName="istio-proxy" containerID="cri-o://5d501b5bd3b7acfe5169d0990e01ea56fb56fb6a8dca96b6d2f6a87e170b14c8" gracePeriod=30 Oct 31 00:55:35.917353 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:55:35.917311 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 00:55:39.224482 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:39.224452 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-101b9e-4s8tx"] Oct 31 00:55:39.225096 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:39.225079 5222 cpu_manager.go:414] "containerMap: removing container" podUID="2721a0cc-1697-4f49-8c2b-b72f84186b8b" containerName="revoke-keys" Oct 31 00:55:39.225162 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:39.225098 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2721a0cc-1697-4f49-8c2b-b72f84186b8b" containerName="revoke-keys" Oct 31 00:55:39.225220 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:39.225183 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="2721a0cc-1697-4f49-8c2b-b72f84186b8b" containerName="revoke-keys" Oct 31 00:55:39.228929 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:39.228912 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-101b9e-4s8tx" Oct 31 00:55:39.282168 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:39.282134 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-101b9e-4s8tx"] Oct 31 00:55:39.302159 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:39.302129 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6eefacd4-395f-42f5-922d-fb2d971401c5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-101b9e-4s8tx\" (UID: \"6eefacd4-395f-42f5-922d-fb2d971401c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-101b9e-4s8tx" Oct 31 00:55:39.302328 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:39.302279 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sfpp\" (UniqueName: \"kubernetes.io/projected/6eefacd4-395f-42f5-922d-fb2d971401c5-kube-api-access-4sfpp\") pod \"maas-api-revoke-keys-e2e-worker-w0-101b9e-4s8tx\" (UID: \"6eefacd4-395f-42f5-922d-fb2d971401c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-101b9e-4s8tx" Oct 31 00:55:39.302449 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:39.302347 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6eefacd4-395f-42f5-922d-fb2d971401c5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-101b9e-4s8tx\" (UID: \"6eefacd4-395f-42f5-922d-fb2d971401c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-101b9e-4s8tx" Oct 31 00:55:39.329642 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:39.329613 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:55:39.329642 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:39.329628 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:55:39.329867 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:39.329670 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfz9z\"" Oct 31 00:55:39.329867 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:39.329704 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:55:39.403846 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:39.403802 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6eefacd4-395f-42f5-922d-fb2d971401c5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-101b9e-4s8tx\" (UID: \"6eefacd4-395f-42f5-922d-fb2d971401c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-101b9e-4s8tx" Oct 31 00:55:39.404062 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:39.403866 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6eefacd4-395f-42f5-922d-fb2d971401c5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-101b9e-4s8tx\" (UID: \"6eefacd4-395f-42f5-922d-fb2d971401c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-101b9e-4s8tx" Oct 31 00:55:39.404062 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:39.403956 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sfpp\" (UniqueName: \"kubernetes.io/projected/6eefacd4-395f-42f5-922d-fb2d971401c5-kube-api-access-4sfpp\") pod \"maas-api-revoke-keys-e2e-worker-w0-101b9e-4s8tx\" (UID: \"6eefacd4-395f-42f5-922d-fb2d971401c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-101b9e-4s8tx" Oct 31 00:55:39.404563 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:39.404537 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6eefacd4-395f-42f5-922d-fb2d971401c5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-101b9e-4s8tx\" (UID: \"6eefacd4-395f-42f5-922d-fb2d971401c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-101b9e-4s8tx" Oct 31 00:55:39.406401 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:39.406361 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6eefacd4-395f-42f5-922d-fb2d971401c5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-101b9e-4s8tx\" (UID: \"6eefacd4-395f-42f5-922d-fb2d971401c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-101b9e-4s8tx" Oct 31 00:55:39.466355 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:39.466316 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sfpp\" (UniqueName: \"kubernetes.io/projected/6eefacd4-395f-42f5-922d-fb2d971401c5-kube-api-access-4sfpp\") pod \"maas-api-revoke-keys-e2e-worker-w0-101b9e-4s8tx\" (UID: \"6eefacd4-395f-42f5-922d-fb2d971401c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-101b9e-4s8tx" Oct 31 00:55:39.498139 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:39.498016 5222 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w5-412743-688d66585f-vk5r7" podUID="a99cbf31-f3f2-4d16-a665-58f0f5af77f0" containerName="payload-processing" containerID="cri-o://66d2dcee34673511c9d5bdcc236e9cac8f2e3852170c01d565e384bf000e246a" gracePeriod=25 Oct 31 00:55:39.543280 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:39.543238 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-101b9e-4s8tx" Oct 31 00:55:39.605994 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:39.605952 5222 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-412743-6d5cb5bd7f-pzb7l" podUID="9dbd9108-ac33-45a0-9eed-ef9b81755398" containerName="payload-pre-processing" containerID="cri-o://c28d50fb6af5559cb39a55913a3db1d9d96be0c43e6c5ab96f9c35b102cbecf4" gracePeriod=25 Oct 31 00:55:39.922590 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:39.922559 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-101b9e-4s8tx"] Oct 31 00:55:39.940587 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:55:39.940556 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6eefacd4_395f_42f5_922d_fb2d971401c5.slice/crio-1a2df770e4f9b161134d1ea9a6c66c02a29f70c8268f227abaa8833c581b52c4 WatchSource:0}: Error finding container 1a2df770e4f9b161134d1ea9a6c66c02a29f70c8268f227abaa8833c581b52c4: Status 404 returned error can't find the container with id 1a2df770e4f9b161134d1ea9a6c66c02a29f70c8268f227abaa8833c581b52c4 Oct 31 00:55:40.094235 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:40.094205 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-lzwsk" Oct 31 00:55:40.213403 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:40.213338 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7747c507-e71c-45eb-8ffb-38b61253999c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7747c507-e71c-45eb-8ffb-38b61253999c-credential-socket\") pod \"7747c507-e71c-45eb-8ffb-38b61253999c\" (UID: \"7747c507-e71c-45eb-8ffb-38b61253999c\") " Oct 31 00:55:40.213689 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:40.213658 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7747c507-e71c-45eb-8ffb-38b61253999c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7747c507-e71c-45eb-8ffb-38b61253999c-istio-data\") pod \"7747c507-e71c-45eb-8ffb-38b61253999c\" (UID: \"7747c507-e71c-45eb-8ffb-38b61253999c\") " Oct 31 00:55:40.214020 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:40.213836 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7747c507-e71c-45eb-8ffb-38b61253999c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7747c507-e71c-45eb-8ffb-38b61253999c-istio-envoy\") pod \"7747c507-e71c-45eb-8ffb-38b61253999c\" (UID: \"7747c507-e71c-45eb-8ffb-38b61253999c\") " Oct 31 00:55:40.214020 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:40.213857 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7747c507-e71c-45eb-8ffb-38b61253999c-credential-socket" pod "7747c507-e71c-45eb-8ffb-38b61253999c" (UID: "7747c507-e71c-45eb-8ffb-38b61253999c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:55:40.214020 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:40.213911 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7747c507-e71c-45eb-8ffb-38b61253999c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7747c507-e71c-45eb-8ffb-38b61253999c-istio-podinfo\") pod \"7747c507-e71c-45eb-8ffb-38b61253999c\" (UID: \"7747c507-e71c-45eb-8ffb-38b61253999c\") " Oct 31 00:55:40.214020 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:40.214000 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7747c507-e71c-45eb-8ffb-38b61253999c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7747c507-e71c-45eb-8ffb-38b61253999c-workload-socket\") pod \"7747c507-e71c-45eb-8ffb-38b61253999c\" (UID: \"7747c507-e71c-45eb-8ffb-38b61253999c\") " Oct 31 00:55:40.214286 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:40.214040 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7747c507-e71c-45eb-8ffb-38b61253999c-kube-api-access-d4ndj\" (UniqueName: \"kubernetes.io/projected/7747c507-e71c-45eb-8ffb-38b61253999c-kube-api-access-d4ndj\") pod \"7747c507-e71c-45eb-8ffb-38b61253999c\" (UID: \"7747c507-e71c-45eb-8ffb-38b61253999c\") " Oct 31 00:55:40.214286 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:40.214113 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7747c507-e71c-45eb-8ffb-38b61253999c-istio-token\" (UniqueName: \"kubernetes.io/projected/7747c507-e71c-45eb-8ffb-38b61253999c-istio-token\") pod \"7747c507-e71c-45eb-8ffb-38b61253999c\" (UID: \"7747c507-e71c-45eb-8ffb-38b61253999c\") " Oct 31 00:55:40.214286 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:40.214150 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7747c507-e71c-45eb-8ffb-38b61253999c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7747c507-e71c-45eb-8ffb-38b61253999c-workload-certs\") pod \"7747c507-e71c-45eb-8ffb-38b61253999c\" (UID: \"7747c507-e71c-45eb-8ffb-38b61253999c\") " Oct 31 00:55:40.214286 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:40.214187 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7747c507-e71c-45eb-8ffb-38b61253999c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7747c507-e71c-45eb-8ffb-38b61253999c-istiod-ca-cert\") pod \"7747c507-e71c-45eb-8ffb-38b61253999c\" (UID: \"7747c507-e71c-45eb-8ffb-38b61253999c\") " Oct 31 00:55:40.214800 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:40.214773 5222 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7747c507-e71c-45eb-8ffb-38b61253999c-credential-socket\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:40.214930 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:40.214889 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7747c507-e71c-45eb-8ffb-38b61253999c-workload-socket" pod "7747c507-e71c-45eb-8ffb-38b61253999c" (UID: "7747c507-e71c-45eb-8ffb-38b61253999c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:55:40.214930 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:40.214786 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7747c507-e71c-45eb-8ffb-38b61253999c-istio-data" pod "7747c507-e71c-45eb-8ffb-38b61253999c" (UID: "7747c507-e71c-45eb-8ffb-38b61253999c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:55:40.215124 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:40.215085 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7747c507-e71c-45eb-8ffb-38b61253999c-workload-certs" pod "7747c507-e71c-45eb-8ffb-38b61253999c" (UID: "7747c507-e71c-45eb-8ffb-38b61253999c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:55:40.215124 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:40.215114 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7747c507-e71c-45eb-8ffb-38b61253999c-istiod-ca-cert" pod "7747c507-e71c-45eb-8ffb-38b61253999c" (UID: "7747c507-e71c-45eb-8ffb-38b61253999c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:55:40.217285 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:40.217259 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7747c507-e71c-45eb-8ffb-38b61253999c-istio-podinfo" pod "7747c507-e71c-45eb-8ffb-38b61253999c" (UID: "7747c507-e71c-45eb-8ffb-38b61253999c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:55:40.217763 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:40.217725 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7747c507-e71c-45eb-8ffb-38b61253999c-kube-api-access-d4ndj" pod "7747c507-e71c-45eb-8ffb-38b61253999c" (UID: "7747c507-e71c-45eb-8ffb-38b61253999c"). InnerVolumeSpecName "kube-api-access-d4ndj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:55:40.218452 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:40.218420 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7747c507-e71c-45eb-8ffb-38b61253999c-istio-token" pod "7747c507-e71c-45eb-8ffb-38b61253999c" (UID: "7747c507-e71c-45eb-8ffb-38b61253999c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:55:40.219993 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:40.219969 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7747c507-e71c-45eb-8ffb-38b61253999c-istio-envoy" pod "7747c507-e71c-45eb-8ffb-38b61253999c" (UID: "7747c507-e71c-45eb-8ffb-38b61253999c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:55:40.315987 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:40.315807 5222 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7747c507-e71c-45eb-8ffb-38b61253999c-workload-certs\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:40.315987 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:40.315846 5222 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7747c507-e71c-45eb-8ffb-38b61253999c-istiod-ca-cert\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:40.315987 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:40.315859 5222 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7747c507-e71c-45eb-8ffb-38b61253999c-istio-data\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:40.315987 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:40.315873 5222 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7747c507-e71c-45eb-8ffb-38b61253999c-istio-envoy\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:40.315987 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:40.315888 5222 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7747c507-e71c-45eb-8ffb-38b61253999c-istio-podinfo\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:40.315987 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:40.315901 5222 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7747c507-e71c-45eb-8ffb-38b61253999c-workload-socket\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:40.315987 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:40.315914 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4ndj\" (UniqueName: \"kubernetes.io/projected/7747c507-e71c-45eb-8ffb-38b61253999c-kube-api-access-d4ndj\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:40.315987 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:40.315928 5222 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7747c507-e71c-45eb-8ffb-38b61253999c-istio-token\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:40.777634 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:40.777600 5222 generic.go:356] "Generic (PLEG): container finished" podID="7747c507-e71c-45eb-8ffb-38b61253999c" containerID="5d501b5bd3b7acfe5169d0990e01ea56fb56fb6a8dca96b6d2f6a87e170b14c8" exitCode=0 Oct 31 00:55:40.777866 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:40.777668 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-lzwsk" Oct 31 00:55:40.777866 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:40.777681 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-lzwsk" event={"ID":"7747c507-e71c-45eb-8ffb-38b61253999c","Type":"ContainerDied","Data":"5d501b5bd3b7acfe5169d0990e01ea56fb56fb6a8dca96b6d2f6a87e170b14c8"} Oct 31 00:55:40.777866 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:40.777738 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-lzwsk" event={"ID":"7747c507-e71c-45eb-8ffb-38b61253999c","Type":"ContainerDied","Data":"43a4b6654b906d12133e2cf080e6e65bf9e13fe0280a4ae6ca42e0e40832ed08"} Oct 31 00:55:40.777866 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:40.777764 5222 scope.go:122] "RemoveContainer" containerID="5d501b5bd3b7acfe5169d0990e01ea56fb56fb6a8dca96b6d2f6a87e170b14c8" Oct 31 00:55:40.779756 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:40.779734 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-101b9e-4s8tx" event={"ID":"6eefacd4-395f-42f5-922d-fb2d971401c5","Type":"ContainerStarted","Data":"8a361ecef0d6ddb72b61057cd2706fe90dc8fa4db98208e2c04ef70c72122c04"} Oct 31 00:55:40.779894 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:40.779773 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-101b9e-4s8tx" event={"ID":"6eefacd4-395f-42f5-922d-fb2d971401c5","Type":"ContainerStarted","Data":"1a2df770e4f9b161134d1ea9a6c66c02a29f70c8268f227abaa8833c581b52c4"} Oct 31 00:55:40.789265 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:40.789246 5222 scope.go:122] "RemoveContainer" containerID="5d501b5bd3b7acfe5169d0990e01ea56fb56fb6a8dca96b6d2f6a87e170b14c8" Oct 31 00:55:40.789606 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:55:40.789586 5222 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d501b5bd3b7acfe5169d0990e01ea56fb56fb6a8dca96b6d2f6a87e170b14c8\": container with ID starting with 5d501b5bd3b7acfe5169d0990e01ea56fb56fb6a8dca96b6d2f6a87e170b14c8 not found: ID does not exist" containerID="5d501b5bd3b7acfe5169d0990e01ea56fb56fb6a8dca96b6d2f6a87e170b14c8" Oct 31 00:55:40.789663 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:40.789618 5222 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d501b5bd3b7acfe5169d0990e01ea56fb56fb6a8dca96b6d2f6a87e170b14c8"} err="failed to get container status \"5d501b5bd3b7acfe5169d0990e01ea56fb56fb6a8dca96b6d2f6a87e170b14c8\": rpc error: code = NotFound desc = could not find container \"5d501b5bd3b7acfe5169d0990e01ea56fb56fb6a8dca96b6d2f6a87e170b14c8\": container with ID starting with 5d501b5bd3b7acfe5169d0990e01ea56fb56fb6a8dca96b6d2f6a87e170b14c8 not found: ID does not exist" Oct 31 00:55:40.951436 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:40.951382 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-101b9e-4s8tx" podStartSLOduration=1.95135115 podStartE2EDuration="1.95135115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:55:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:55:40.950166335 +0000 UTC m=+2433.421336083" watchObservedRunningTime="2026-10-31 00:55:40.95135115 +0000 UTC m=+2433.422520937" Oct 31 00:55:41.066198 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:41.066094 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-lzwsk"] Oct 31 00:55:41.117441 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:41.117399 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-lzwsk"] Oct 31 00:55:41.656499 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:41.656463 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7747c507-e71c-45eb-8ffb-38b61253999c" path="/var/lib/kubelet/pods/7747c507-e71c-45eb-8ffb-38b61253999c/volumes" Oct 31 00:55:41.793669 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:41.793636 5222 generic.go:356] "Generic (PLEG): container finished" podID="6eefacd4-395f-42f5-922d-fb2d971401c5" containerID="8a361ecef0d6ddb72b61057cd2706fe90dc8fa4db98208e2c04ef70c72122c04" exitCode=0 Oct 31 00:55:41.793877 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:41.793732 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-101b9e-4s8tx" event={"ID":"6eefacd4-395f-42f5-922d-fb2d971401c5","Type":"ContainerDied","Data":"8a361ecef0d6ddb72b61057cd2706fe90dc8fa4db98208e2c04ef70c72122c04"} Oct 31 00:55:42.493439 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:42.493360 5222 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-412743/distinct-2-w5-412743-kserve-5b679b6b66-59prz" podUID="2f6cbca5-d586-4bc2-9114-7212e9cc8a01" containerName="main" containerID="cri-o://d294543ced999ad1705a2300a6ba610d93c3b8724d4247fe4587a2316de63800" gracePeriod=45 Oct 31 00:55:42.495017 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:42.494981 5222 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-412743/distinct-w5-412743-kserve-64d65f7d8f-c4tfs" podUID="d0e8d5c5-aa08-41f6-badf-792648a20a68" containerName="main" containerID="cri-o://af8a8bb0d4914d263c0a1b9186a10009c551064f85aca4355365e584cba73091" gracePeriod=45 Oct 31 00:55:42.548888 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:42.548841 5222 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-412743/simulator-w5-412743-kserve-6f897dd967-5bkq5" podUID="68d7fd60-bcad-45f7-84c0-11424e26f26d" containerName="main" containerID="cri-o://11580e2d56855201eba5f576e3f0a083d778374e459e175137997cc834eb701a" gracePeriod=45 Oct 31 00:55:42.557608 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:42.557565 5222 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-412743/distinct-2-w5-412743-kserve-5b679b6b66-59prz" podUID="2f6cbca5-d586-4bc2-9114-7212e9cc8a01" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": dial tcp 10.134.0.90:8000: connect: connection refused" Oct 31 00:55:42.616157 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:42.616123 5222 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-412743/distinct-w5-412743-kserve-64d65f7d8f-c4tfs" podUID="d0e8d5c5-aa08-41f6-badf-792648a20a68" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": dial tcp 10.134.0.83:8000: connect: connection refused" Oct 31 00:55:42.683622 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:42.683587 5222 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-412743/simulator-w5-412743-kserve-6f897dd967-5bkq5" podUID="68d7fd60-bcad-45f7-84c0-11424e26f26d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": dial tcp 10.134.0.49:8000: connect: connection refused" Oct 31 00:55:42.808630 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:42.808585 5222 generic.go:356] "Generic (PLEG): container finished" podID="2f6cbca5-d586-4bc2-9114-7212e9cc8a01" containerID="d294543ced999ad1705a2300a6ba610d93c3b8724d4247fe4587a2316de63800" exitCode=0 Oct 31 00:55:42.808875 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:42.808728 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-412743/distinct-2-w5-412743-kserve-5b679b6b66-59prz" event={"ID":"2f6cbca5-d586-4bc2-9114-7212e9cc8a01","Type":"ContainerDied","Data":"d294543ced999ad1705a2300a6ba610d93c3b8724d4247fe4587a2316de63800"} Oct 31 00:55:42.810972 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:42.810944 5222 generic.go:356] "Generic (PLEG): container finished" podID="68d7fd60-bcad-45f7-84c0-11424e26f26d" containerID="11580e2d56855201eba5f576e3f0a083d778374e459e175137997cc834eb701a" exitCode=0 Oct 31 00:55:42.811123 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:42.811060 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-412743/simulator-w5-412743-kserve-6f897dd967-5bkq5" event={"ID":"68d7fd60-bcad-45f7-84c0-11424e26f26d","Type":"ContainerDied","Data":"11580e2d56855201eba5f576e3f0a083d778374e459e175137997cc834eb701a"} Oct 31 00:55:42.813806 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:42.813772 5222 generic.go:356] "Generic (PLEG): container finished" podID="d0e8d5c5-aa08-41f6-badf-792648a20a68" containerID="af8a8bb0d4914d263c0a1b9186a10009c551064f85aca4355365e584cba73091" exitCode=0 Oct 31 00:55:42.814078 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:42.814054 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-412743/distinct-w5-412743-kserve-64d65f7d8f-c4tfs" event={"ID":"d0e8d5c5-aa08-41f6-badf-792648a20a68","Type":"ContainerDied","Data":"af8a8bb0d4914d263c0a1b9186a10009c551064f85aca4355365e584cba73091"} Oct 31 00:55:42.871310 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:42.871180 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-412743/distinct-w5-412743-kserve-64d65f7d8f-c4tfs" Oct 31 00:55:42.875247 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:42.875221 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-412743/distinct-2-w5-412743-kserve-5b679b6b66-59prz" Oct 31 00:55:42.880562 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:42.880537 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-412743/simulator-w5-412743-kserve-6f897dd967-5bkq5" Oct 31 00:55:42.946205 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:42.946171 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68d7fd60-bcad-45f7-84c0-11424e26f26d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68d7fd60-bcad-45f7-84c0-11424e26f26d-model-cache\") pod \"68d7fd60-bcad-45f7-84c0-11424e26f26d\" (UID: \"68d7fd60-bcad-45f7-84c0-11424e26f26d\") " Oct 31 00:55:42.946430 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:42.946226 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f6cbca5-d586-4bc2-9114-7212e9cc8a01-kube-api-access-54djx\" (UniqueName: \"kubernetes.io/projected/2f6cbca5-d586-4bc2-9114-7212e9cc8a01-kube-api-access-54djx\") pod \"2f6cbca5-d586-4bc2-9114-7212e9cc8a01\" (UID: \"2f6cbca5-d586-4bc2-9114-7212e9cc8a01\") " Oct 31 00:55:42.946430 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:42.946283 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68d7fd60-bcad-45f7-84c0-11424e26f26d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/68d7fd60-bcad-45f7-84c0-11424e26f26d-dshm\") pod \"68d7fd60-bcad-45f7-84c0-11424e26f26d\" (UID: \"68d7fd60-bcad-45f7-84c0-11424e26f26d\") " Oct 31 00:55:42.946430 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:42.946317 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f6cbca5-d586-4bc2-9114-7212e9cc8a01-tls-certs\" (UniqueName: \"kubernetes.io/secret/2f6cbca5-d586-4bc2-9114-7212e9cc8a01-tls-certs\") pod \"2f6cbca5-d586-4bc2-9114-7212e9cc8a01\" (UID: \"2f6cbca5-d586-4bc2-9114-7212e9cc8a01\") " Oct 31 00:55:42.946430 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:42.946344 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0e8d5c5-aa08-41f6-badf-792648a20a68-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0e8d5c5-aa08-41f6-badf-792648a20a68-dshm\") pod \"d0e8d5c5-aa08-41f6-badf-792648a20a68\" (UID: \"d0e8d5c5-aa08-41f6-badf-792648a20a68\") " Oct 31 00:55:42.946430 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:42.946409 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0e8d5c5-aa08-41f6-badf-792648a20a68-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0e8d5c5-aa08-41f6-badf-792648a20a68-tmp-dir\") pod \"d0e8d5c5-aa08-41f6-badf-792648a20a68\" (UID: \"d0e8d5c5-aa08-41f6-badf-792648a20a68\") " Oct 31 00:55:42.946729 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:42.946450 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68d7fd60-bcad-45f7-84c0-11424e26f26d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68d7fd60-bcad-45f7-84c0-11424e26f26d-tmp-dir\") pod \"68d7fd60-bcad-45f7-84c0-11424e26f26d\" (UID: \"68d7fd60-bcad-45f7-84c0-11424e26f26d\") " Oct 31 00:55:42.946729 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:42.946539 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0e8d5c5-aa08-41f6-badf-792648a20a68-tls-certs\" (UniqueName: \"kubernetes.io/secret/d0e8d5c5-aa08-41f6-badf-792648a20a68-tls-certs\") pod \"d0e8d5c5-aa08-41f6-badf-792648a20a68\" (UID: \"d0e8d5c5-aa08-41f6-badf-792648a20a68\") " Oct 31 00:55:42.946729 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:42.946584 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f6cbca5-d586-4bc2-9114-7212e9cc8a01-home\" (UniqueName: \"kubernetes.io/empty-dir/2f6cbca5-d586-4bc2-9114-7212e9cc8a01-home\") pod \"2f6cbca5-d586-4bc2-9114-7212e9cc8a01\" (UID: \"2f6cbca5-d586-4bc2-9114-7212e9cc8a01\") " Oct 31 00:55:42.946729 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:42.946597 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68d7fd60-bcad-45f7-84c0-11424e26f26d-model-cache" pod "68d7fd60-bcad-45f7-84c0-11424e26f26d" (UID: "68d7fd60-bcad-45f7-84c0-11424e26f26d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:55:42.946729 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:42.946651 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0e8d5c5-aa08-41f6-badf-792648a20a68-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0e8d5c5-aa08-41f6-badf-792648a20a68-model-cache\") pod \"d0e8d5c5-aa08-41f6-badf-792648a20a68\" (UID: \"d0e8d5c5-aa08-41f6-badf-792648a20a68\") " Oct 31 00:55:42.946729 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:42.946697 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f6cbca5-d586-4bc2-9114-7212e9cc8a01-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f6cbca5-d586-4bc2-9114-7212e9cc8a01-tmp-dir\") pod \"2f6cbca5-d586-4bc2-9114-7212e9cc8a01\" (UID: \"2f6cbca5-d586-4bc2-9114-7212e9cc8a01\") " Oct 31 00:55:42.947044 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:42.946733 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68d7fd60-bcad-45f7-84c0-11424e26f26d-kube-api-access-vqthd\" (UniqueName: \"kubernetes.io/projected/68d7fd60-bcad-45f7-84c0-11424e26f26d-kube-api-access-vqthd\") pod \"68d7fd60-bcad-45f7-84c0-11424e26f26d\" (UID: \"68d7fd60-bcad-45f7-84c0-11424e26f26d\") " Oct 31 00:55:42.947044 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:42.946770 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0e8d5c5-aa08-41f6-badf-792648a20a68-kube-api-access-lxzgk\" (UniqueName: \"kubernetes.io/projected/d0e8d5c5-aa08-41f6-badf-792648a20a68-kube-api-access-lxzgk\") pod \"d0e8d5c5-aa08-41f6-badf-792648a20a68\" (UID: \"d0e8d5c5-aa08-41f6-badf-792648a20a68\") " Oct 31 00:55:42.947044 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:42.946810 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0e8d5c5-aa08-41f6-badf-792648a20a68-tmp-dir" pod "d0e8d5c5-aa08-41f6-badf-792648a20a68" (UID: "d0e8d5c5-aa08-41f6-badf-792648a20a68"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:55:42.947044 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:42.946831 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68d7fd60-bcad-45f7-84c0-11424e26f26d-tls-certs\" (UniqueName: \"kubernetes.io/secret/68d7fd60-bcad-45f7-84c0-11424e26f26d-tls-certs\") pod \"68d7fd60-bcad-45f7-84c0-11424e26f26d\" (UID: \"68d7fd60-bcad-45f7-84c0-11424e26f26d\") " Oct 31 00:55:42.947044 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:42.946895 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68d7fd60-bcad-45f7-84c0-11424e26f26d-home\" (UniqueName: \"kubernetes.io/empty-dir/68d7fd60-bcad-45f7-84c0-11424e26f26d-home\") pod \"68d7fd60-bcad-45f7-84c0-11424e26f26d\" (UID: \"68d7fd60-bcad-45f7-84c0-11424e26f26d\") " Oct 31 00:55:42.947044 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:42.946936 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0e8d5c5-aa08-41f6-badf-792648a20a68-home\" (UniqueName: \"kubernetes.io/empty-dir/d0e8d5c5-aa08-41f6-badf-792648a20a68-home\") pod \"d0e8d5c5-aa08-41f6-badf-792648a20a68\" (UID: \"d0e8d5c5-aa08-41f6-badf-792648a20a68\") " Oct 31 00:55:42.947044 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:42.946975 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f6cbca5-d586-4bc2-9114-7212e9cc8a01-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f6cbca5-d586-4bc2-9114-7212e9cc8a01-dshm\") pod \"2f6cbca5-d586-4bc2-9114-7212e9cc8a01\" (UID: \"2f6cbca5-d586-4bc2-9114-7212e9cc8a01\") " Oct 31 00:55:42.947044 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:42.947032 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f6cbca5-d586-4bc2-9114-7212e9cc8a01-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f6cbca5-d586-4bc2-9114-7212e9cc8a01-model-cache\") pod \"2f6cbca5-d586-4bc2-9114-7212e9cc8a01\" (UID: \"2f6cbca5-d586-4bc2-9114-7212e9cc8a01\") " Oct 31 00:55:42.947500 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:42.947222 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0e8d5c5-aa08-41f6-badf-792648a20a68-model-cache" pod "d0e8d5c5-aa08-41f6-badf-792648a20a68" (UID: "d0e8d5c5-aa08-41f6-badf-792648a20a68"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:55:42.947563 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:42.947507 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f6cbca5-d586-4bc2-9114-7212e9cc8a01-model-cache" pod "2f6cbca5-d586-4bc2-9114-7212e9cc8a01" (UID: "2f6cbca5-d586-4bc2-9114-7212e9cc8a01"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:55:42.947803 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:42.947676 5222 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f6cbca5-d586-4bc2-9114-7212e9cc8a01-model-cache\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:42.947803 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:42.947700 5222 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68d7fd60-bcad-45f7-84c0-11424e26f26d-model-cache\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:42.947803 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:42.947705 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68d7fd60-bcad-45f7-84c0-11424e26f26d-tmp-dir" pod "68d7fd60-bcad-45f7-84c0-11424e26f26d" (UID: "68d7fd60-bcad-45f7-84c0-11424e26f26d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:55:42.947803 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:42.947715 5222 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0e8d5c5-aa08-41f6-badf-792648a20a68-tmp-dir\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:42.947803 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:42.947752 5222 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0e8d5c5-aa08-41f6-badf-792648a20a68-model-cache\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:42.948139 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:42.947865 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f6cbca5-d586-4bc2-9114-7212e9cc8a01-tmp-dir" pod "2f6cbca5-d586-4bc2-9114-7212e9cc8a01" (UID: "2f6cbca5-d586-4bc2-9114-7212e9cc8a01"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:55:42.948139 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:42.948049 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68d7fd60-bcad-45f7-84c0-11424e26f26d-home" pod "68d7fd60-bcad-45f7-84c0-11424e26f26d" (UID: "68d7fd60-bcad-45f7-84c0-11424e26f26d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:55:42.948257 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:42.948214 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0e8d5c5-aa08-41f6-badf-792648a20a68-home" pod "d0e8d5c5-aa08-41f6-badf-792648a20a68" (UID: "d0e8d5c5-aa08-41f6-badf-792648a20a68"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:55:42.948677 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:42.948361 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f6cbca5-d586-4bc2-9114-7212e9cc8a01-home" pod "2f6cbca5-d586-4bc2-9114-7212e9cc8a01" (UID: "2f6cbca5-d586-4bc2-9114-7212e9cc8a01"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:55:42.950061 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:42.950036 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68d7fd60-bcad-45f7-84c0-11424e26f26d-dshm" pod "68d7fd60-bcad-45f7-84c0-11424e26f26d" (UID: "68d7fd60-bcad-45f7-84c0-11424e26f26d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:55:42.950170 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:42.950142 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0e8d5c5-aa08-41f6-badf-792648a20a68-dshm" pod "d0e8d5c5-aa08-41f6-badf-792648a20a68" (UID: "d0e8d5c5-aa08-41f6-badf-792648a20a68"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:55:42.950319 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:42.950291 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f6cbca5-d586-4bc2-9114-7212e9cc8a01-kube-api-access-54djx" pod "2f6cbca5-d586-4bc2-9114-7212e9cc8a01" (UID: "2f6cbca5-d586-4bc2-9114-7212e9cc8a01"). InnerVolumeSpecName "kube-api-access-54djx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:55:42.950768 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:42.950748 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f6cbca5-d586-4bc2-9114-7212e9cc8a01-tls-certs" pod "2f6cbca5-d586-4bc2-9114-7212e9cc8a01" (UID: "2f6cbca5-d586-4bc2-9114-7212e9cc8a01"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:55:42.950941 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:42.950920 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f6cbca5-d586-4bc2-9114-7212e9cc8a01-dshm" pod "2f6cbca5-d586-4bc2-9114-7212e9cc8a01" (UID: "2f6cbca5-d586-4bc2-9114-7212e9cc8a01"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:55:42.951487 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:42.951468 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68d7fd60-bcad-45f7-84c0-11424e26f26d-kube-api-access-vqthd" pod "68d7fd60-bcad-45f7-84c0-11424e26f26d" (UID: "68d7fd60-bcad-45f7-84c0-11424e26f26d"). InnerVolumeSpecName "kube-api-access-vqthd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:55:42.951722 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:42.951703 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68d7fd60-bcad-45f7-84c0-11424e26f26d-tls-certs" pod "68d7fd60-bcad-45f7-84c0-11424e26f26d" (UID: "68d7fd60-bcad-45f7-84c0-11424e26f26d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:55:42.951888 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:42.951861 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0e8d5c5-aa08-41f6-badf-792648a20a68-tls-certs" pod "d0e8d5c5-aa08-41f6-badf-792648a20a68" (UID: "d0e8d5c5-aa08-41f6-badf-792648a20a68"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:55:42.953239 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:42.953216 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0e8d5c5-aa08-41f6-badf-792648a20a68-kube-api-access-lxzgk" pod "d0e8d5c5-aa08-41f6-badf-792648a20a68" (UID: "d0e8d5c5-aa08-41f6-badf-792648a20a68"). InnerVolumeSpecName "kube-api-access-lxzgk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:55:42.966235 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:42.966210 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-101b9e-4s8tx" Oct 31 00:55:43.048914 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:43.048878 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6eefacd4-395f-42f5-922d-fb2d971401c5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6eefacd4-395f-42f5-922d-fb2d971401c5-maas-api-xks-ca\") pod \"6eefacd4-395f-42f5-922d-fb2d971401c5\" (UID: \"6eefacd4-395f-42f5-922d-fb2d971401c5\") " Oct 31 00:55:43.049091 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:43.048974 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6eefacd4-395f-42f5-922d-fb2d971401c5-kube-api-access-4sfpp\" (UniqueName: \"kubernetes.io/projected/6eefacd4-395f-42f5-922d-fb2d971401c5-kube-api-access-4sfpp\") pod \"6eefacd4-395f-42f5-922d-fb2d971401c5\" (UID: \"6eefacd4-395f-42f5-922d-fb2d971401c5\") " Oct 31 00:55:43.049139 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:43.049102 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6eefacd4-395f-42f5-922d-fb2d971401c5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6eefacd4-395f-42f5-922d-fb2d971401c5-maas-api-openshift-ca\") pod \"6eefacd4-395f-42f5-922d-fb2d971401c5\" (UID: \"6eefacd4-395f-42f5-922d-fb2d971401c5\") " Oct 31 00:55:43.049559 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:43.049497 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6eefacd4-395f-42f5-922d-fb2d971401c5-maas-api-openshift-ca" pod "6eefacd4-395f-42f5-922d-fb2d971401c5" (UID: "6eefacd4-395f-42f5-922d-fb2d971401c5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:55:43.049559 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:43.049531 5222 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f6cbca5-d586-4bc2-9114-7212e9cc8a01-tmp-dir\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:43.049559 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:43.049550 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqthd\" (UniqueName: \"kubernetes.io/projected/68d7fd60-bcad-45f7-84c0-11424e26f26d-kube-api-access-vqthd\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:43.049559 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:43.049562 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxzgk\" (UniqueName: \"kubernetes.io/projected/d0e8d5c5-aa08-41f6-badf-792648a20a68-kube-api-access-lxzgk\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:43.049559 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:43.049571 5222 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68d7fd60-bcad-45f7-84c0-11424e26f26d-tls-certs\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:43.049559 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:43.049585 5222 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68d7fd60-bcad-45f7-84c0-11424e26f26d-home\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:43.049559 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:43.049596 5222 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0e8d5c5-aa08-41f6-badf-792648a20a68-home\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:43.049559 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:43.049608 5222 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f6cbca5-d586-4bc2-9114-7212e9cc8a01-dshm\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:43.049559 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:43.049621 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54djx\" (UniqueName: \"kubernetes.io/projected/2f6cbca5-d586-4bc2-9114-7212e9cc8a01-kube-api-access-54djx\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:43.049559 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:43.049633 5222 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68d7fd60-bcad-45f7-84c0-11424e26f26d-dshm\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:43.049559 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:43.049643 5222 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f6cbca5-d586-4bc2-9114-7212e9cc8a01-tls-certs\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:43.049559 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:43.049653 5222 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0e8d5c5-aa08-41f6-badf-792648a20a68-dshm\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:43.049559 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:43.049660 5222 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68d7fd60-bcad-45f7-84c0-11424e26f26d-tmp-dir\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:43.049559 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:43.049670 5222 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0e8d5c5-aa08-41f6-badf-792648a20a68-tls-certs\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:43.049559 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:43.049677 5222 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f6cbca5-d586-4bc2-9114-7212e9cc8a01-home\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:43.051179 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:43.051159 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6eefacd4-395f-42f5-922d-fb2d971401c5-kube-api-access-4sfpp" pod "6eefacd4-395f-42f5-922d-fb2d971401c5" (UID: "6eefacd4-395f-42f5-922d-fb2d971401c5"). InnerVolumeSpecName "kube-api-access-4sfpp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:55:43.051331 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:43.051310 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6eefacd4-395f-42f5-922d-fb2d971401c5-maas-api-xks-ca" pod "6eefacd4-395f-42f5-922d-fb2d971401c5" (UID: "6eefacd4-395f-42f5-922d-fb2d971401c5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:55:43.151257 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:43.151224 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6eefacd4-395f-42f5-922d-fb2d971401c5-maas-api-openshift-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:43.151257 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:43.151254 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6eefacd4-395f-42f5-922d-fb2d971401c5-maas-api-xks-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:43.151563 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:43.151268 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4sfpp\" (UniqueName: \"kubernetes.io/projected/6eefacd4-395f-42f5-922d-fb2d971401c5-kube-api-access-4sfpp\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:43.676863 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:43.676832 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-101b9e/distinct-2-w0-101b9e-kserve-74548f6d6-v5phc"] Oct 31 00:55:43.680506 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:43.680477 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-101b9e/distinct-w0-101b9e-kserve-6dcc4cc94f-kwv9w"] Oct 31 00:55:43.685813 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:43.685787 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-101b9e/premium-w0-101b9e-kserve-dc65f694b-m4tsk"] Oct 31 00:55:43.822069 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:43.822027 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-101b9e-4s8tx" Oct 31 00:55:43.822069 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:43.822057 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-101b9e-4s8tx" event={"ID":"6eefacd4-395f-42f5-922d-fb2d971401c5","Type":"ContainerDied","Data":"1a2df770e4f9b161134d1ea9a6c66c02a29f70c8268f227abaa8833c581b52c4"} Oct 31 00:55:43.822315 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:43.822117 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1a2df770e4f9b161134d1ea9a6c66c02a29f70c8268f227abaa8833c581b52c4" Oct 31 00:55:43.823616 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:43.823587 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-412743/simulator-w5-412743-kserve-6f897dd967-5bkq5" event={"ID":"68d7fd60-bcad-45f7-84c0-11424e26f26d","Type":"ContainerDied","Data":"3360d1c4d849af0828635dd57fb1582fb43ad23ad7765db445db53a48a50ee29"} Oct 31 00:55:43.823616 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:43.823611 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-412743/simulator-w5-412743-kserve-6f897dd967-5bkq5" Oct 31 00:55:43.823832 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:43.823639 5222 scope.go:122] "RemoveContainer" containerID="11580e2d56855201eba5f576e3f0a083d778374e459e175137997cc834eb701a" Oct 31 00:55:43.826138 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:43.826114 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-412743/distinct-w5-412743-kserve-64d65f7d8f-c4tfs" event={"ID":"d0e8d5c5-aa08-41f6-badf-792648a20a68","Type":"ContainerDied","Data":"d7f5e2a1b1245344f532b700b30fb938a3bbddce02e33e32bea18c49f5e18b1f"} Oct 31 00:55:43.826352 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:43.826272 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-412743/distinct-w5-412743-kserve-64d65f7d8f-c4tfs" Oct 31 00:55:43.830915 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:43.830884 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-412743/distinct-2-w5-412743-kserve-5b679b6b66-59prz" event={"ID":"2f6cbca5-d586-4bc2-9114-7212e9cc8a01","Type":"ContainerDied","Data":"a9ba501a899a961eaef0739b040f8cca89566f82296780737d7901e55af6fe0a"} Oct 31 00:55:43.832332 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:43.830999 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-412743/distinct-2-w5-412743-kserve-5b679b6b66-59prz" Oct 31 00:55:43.840134 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:43.840111 5222 scope.go:122] "RemoveContainer" containerID="af8a8bb0d4914d263c0a1b9186a10009c551064f85aca4355365e584cba73091" Oct 31 00:55:43.849416 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:43.849391 5222 scope.go:122] "RemoveContainer" containerID="d294543ced999ad1705a2300a6ba610d93c3b8724d4247fe4587a2316de63800" Oct 31 00:55:44.151928 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:44.151823 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-412743/distinct-w5-412743-kserve-64d65f7d8f-c4tfs"] Oct 31 00:55:44.155282 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:44.155257 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-412743/distinct-w5-412743-kserve-64d65f7d8f-c4tfs"] Oct 31 00:55:44.269408 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:44.269356 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-412743/distinct-2-w5-412743-kserve-5b679b6b66-59prz"] Oct 31 00:55:44.273346 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:44.273318 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-412743/distinct-2-w5-412743-kserve-5b679b6b66-59prz"] Oct 31 00:55:44.439356 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:44.439209 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-412743/simulator-w5-412743-kserve-6f897dd967-5bkq5"] Oct 31 00:55:44.443193 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:44.443163 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-412743/simulator-w5-412743-kserve-6f897dd967-5bkq5"] Oct 31 00:55:45.655511 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:45.655482 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f6cbca5-d586-4bc2-9114-7212e9cc8a01" path="/var/lib/kubelet/pods/2f6cbca5-d586-4bc2-9114-7212e9cc8a01/volumes" Oct 31 00:55:45.656079 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:45.656063 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68d7fd60-bcad-45f7-84c0-11424e26f26d" path="/var/lib/kubelet/pods/68d7fd60-bcad-45f7-84c0-11424e26f26d/volumes" Oct 31 00:55:45.656477 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:45.656464 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0e8d5c5-aa08-41f6-badf-792648a20a68" path="/var/lib/kubelet/pods/d0e8d5c5-aa08-41f6-badf-792648a20a68/volumes" Oct 31 00:55:48.935235 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:48.935195 5222 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-412743-6d5cb5bd7f-pzb7l" podUID="9dbd9108-ac33-45a0-9eed-ef9b81755398" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:55:48.935735 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:48.935205 5222 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-412743-688d66585f-vk5r7" podUID="a99cbf31-f3f2-4d16-a665-58f0f5af77f0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:55:50.611805 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:50.611769 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a6f4a5-59r9m"] Oct 31 00:55:50.662698 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:50.662668 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a6f4a5-59r9m"] Oct 31 00:55:50.758897 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:50.758863 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-101b9e-688fd6f5d7-7nrcm"] Oct 31 00:55:51.656659 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:51.656616 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc45d98c-3404-458c-a1ab-9b1a2c376579" path="/var/lib/kubelet/pods/fc45d98c-3404-458c-a1ab-9b1a2c376579/volumes" Oct 31 00:55:55.759980 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:55.759922 5222 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-101b9e-688fd6f5d7-7nrcm" podUID="38bd486f-e3b6-4d6e-8ca2-6bb7591b9ba3" containerName="payload-pre-processing" containerID="cri-o://52aae2a664052461da0e54e45e312a372bc73cf5d96aeeb28fb5a40434b06f65" gracePeriod=25 Oct 31 00:55:55.997830 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:55.997796 5222 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-101b9e-688fd6f5d7-7nrcm" podUID="38bd486f-e3b6-4d6e-8ca2-6bb7591b9ba3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:55:56.749092 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:56.749052 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-79tk8"] Oct 31 00:55:58.707398 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:58.703602 5222 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-101b9e/distinct-w0-101b9e-kserve-6dcc4cc94f-kwv9w" podUID="73487e30-1bc6-49b2-b07b-1b004137232e" containerName="main" containerID="cri-o://edb07311cd170b00ceaadc005648108ea352ce910f7179de453809f9ebc1ad5d" gracePeriod=45 Oct 31 00:55:58.717829 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:58.709873 5222 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-101b9e/distinct-2-w0-101b9e-kserve-74548f6d6-v5phc" podUID="b8049531-bcf6-4f19-84d2-b974b9236223" containerName="main" containerID="cri-o://3de9a26b71ba3e866fdf0760764d8ec12776a0d1d10a27cd71ede4821cad7543" gracePeriod=45 Oct 31 00:55:58.718410 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:58.718351 5222 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-101b9e/premium-w0-101b9e-kserve-dc65f694b-m4tsk" podUID="fbf98c81-6f05-41b8-ba3b-c10ecbbff5d8" containerName="main" containerID="cri-o://3e93fc0c53f37807e7cb279c8763e1f1bf5e2414e5424875b92b886d6127dcbe" gracePeriod=45 Oct 31 00:55:58.725385 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:58.725337 5222 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-101b9e/premium-w0-101b9e-kserve-dc65f694b-m4tsk" podUID="fbf98c81-6f05-41b8-ba3b-c10ecbbff5d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": read tcp 10.134.0.2:53638->10.134.0.58:8000: read: connection reset by peer" Oct 31 00:55:58.937748 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:58.937303 5222 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-412743-688d66585f-vk5r7" podUID="a99cbf31-f3f2-4d16-a665-58f0f5af77f0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:55:58.944405 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:58.944274 5222 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-412743-6d5cb5bd7f-pzb7l" podUID="9dbd9108-ac33-45a0-9eed-ef9b81755398" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:55:58.960604 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:58.960202 5222 generic.go:356] "Generic (PLEG): container finished" podID="b8049531-bcf6-4f19-84d2-b974b9236223" containerID="3de9a26b71ba3e866fdf0760764d8ec12776a0d1d10a27cd71ede4821cad7543" exitCode=0 Oct 31 00:55:58.960604 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:58.960482 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-101b9e/distinct-2-w0-101b9e-kserve-74548f6d6-v5phc" event={"ID":"b8049531-bcf6-4f19-84d2-b974b9236223","Type":"ContainerDied","Data":"3de9a26b71ba3e866fdf0760764d8ec12776a0d1d10a27cd71ede4821cad7543"} Oct 31 00:55:59.015414 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:59.007476 5222 generic.go:356] "Generic (PLEG): container finished" podID="73487e30-1bc6-49b2-b07b-1b004137232e" containerID="edb07311cd170b00ceaadc005648108ea352ce910f7179de453809f9ebc1ad5d" exitCode=0 Oct 31 00:55:59.015414 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:59.007571 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-101b9e/distinct-w0-101b9e-kserve-6dcc4cc94f-kwv9w" event={"ID":"73487e30-1bc6-49b2-b07b-1b004137232e","Type":"ContainerDied","Data":"edb07311cd170b00ceaadc005648108ea352ce910f7179de453809f9ebc1ad5d"} Oct 31 00:55:59.016051 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:59.016018 5222 generic.go:356] "Generic (PLEG): container finished" podID="fbf98c81-6f05-41b8-ba3b-c10ecbbff5d8" containerID="3e93fc0c53f37807e7cb279c8763e1f1bf5e2414e5424875b92b886d6127dcbe" exitCode=0 Oct 31 00:55:59.016180 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:59.016163 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-101b9e/premium-w0-101b9e-kserve-dc65f694b-m4tsk" event={"ID":"fbf98c81-6f05-41b8-ba3b-c10ecbbff5d8","Type":"ContainerDied","Data":"3e93fc0c53f37807e7cb279c8763e1f1bf5e2414e5424875b92b886d6127dcbe"} Oct 31 00:55:59.222120 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:59.222011 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-101b9e/distinct-2-w0-101b9e-kserve-74548f6d6-v5phc" Oct 31 00:55:59.224761 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:59.224739 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-101b9e/distinct-w0-101b9e-kserve-6dcc4cc94f-kwv9w" Oct 31 00:55:59.235230 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:59.235205 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-101b9e/premium-w0-101b9e-kserve-dc65f694b-m4tsk" Oct 31 00:55:59.412746 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:59.412715 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8049531-bcf6-4f19-84d2-b974b9236223-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8049531-bcf6-4f19-84d2-b974b9236223-model-cache\") pod \"b8049531-bcf6-4f19-84d2-b974b9236223\" (UID: \"b8049531-bcf6-4f19-84d2-b974b9236223\") " Oct 31 00:55:59.412949 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:59.412758 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbf98c81-6f05-41b8-ba3b-c10ecbbff5d8-home\" (UniqueName: \"kubernetes.io/empty-dir/fbf98c81-6f05-41b8-ba3b-c10ecbbff5d8-home\") pod \"fbf98c81-6f05-41b8-ba3b-c10ecbbff5d8\" (UID: \"fbf98c81-6f05-41b8-ba3b-c10ecbbff5d8\") " Oct 31 00:55:59.412949 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:59.412783 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbf98c81-6f05-41b8-ba3b-c10ecbbff5d8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbf98c81-6f05-41b8-ba3b-c10ecbbff5d8-model-cache\") pod \"fbf98c81-6f05-41b8-ba3b-c10ecbbff5d8\" (UID: \"fbf98c81-6f05-41b8-ba3b-c10ecbbff5d8\") " Oct 31 00:55:59.412949 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:59.412836 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73487e30-1bc6-49b2-b07b-1b004137232e-tls-certs\" (UniqueName: \"kubernetes.io/secret/73487e30-1bc6-49b2-b07b-1b004137232e-tls-certs\") pod \"73487e30-1bc6-49b2-b07b-1b004137232e\" (UID: \"73487e30-1bc6-49b2-b07b-1b004137232e\") " Oct 31 00:55:59.412949 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:59.412868 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbf98c81-6f05-41b8-ba3b-c10ecbbff5d8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbf98c81-6f05-41b8-ba3b-c10ecbbff5d8-dshm\") pod \"fbf98c81-6f05-41b8-ba3b-c10ecbbff5d8\" (UID: \"fbf98c81-6f05-41b8-ba3b-c10ecbbff5d8\") " Oct 31 00:55:59.412949 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:59.412902 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73487e30-1bc6-49b2-b07b-1b004137232e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73487e30-1bc6-49b2-b07b-1b004137232e-model-cache\") pod \"73487e30-1bc6-49b2-b07b-1b004137232e\" (UID: \"73487e30-1bc6-49b2-b07b-1b004137232e\") " Oct 31 00:55:59.412949 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:59.412933 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73487e30-1bc6-49b2-b07b-1b004137232e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73487e30-1bc6-49b2-b07b-1b004137232e-tmp-dir\") pod \"73487e30-1bc6-49b2-b07b-1b004137232e\" (UID: \"73487e30-1bc6-49b2-b07b-1b004137232e\") " Oct 31 00:55:59.413286 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:59.413005 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbf98c81-6f05-41b8-ba3b-c10ecbbff5d8-kube-api-access-459r7\" (UniqueName: \"kubernetes.io/projected/fbf98c81-6f05-41b8-ba3b-c10ecbbff5d8-kube-api-access-459r7\") pod \"fbf98c81-6f05-41b8-ba3b-c10ecbbff5d8\" (UID: \"fbf98c81-6f05-41b8-ba3b-c10ecbbff5d8\") " Oct 31 00:55:59.413286 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:59.413036 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbf98c81-6f05-41b8-ba3b-c10ecbbff5d8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbf98c81-6f05-41b8-ba3b-c10ecbbff5d8-tmp-dir\") pod \"fbf98c81-6f05-41b8-ba3b-c10ecbbff5d8\" (UID: \"fbf98c81-6f05-41b8-ba3b-c10ecbbff5d8\") " Oct 31 00:55:59.413286 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:59.413043 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbf98c81-6f05-41b8-ba3b-c10ecbbff5d8-home" pod "fbf98c81-6f05-41b8-ba3b-c10ecbbff5d8" (UID: "fbf98c81-6f05-41b8-ba3b-c10ecbbff5d8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:55:59.413286 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:59.413051 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8049531-bcf6-4f19-84d2-b974b9236223-model-cache" pod "b8049531-bcf6-4f19-84d2-b974b9236223" (UID: "b8049531-bcf6-4f19-84d2-b974b9236223"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:55:59.413286 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:59.413096 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8049531-bcf6-4f19-84d2-b974b9236223-home\" (UniqueName: \"kubernetes.io/empty-dir/b8049531-bcf6-4f19-84d2-b974b9236223-home\") pod \"b8049531-bcf6-4f19-84d2-b974b9236223\" (UID: \"b8049531-bcf6-4f19-84d2-b974b9236223\") " Oct 31 00:55:59.413286 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:59.413164 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8049531-bcf6-4f19-84d2-b974b9236223-tls-certs\" (UniqueName: \"kubernetes.io/secret/b8049531-bcf6-4f19-84d2-b974b9236223-tls-certs\") pod \"b8049531-bcf6-4f19-84d2-b974b9236223\" (UID: \"b8049531-bcf6-4f19-84d2-b974b9236223\") " Oct 31 00:55:59.413286 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:59.413212 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73487e30-1bc6-49b2-b07b-1b004137232e-kube-api-access-s4598\" (UniqueName: \"kubernetes.io/projected/73487e30-1bc6-49b2-b07b-1b004137232e-kube-api-access-s4598\") pod \"73487e30-1bc6-49b2-b07b-1b004137232e\" (UID: \"73487e30-1bc6-49b2-b07b-1b004137232e\") " Oct 31 00:55:59.413286 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:59.413251 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8049531-bcf6-4f19-84d2-b974b9236223-kube-api-access-kqdb8\" (UniqueName: \"kubernetes.io/projected/b8049531-bcf6-4f19-84d2-b974b9236223-kube-api-access-kqdb8\") pod \"b8049531-bcf6-4f19-84d2-b974b9236223\" (UID: \"b8049531-bcf6-4f19-84d2-b974b9236223\") " Oct 31 00:55:59.413286 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:59.413275 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73487e30-1bc6-49b2-b07b-1b004137232e-tmp-dir" pod "73487e30-1bc6-49b2-b07b-1b004137232e" (UID: "73487e30-1bc6-49b2-b07b-1b004137232e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:55:59.413894 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:59.413307 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73487e30-1bc6-49b2-b07b-1b004137232e-home\" (UniqueName: \"kubernetes.io/empty-dir/73487e30-1bc6-49b2-b07b-1b004137232e-home\") pod \"73487e30-1bc6-49b2-b07b-1b004137232e\" (UID: \"73487e30-1bc6-49b2-b07b-1b004137232e\") " Oct 31 00:55:59.413894 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:59.413408 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8049531-bcf6-4f19-84d2-b974b9236223-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8049531-bcf6-4f19-84d2-b974b9236223-tmp-dir\") pod \"b8049531-bcf6-4f19-84d2-b974b9236223\" (UID: \"b8049531-bcf6-4f19-84d2-b974b9236223\") " Oct 31 00:55:59.413894 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:59.413477 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73487e30-1bc6-49b2-b07b-1b004137232e-model-cache" pod "73487e30-1bc6-49b2-b07b-1b004137232e" (UID: "73487e30-1bc6-49b2-b07b-1b004137232e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:55:59.413894 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:59.413636 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8049531-bcf6-4f19-84d2-b974b9236223-tmp-dir" pod "b8049531-bcf6-4f19-84d2-b974b9236223" (UID: "b8049531-bcf6-4f19-84d2-b974b9236223"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:55:59.413894 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:59.413692 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbf98c81-6f05-41b8-ba3b-c10ecbbff5d8-tmp-dir" pod "fbf98c81-6f05-41b8-ba3b-c10ecbbff5d8" (UID: "fbf98c81-6f05-41b8-ba3b-c10ecbbff5d8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:55:59.413894 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:59.413806 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8049531-bcf6-4f19-84d2-b974b9236223-home" pod "b8049531-bcf6-4f19-84d2-b974b9236223" (UID: "b8049531-bcf6-4f19-84d2-b974b9236223"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:55:59.414277 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:59.414016 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fbf98c81-6f05-41b8-ba3b-c10ecbbff5d8-tls-certs\" (UniqueName: \"kubernetes.io/secret/fbf98c81-6f05-41b8-ba3b-c10ecbbff5d8-tls-certs\") pod \"fbf98c81-6f05-41b8-ba3b-c10ecbbff5d8\" (UID: \"fbf98c81-6f05-41b8-ba3b-c10ecbbff5d8\") " Oct 31 00:55:59.414277 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:59.414020 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73487e30-1bc6-49b2-b07b-1b004137232e-home" pod "73487e30-1bc6-49b2-b07b-1b004137232e" (UID: "73487e30-1bc6-49b2-b07b-1b004137232e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:55:59.414277 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:59.414060 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73487e30-1bc6-49b2-b07b-1b004137232e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/73487e30-1bc6-49b2-b07b-1b004137232e-dshm\") pod \"73487e30-1bc6-49b2-b07b-1b004137232e\" (UID: \"73487e30-1bc6-49b2-b07b-1b004137232e\") " Oct 31 00:55:59.414277 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:59.414105 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8049531-bcf6-4f19-84d2-b974b9236223-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8049531-bcf6-4f19-84d2-b974b9236223-dshm\") pod \"b8049531-bcf6-4f19-84d2-b974b9236223\" (UID: \"b8049531-bcf6-4f19-84d2-b974b9236223\") " Oct 31 00:55:59.414703 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:59.414645 5222 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8049531-bcf6-4f19-84d2-b974b9236223-model-cache\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:59.414703 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:59.414669 5222 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbf98c81-6f05-41b8-ba3b-c10ecbbff5d8-home\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:59.414703 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:59.414685 5222 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73487e30-1bc6-49b2-b07b-1b004137232e-model-cache\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:59.414703 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:59.414688 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbf98c81-6f05-41b8-ba3b-c10ecbbff5d8-model-cache" pod "fbf98c81-6f05-41b8-ba3b-c10ecbbff5d8" (UID: "fbf98c81-6f05-41b8-ba3b-c10ecbbff5d8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:55:59.414703 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:59.414700 5222 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73487e30-1bc6-49b2-b07b-1b004137232e-tmp-dir\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:59.414986 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:59.414715 5222 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbf98c81-6f05-41b8-ba3b-c10ecbbff5d8-tmp-dir\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:59.414986 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:59.414726 5222 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8049531-bcf6-4f19-84d2-b974b9236223-home\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:59.414986 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:59.414740 5222 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73487e30-1bc6-49b2-b07b-1b004137232e-home\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:59.414986 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:59.414752 5222 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8049531-bcf6-4f19-84d2-b974b9236223-tmp-dir\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:59.417877 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:59.417842 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73487e30-1bc6-49b2-b07b-1b004137232e-dshm" pod "73487e30-1bc6-49b2-b07b-1b004137232e" (UID: "73487e30-1bc6-49b2-b07b-1b004137232e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:55:59.418342 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:59.418227 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73487e30-1bc6-49b2-b07b-1b004137232e-kube-api-access-s4598" pod "73487e30-1bc6-49b2-b07b-1b004137232e" (UID: "73487e30-1bc6-49b2-b07b-1b004137232e"). InnerVolumeSpecName "kube-api-access-s4598". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:55:59.418342 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:59.418234 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73487e30-1bc6-49b2-b07b-1b004137232e-tls-certs" pod "73487e30-1bc6-49b2-b07b-1b004137232e" (UID: "73487e30-1bc6-49b2-b07b-1b004137232e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:55:59.418342 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:59.418295 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbf98c81-6f05-41b8-ba3b-c10ecbbff5d8-dshm" pod "fbf98c81-6f05-41b8-ba3b-c10ecbbff5d8" (UID: "fbf98c81-6f05-41b8-ba3b-c10ecbbff5d8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:55:59.418342 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:59.418304 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbf98c81-6f05-41b8-ba3b-c10ecbbff5d8-kube-api-access-459r7" pod "fbf98c81-6f05-41b8-ba3b-c10ecbbff5d8" (UID: "fbf98c81-6f05-41b8-ba3b-c10ecbbff5d8"). InnerVolumeSpecName "kube-api-access-459r7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:55:59.418673 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:59.418395 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbf98c81-6f05-41b8-ba3b-c10ecbbff5d8-tls-certs" pod "fbf98c81-6f05-41b8-ba3b-c10ecbbff5d8" (UID: "fbf98c81-6f05-41b8-ba3b-c10ecbbff5d8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:55:59.418673 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:59.418571 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8049531-bcf6-4f19-84d2-b974b9236223-dshm" pod "b8049531-bcf6-4f19-84d2-b974b9236223" (UID: "b8049531-bcf6-4f19-84d2-b974b9236223"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:55:59.419100 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:59.419070 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8049531-bcf6-4f19-84d2-b974b9236223-kube-api-access-kqdb8" pod "b8049531-bcf6-4f19-84d2-b974b9236223" (UID: "b8049531-bcf6-4f19-84d2-b974b9236223"). InnerVolumeSpecName "kube-api-access-kqdb8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:55:59.419902 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:59.419880 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8049531-bcf6-4f19-84d2-b974b9236223-tls-certs" pod "b8049531-bcf6-4f19-84d2-b974b9236223" (UID: "b8049531-bcf6-4f19-84d2-b974b9236223"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:55:59.516931 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:59.516898 5222 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbf98c81-6f05-41b8-ba3b-c10ecbbff5d8-model-cache\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:59.516931 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:59.516928 5222 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73487e30-1bc6-49b2-b07b-1b004137232e-tls-certs\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:59.516931 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:59.516939 5222 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbf98c81-6f05-41b8-ba3b-c10ecbbff5d8-dshm\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:59.517218 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:59.516950 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-459r7\" (UniqueName: \"kubernetes.io/projected/fbf98c81-6f05-41b8-ba3b-c10ecbbff5d8-kube-api-access-459r7\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:59.517218 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:59.516960 5222 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8049531-bcf6-4f19-84d2-b974b9236223-tls-certs\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:59.517218 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:59.516968 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4598\" (UniqueName: \"kubernetes.io/projected/73487e30-1bc6-49b2-b07b-1b004137232e-kube-api-access-s4598\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:59.517218 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:59.516977 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqdb8\" (UniqueName: \"kubernetes.io/projected/b8049531-bcf6-4f19-84d2-b974b9236223-kube-api-access-kqdb8\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:59.517218 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:59.516985 5222 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbf98c81-6f05-41b8-ba3b-c10ecbbff5d8-tls-certs\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:59.517218 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:59.516993 5222 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73487e30-1bc6-49b2-b07b-1b004137232e-dshm\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:59.517218 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:59.517001 5222 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8049531-bcf6-4f19-84d2-b974b9236223-dshm\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:59.919495 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:59.919141 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-412743-688d66585f-vk5r7" Oct 31 00:55:59.923005 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:59.922555 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a99cbf31-f3f2-4d16-a665-58f0f5af77f0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a99cbf31-f3f2-4d16-a665-58f0f5af77f0-plugins-config-volume\") pod \"a99cbf31-f3f2-4d16-a665-58f0f5af77f0\" (UID: \"a99cbf31-f3f2-4d16-a665-58f0f5af77f0\") " Oct 31 00:55:59.923005 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:59.922595 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a99cbf31-f3f2-4d16-a665-58f0f5af77f0-kube-api-access-79v4x\" (UniqueName: \"kubernetes.io/projected/a99cbf31-f3f2-4d16-a665-58f0f5af77f0-kube-api-access-79v4x\") pod \"a99cbf31-f3f2-4d16-a665-58f0f5af77f0\" (UID: \"a99cbf31-f3f2-4d16-a665-58f0f5af77f0\") " Oct 31 00:55:59.923005 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:59.922956 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a99cbf31-f3f2-4d16-a665-58f0f5af77f0-plugins-config-volume" pod "a99cbf31-f3f2-4d16-a665-58f0f5af77f0" (UID: "a99cbf31-f3f2-4d16-a665-58f0f5af77f0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:55:59.924957 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:59.924934 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a99cbf31-f3f2-4d16-a665-58f0f5af77f0-kube-api-access-79v4x" pod "a99cbf31-f3f2-4d16-a665-58f0f5af77f0" (UID: "a99cbf31-f3f2-4d16-a665-58f0f5af77f0"). InnerVolumeSpecName "kube-api-access-79v4x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:55:59.989171 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:55:59.989141 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-412743-6d5cb5bd7f-pzb7l" Oct 31 00:56:00.023469 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:00.023432 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9dbd9108-ac33-45a0-9eed-ef9b81755398-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9dbd9108-ac33-45a0-9eed-ef9b81755398-plugins-config-volume\") pod \"9dbd9108-ac33-45a0-9eed-ef9b81755398\" (UID: \"9dbd9108-ac33-45a0-9eed-ef9b81755398\") " Oct 31 00:56:00.023678 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:00.023585 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9dbd9108-ac33-45a0-9eed-ef9b81755398-kube-api-access-9vbf4\" (UniqueName: \"kubernetes.io/projected/9dbd9108-ac33-45a0-9eed-ef9b81755398-kube-api-access-9vbf4\") pod \"9dbd9108-ac33-45a0-9eed-ef9b81755398\" (UID: \"9dbd9108-ac33-45a0-9eed-ef9b81755398\") " Oct 31 00:56:00.023942 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:00.023914 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9dbd9108-ac33-45a0-9eed-ef9b81755398-plugins-config-volume" pod "9dbd9108-ac33-45a0-9eed-ef9b81755398" (UID: "9dbd9108-ac33-45a0-9eed-ef9b81755398"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:56:00.024498 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:00.024473 5222 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a99cbf31-f3f2-4d16-a665-58f0f5af77f0-plugins-config-volume\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:00.024621 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:00.024502 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79v4x\" (UniqueName: \"kubernetes.io/projected/a99cbf31-f3f2-4d16-a665-58f0f5af77f0-kube-api-access-79v4x\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:00.024621 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:00.024518 5222 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9dbd9108-ac33-45a0-9eed-ef9b81755398-plugins-config-volume\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:00.026131 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:00.026105 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dbd9108-ac33-45a0-9eed-ef9b81755398-kube-api-access-9vbf4" pod "9dbd9108-ac33-45a0-9eed-ef9b81755398" (UID: "9dbd9108-ac33-45a0-9eed-ef9b81755398"). InnerVolumeSpecName "kube-api-access-9vbf4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:56:00.027322 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:00.027299 5222 generic.go:356] "Generic (PLEG): container finished" podID="a99cbf31-f3f2-4d16-a665-58f0f5af77f0" containerID="66d2dcee34673511c9d5bdcc236e9cac8f2e3852170c01d565e384bf000e246a" exitCode=0 Oct 31 00:56:00.027441 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:00.027360 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-412743-688d66585f-vk5r7" Oct 31 00:56:00.027441 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:00.027402 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-412743-688d66585f-vk5r7" event={"ID":"a99cbf31-f3f2-4d16-a665-58f0f5af77f0","Type":"ContainerDied","Data":"66d2dcee34673511c9d5bdcc236e9cac8f2e3852170c01d565e384bf000e246a"} Oct 31 00:56:00.027548 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:00.027475 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-412743-688d66585f-vk5r7" event={"ID":"a99cbf31-f3f2-4d16-a665-58f0f5af77f0","Type":"ContainerDied","Data":"7b37376c0e917b8e04dcfa56028b71bda9770e1f3e0725d89b826da1d5294220"} Oct 31 00:56:00.027548 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:00.027511 5222 scope.go:122] "RemoveContainer" containerID="66d2dcee34673511c9d5bdcc236e9cac8f2e3852170c01d565e384bf000e246a" Oct 31 00:56:00.029356 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:00.029336 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-101b9e/distinct-2-w0-101b9e-kserve-74548f6d6-v5phc" Oct 31 00:56:00.029473 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:00.029418 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-101b9e/distinct-2-w0-101b9e-kserve-74548f6d6-v5phc" event={"ID":"b8049531-bcf6-4f19-84d2-b974b9236223","Type":"ContainerDied","Data":"2db2c97e6643f3877978cccc02df5d74e80f8398c7d9940245d2e7abbc55298d"} Oct 31 00:56:00.046615 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:00.046576 5222 generic.go:356] "Generic (PLEG): container finished" podID="9dbd9108-ac33-45a0-9eed-ef9b81755398" containerID="c28d50fb6af5559cb39a55913a3db1d9d96be0c43e6c5ab96f9c35b102cbecf4" exitCode=0 Oct 31 00:56:00.046819 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:00.046800 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-412743-6d5cb5bd7f-pzb7l" Oct 31 00:56:00.047639 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:00.047472 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-412743-6d5cb5bd7f-pzb7l" event={"ID":"9dbd9108-ac33-45a0-9eed-ef9b81755398","Type":"ContainerDied","Data":"c28d50fb6af5559cb39a55913a3db1d9d96be0c43e6c5ab96f9c35b102cbecf4"} Oct 31 00:56:00.047639 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:00.047534 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-412743-6d5cb5bd7f-pzb7l" event={"ID":"9dbd9108-ac33-45a0-9eed-ef9b81755398","Type":"ContainerDied","Data":"18353d6fa5213acc2dee87b0262306f9097e146d70c1e0891135e82b948eddab"} Oct 31 00:56:00.050353 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:00.049660 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-101b9e/distinct-w0-101b9e-kserve-6dcc4cc94f-kwv9w" event={"ID":"73487e30-1bc6-49b2-b07b-1b004137232e","Type":"ContainerDied","Data":"507449e4d357c1c7f493084b6278c275c890cab59d60a41488cbc6669d87a312"} Oct 31 00:56:00.050353 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:00.049748 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-101b9e/distinct-w0-101b9e-kserve-6dcc4cc94f-kwv9w" Oct 31 00:56:00.052197 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:00.052086 5222 scope.go:122] "RemoveContainer" containerID="66d2dcee34673511c9d5bdcc236e9cac8f2e3852170c01d565e384bf000e246a" Oct 31 00:56:00.052542 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:56:00.052515 5222 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66d2dcee34673511c9d5bdcc236e9cac8f2e3852170c01d565e384bf000e246a\": container with ID starting with 66d2dcee34673511c9d5bdcc236e9cac8f2e3852170c01d565e384bf000e246a not found: ID does not exist" containerID="66d2dcee34673511c9d5bdcc236e9cac8f2e3852170c01d565e384bf000e246a" Oct 31 00:56:00.052645 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:00.052554 5222 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66d2dcee34673511c9d5bdcc236e9cac8f2e3852170c01d565e384bf000e246a"} err="failed to get container status \"66d2dcee34673511c9d5bdcc236e9cac8f2e3852170c01d565e384bf000e246a\": rpc error: code = NotFound desc = could not find container \"66d2dcee34673511c9d5bdcc236e9cac8f2e3852170c01d565e384bf000e246a\": container with ID starting with 66d2dcee34673511c9d5bdcc236e9cac8f2e3852170c01d565e384bf000e246a not found: ID does not exist" Oct 31 00:56:00.052645 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:00.052580 5222 scope.go:122] "RemoveContainer" containerID="3de9a26b71ba3e866fdf0760764d8ec12776a0d1d10a27cd71ede4821cad7543" Oct 31 00:56:00.057040 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:00.055905 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-101b9e/premium-w0-101b9e-kserve-dc65f694b-m4tsk" event={"ID":"fbf98c81-6f05-41b8-ba3b-c10ecbbff5d8","Type":"ContainerDied","Data":"25f790bd521fe1376d4fbb0f9854d13eeae899987403a28c37abebeb4f57b0c5"} Oct 31 00:56:00.057040 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:00.055993 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-101b9e/premium-w0-101b9e-kserve-dc65f694b-m4tsk" Oct 31 00:56:00.070489 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:00.070463 5222 scope.go:122] "RemoveContainer" containerID="c28d50fb6af5559cb39a55913a3db1d9d96be0c43e6c5ab96f9c35b102cbecf4" Oct 31 00:56:00.081490 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:00.081470 5222 scope.go:122] "RemoveContainer" containerID="c28d50fb6af5559cb39a55913a3db1d9d96be0c43e6c5ab96f9c35b102cbecf4" Oct 31 00:56:00.081824 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:56:00.081799 5222 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c28d50fb6af5559cb39a55913a3db1d9d96be0c43e6c5ab96f9c35b102cbecf4\": container with ID starting with c28d50fb6af5559cb39a55913a3db1d9d96be0c43e6c5ab96f9c35b102cbecf4 not found: ID does not exist" containerID="c28d50fb6af5559cb39a55913a3db1d9d96be0c43e6c5ab96f9c35b102cbecf4" Oct 31 00:56:00.081905 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:00.081834 5222 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c28d50fb6af5559cb39a55913a3db1d9d96be0c43e6c5ab96f9c35b102cbecf4"} err="failed to get container status \"c28d50fb6af5559cb39a55913a3db1d9d96be0c43e6c5ab96f9c35b102cbecf4\": rpc error: code = NotFound desc = could not find container \"c28d50fb6af5559cb39a55913a3db1d9d96be0c43e6c5ab96f9c35b102cbecf4\": container with ID starting with c28d50fb6af5559cb39a55913a3db1d9d96be0c43e6c5ab96f9c35b102cbecf4 not found: ID does not exist" Oct 31 00:56:00.081905 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:00.081856 5222 scope.go:122] "RemoveContainer" containerID="edb07311cd170b00ceaadc005648108ea352ce910f7179de453809f9ebc1ad5d" Oct 31 00:56:00.091574 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:00.091555 5222 scope.go:122] "RemoveContainer" containerID="3e93fc0c53f37807e7cb279c8763e1f1bf5e2414e5424875b92b886d6127dcbe" Oct 31 00:56:00.125776 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:00.125742 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vbf4\" (UniqueName: \"kubernetes.io/projected/9dbd9108-ac33-45a0-9eed-ef9b81755398-kube-api-access-9vbf4\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:00.159109 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:00.159023 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-162fe38d-xcsh4"] Oct 31 00:56:00.159604 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:00.159587 5222 cpu_manager.go:414] "containerMap: removing container" podUID="b8049531-bcf6-4f19-84d2-b974b9236223" containerName="main" Oct 31 00:56:00.159656 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:00.159606 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8049531-bcf6-4f19-84d2-b974b9236223" containerName="main" Oct 31 00:56:00.159656 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:00.159622 5222 cpu_manager.go:414] "containerMap: removing container" podUID="2f6cbca5-d586-4bc2-9114-7212e9cc8a01" containerName="main" Oct 31 00:56:00.159656 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:00.159627 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f6cbca5-d586-4bc2-9114-7212e9cc8a01" containerName="main" Oct 31 00:56:00.159656 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:00.159638 5222 cpu_manager.go:414] "containerMap: removing container" podUID="68d7fd60-bcad-45f7-84c0-11424e26f26d" containerName="main" Oct 31 00:56:00.159656 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:00.159643 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68d7fd60-bcad-45f7-84c0-11424e26f26d" containerName="main" Oct 31 00:56:00.159656 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:00.159653 5222 cpu_manager.go:414] "containerMap: removing container" podUID="9dbd9108-ac33-45a0-9eed-ef9b81755398" containerName="payload-pre-processing" Oct 31 00:56:00.159865 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:00.159658 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9dbd9108-ac33-45a0-9eed-ef9b81755398" containerName="payload-pre-processing" Oct 31 00:56:00.159865 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:00.159671 5222 cpu_manager.go:414] "containerMap: removing container" podUID="73487e30-1bc6-49b2-b07b-1b004137232e" containerName="main" Oct 31 00:56:00.159865 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:00.159676 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73487e30-1bc6-49b2-b07b-1b004137232e" containerName="main" Oct 31 00:56:00.159865 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:00.159683 5222 cpu_manager.go:414] "containerMap: removing container" podUID="7747c507-e71c-45eb-8ffb-38b61253999c" containerName="istio-proxy" Oct 31 00:56:00.159865 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:00.159689 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7747c507-e71c-45eb-8ffb-38b61253999c" containerName="istio-proxy" Oct 31 00:56:00.159865 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:00.159699 5222 cpu_manager.go:414] "containerMap: removing container" podUID="d0e8d5c5-aa08-41f6-badf-792648a20a68" containerName="main" Oct 31 00:56:00.159865 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:00.159703 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0e8d5c5-aa08-41f6-badf-792648a20a68" containerName="main" Oct 31 00:56:00.159865 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:00.159715 5222 cpu_manager.go:414] "containerMap: removing container" podUID="a99cbf31-f3f2-4d16-a665-58f0f5af77f0" containerName="payload-processing" Oct 31 00:56:00.159865 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:00.159719 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a99cbf31-f3f2-4d16-a665-58f0f5af77f0" containerName="payload-processing" Oct 31 00:56:00.159865 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:00.159729 5222 cpu_manager.go:414] "containerMap: removing container" podUID="fbf98c81-6f05-41b8-ba3b-c10ecbbff5d8" containerName="main" Oct 31 00:56:00.159865 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:00.159733 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbf98c81-6f05-41b8-ba3b-c10ecbbff5d8" containerName="main" Oct 31 00:56:00.159865 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:00.159740 5222 cpu_manager.go:414] "containerMap: removing container" podUID="6eefacd4-395f-42f5-922d-fb2d971401c5" containerName="revoke-keys" Oct 31 00:56:00.159865 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:00.159744 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6eefacd4-395f-42f5-922d-fb2d971401c5" containerName="revoke-keys" Oct 31 00:56:00.159865 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:00.159814 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f6cbca5-d586-4bc2-9114-7212e9cc8a01" containerName="main" Oct 31 00:56:00.159865 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:00.159825 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="68d7fd60-bcad-45f7-84c0-11424e26f26d" containerName="main" Oct 31 00:56:00.159865 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:00.159830 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="9dbd9108-ac33-45a0-9eed-ef9b81755398" containerName="payload-pre-processing" Oct 31 00:56:00.159865 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:00.159836 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="73487e30-1bc6-49b2-b07b-1b004137232e" containerName="main" Oct 31 00:56:00.159865 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:00.159843 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="7747c507-e71c-45eb-8ffb-38b61253999c" containerName="istio-proxy" Oct 31 00:56:00.159865 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:00.159847 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0e8d5c5-aa08-41f6-badf-792648a20a68" containerName="main" Oct 31 00:56:00.159865 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:00.159854 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="a99cbf31-f3f2-4d16-a665-58f0f5af77f0" containerName="payload-processing" Oct 31 00:56:00.159865 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:00.159860 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbf98c81-6f05-41b8-ba3b-c10ecbbff5d8" containerName="main" Oct 31 00:56:00.159865 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:00.159864 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="6eefacd4-395f-42f5-922d-fb2d971401c5" containerName="revoke-keys" Oct 31 00:56:00.159865 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:00.159872 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8049531-bcf6-4f19-84d2-b974b9236223" containerName="main" Oct 31 00:56:00.164637 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:00.164620 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-162fe38d-xcsh4" Oct 31 00:56:00.216820 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:00.216785 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-162fe38d-xcsh4"] Oct 31 00:56:00.219490 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:00.219464 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:56:00.219490 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:00.219469 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:56:00.219671 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:00.219550 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:56:00.219711 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:00.219696 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfz9z\"" Oct 31 00:56:00.223273 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:00.223246 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ef266b8-vkmzd"] Oct 31 00:56:00.227178 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:00.227156 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/343c42b7-22b0-4f9d-8be7-0ffbd7ce64e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-162fe38d-xcsh4\" (UID: \"343c42b7-22b0-4f9d-8be7-0ffbd7ce64e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-162fe38d-xcsh4" Oct 31 00:56:00.227287 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:00.227202 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/343c42b7-22b0-4f9d-8be7-0ffbd7ce64e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-162fe38d-xcsh4\" (UID: \"343c42b7-22b0-4f9d-8be7-0ffbd7ce64e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-162fe38d-xcsh4" Oct 31 00:56:00.227287 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:00.227222 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42vtx\" (UniqueName: \"kubernetes.io/projected/343c42b7-22b0-4f9d-8be7-0ffbd7ce64e1-kube-api-access-42vtx\") pod \"maas-api-revoke-sub-162fe38d-xcsh4\" (UID: \"343c42b7-22b0-4f9d-8be7-0ffbd7ce64e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-162fe38d-xcsh4" Oct 31 00:56:00.227287 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:00.227236 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ef266b8-vkmzd" Oct 31 00:56:00.315186 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:00.315153 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ef266b8-vkmzd"] Oct 31 00:56:00.329005 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:00.328965 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/684d2727-5c85-4374-bc7f-69634fd00e17-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ef266b8-vkmzd\" (UID: \"684d2727-5c85-4374-bc7f-69634fd00e17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ef266b8-vkmzd" Oct 31 00:56:00.329228 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:00.329206 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/684d2727-5c85-4374-bc7f-69634fd00e17-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ef266b8-vkmzd\" (UID: \"684d2727-5c85-4374-bc7f-69634fd00e17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ef266b8-vkmzd" Oct 31 00:56:00.329304 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:00.329276 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/343c42b7-22b0-4f9d-8be7-0ffbd7ce64e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-162fe38d-xcsh4\" (UID: \"343c42b7-22b0-4f9d-8be7-0ffbd7ce64e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-162fe38d-xcsh4" Oct 31 00:56:00.338699 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:00.331923 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/343c42b7-22b0-4f9d-8be7-0ffbd7ce64e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-162fe38d-xcsh4\" (UID: \"343c42b7-22b0-4f9d-8be7-0ffbd7ce64e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-162fe38d-xcsh4" Oct 31 00:56:00.338699 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:00.331996 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42vtx\" (UniqueName: \"kubernetes.io/projected/343c42b7-22b0-4f9d-8be7-0ffbd7ce64e1-kube-api-access-42vtx\") pod \"maas-api-revoke-sub-162fe38d-xcsh4\" (UID: \"343c42b7-22b0-4f9d-8be7-0ffbd7ce64e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-162fe38d-xcsh4" Oct 31 00:56:00.338699 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:00.332413 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrhmn\" (UniqueName: \"kubernetes.io/projected/684d2727-5c85-4374-bc7f-69634fd00e17-kube-api-access-vrhmn\") pod \"maas-api-revoke-sub-9ef266b8-vkmzd\" (UID: \"684d2727-5c85-4374-bc7f-69634fd00e17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ef266b8-vkmzd" Oct 31 00:56:00.338699 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:00.332907 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/343c42b7-22b0-4f9d-8be7-0ffbd7ce64e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-162fe38d-xcsh4\" (UID: \"343c42b7-22b0-4f9d-8be7-0ffbd7ce64e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-162fe38d-xcsh4" Oct 31 00:56:00.342522 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:00.342461 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-101b9e/distinct-2-w0-101b9e-kserve-74548f6d6-v5phc"] Oct 31 00:56:00.345733 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:00.345700 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-101b9e/distinct-2-w0-101b9e-kserve-74548f6d6-v5phc"] Oct 31 00:56:00.345733 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:00.345725 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/343c42b7-22b0-4f9d-8be7-0ffbd7ce64e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-162fe38d-xcsh4\" (UID: \"343c42b7-22b0-4f9d-8be7-0ffbd7ce64e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-162fe38d-xcsh4" Oct 31 00:56:00.394758 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:00.394726 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42vtx\" (UniqueName: \"kubernetes.io/projected/343c42b7-22b0-4f9d-8be7-0ffbd7ce64e1-kube-api-access-42vtx\") pod \"maas-api-revoke-sub-162fe38d-xcsh4\" (UID: \"343c42b7-22b0-4f9d-8be7-0ffbd7ce64e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-162fe38d-xcsh4" Oct 31 00:56:00.434145 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:00.434056 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/684d2727-5c85-4374-bc7f-69634fd00e17-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ef266b8-vkmzd\" (UID: \"684d2727-5c85-4374-bc7f-69634fd00e17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ef266b8-vkmzd" Oct 31 00:56:00.434145 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:00.434123 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrhmn\" (UniqueName: \"kubernetes.io/projected/684d2727-5c85-4374-bc7f-69634fd00e17-kube-api-access-vrhmn\") pod \"maas-api-revoke-sub-9ef266b8-vkmzd\" (UID: \"684d2727-5c85-4374-bc7f-69634fd00e17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ef266b8-vkmzd" Oct 31 00:56:00.434344 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:00.434175 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/684d2727-5c85-4374-bc7f-69634fd00e17-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ef266b8-vkmzd\" (UID: \"684d2727-5c85-4374-bc7f-69634fd00e17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ef266b8-vkmzd" Oct 31 00:56:00.434785 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:00.434760 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/684d2727-5c85-4374-bc7f-69634fd00e17-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ef266b8-vkmzd\" (UID: \"684d2727-5c85-4374-bc7f-69634fd00e17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ef266b8-vkmzd" Oct 31 00:56:00.436841 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:00.436812 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/684d2727-5c85-4374-bc7f-69634fd00e17-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ef266b8-vkmzd\" (UID: \"684d2727-5c85-4374-bc7f-69634fd00e17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ef266b8-vkmzd" Oct 31 00:56:00.449197 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:00.449170 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-101b9e/distinct-w0-101b9e-kserve-6dcc4cc94f-kwv9w"] Oct 31 00:56:00.457659 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:00.457629 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-101b9e/distinct-w0-101b9e-kserve-6dcc4cc94f-kwv9w"] Oct 31 00:56:00.478359 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:00.478328 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-162fe38d-xcsh4" Oct 31 00:56:00.496491 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:00.496465 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrhmn\" (UniqueName: \"kubernetes.io/projected/684d2727-5c85-4374-bc7f-69634fd00e17-kube-api-access-vrhmn\") pod \"maas-api-revoke-sub-9ef266b8-vkmzd\" (UID: \"684d2727-5c85-4374-bc7f-69634fd00e17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ef266b8-vkmzd" Oct 31 00:56:00.542794 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:00.542754 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ef266b8-vkmzd" Oct 31 00:56:00.802306 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:00.802265 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-101b9e/premium-w0-101b9e-kserve-dc65f694b-m4tsk"] Oct 31 00:56:00.806800 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:00.806769 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-101b9e/premium-w0-101b9e-kserve-dc65f694b-m4tsk"] Oct 31 00:56:00.852202 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:56:00.852168 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod343c42b7_22b0_4f9d_8be7_0ffbd7ce64e1.slice/crio-0373ec7340604008efd6f60f23a15b9cb3cf68e24c92d67d74ca1ea80591107c WatchSource:0}: Error finding container 0373ec7340604008efd6f60f23a15b9cb3cf68e24c92d67d74ca1ea80591107c: Status 404 returned error can't find the container with id 0373ec7340604008efd6f60f23a15b9cb3cf68e24c92d67d74ca1ea80591107c Oct 31 00:56:00.855176 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:00.855158 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-162fe38d-xcsh4"] Oct 31 00:56:00.917892 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:00.917854 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-412743-688d66585f-vk5r7"] Oct 31 00:56:00.921806 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:00.921778 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-412743-688d66585f-vk5r7"] Oct 31 00:56:00.931050 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:56:00.931006 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod684d2727_5c85_4374_bc7f_69634fd00e17.slice/crio-9a6891fa616d6424152b4d1d0eda96e83044c4029e94213cfcb262ec80953630 WatchSource:0}: Error finding container 9a6891fa616d6424152b4d1d0eda96e83044c4029e94213cfcb262ec80953630: Status 404 returned error can't find the container with id 9a6891fa616d6424152b4d1d0eda96e83044c4029e94213cfcb262ec80953630 Oct 31 00:56:00.934139 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:00.934117 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ef266b8-vkmzd"] Oct 31 00:56:01.035552 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:01.033485 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-412743-6d5cb5bd7f-pzb7l"] Oct 31 00:56:01.061186 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:01.061123 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-412743-6d5cb5bd7f-pzb7l"] Oct 31 00:56:01.074845 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:01.074810 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-162fe38d-xcsh4" event={"ID":"343c42b7-22b0-4f9d-8be7-0ffbd7ce64e1","Type":"ContainerStarted","Data":"0373ec7340604008efd6f60f23a15b9cb3cf68e24c92d67d74ca1ea80591107c"} Oct 31 00:56:01.078907 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:01.078868 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ef266b8-vkmzd" event={"ID":"684d2727-5c85-4374-bc7f-69634fd00e17","Type":"ContainerStarted","Data":"9a6891fa616d6424152b4d1d0eda96e83044c4029e94213cfcb262ec80953630"} Oct 31 00:56:01.659129 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:01.659087 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73487e30-1bc6-49b2-b07b-1b004137232e" path="/var/lib/kubelet/pods/73487e30-1bc6-49b2-b07b-1b004137232e/volumes" Oct 31 00:56:01.659573 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:01.659557 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9dbd9108-ac33-45a0-9eed-ef9b81755398" path="/var/lib/kubelet/pods/9dbd9108-ac33-45a0-9eed-ef9b81755398/volumes" Oct 31 00:56:01.659904 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:01.659887 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a99cbf31-f3f2-4d16-a665-58f0f5af77f0" path="/var/lib/kubelet/pods/a99cbf31-f3f2-4d16-a665-58f0f5af77f0/volumes" Oct 31 00:56:01.660784 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:01.660761 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8049531-bcf6-4f19-84d2-b974b9236223" path="/var/lib/kubelet/pods/b8049531-bcf6-4f19-84d2-b974b9236223/volumes" Oct 31 00:56:01.661296 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:01.661281 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbf98c81-6f05-41b8-ba3b-c10ecbbff5d8" path="/var/lib/kubelet/pods/fbf98c81-6f05-41b8-ba3b-c10ecbbff5d8/volumes" Oct 31 00:56:02.092114 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:02.092082 5222 generic.go:356] "Generic (PLEG): container finished" podID="343c42b7-22b0-4f9d-8be7-0ffbd7ce64e1" containerID="2367c2763e7e539f66669c4ca65bcd1c2e193dd1ff670567ba15c0df69f39857" exitCode=0 Oct 31 00:56:02.092606 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:02.092179 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-162fe38d-xcsh4" event={"ID":"343c42b7-22b0-4f9d-8be7-0ffbd7ce64e1","Type":"ContainerDied","Data":"2367c2763e7e539f66669c4ca65bcd1c2e193dd1ff670567ba15c0df69f39857"} Oct 31 00:56:02.094075 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:02.094041 5222 generic.go:356] "Generic (PLEG): container finished" podID="684d2727-5c85-4374-bc7f-69634fd00e17" containerID="c0ffd0c565dae97df963b3ef8f27203430423a2ddaca0f17ba1ca7cbe777e327" exitCode=0 Oct 31 00:56:02.094216 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:02.094121 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ef266b8-vkmzd" event={"ID":"684d2727-5c85-4374-bc7f-69634fd00e17","Type":"ContainerDied","Data":"c0ffd0c565dae97df963b3ef8f27203430423a2ddaca0f17ba1ca7cbe777e327"} Oct 31 00:56:03.276211 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:03.276145 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-162fe38d-xcsh4" Oct 31 00:56:03.279590 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:03.279568 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ef266b8-vkmzd" Oct 31 00:56:03.373550 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:03.373453 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/684d2727-5c85-4374-bc7f-69634fd00e17-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/684d2727-5c85-4374-bc7f-69634fd00e17-maas-api-openshift-ca\") pod \"684d2727-5c85-4374-bc7f-69634fd00e17\" (UID: \"684d2727-5c85-4374-bc7f-69634fd00e17\") " Oct 31 00:56:03.373550 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:03.373527 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/343c42b7-22b0-4f9d-8be7-0ffbd7ce64e1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/343c42b7-22b0-4f9d-8be7-0ffbd7ce64e1-maas-api-openshift-ca\") pod \"343c42b7-22b0-4f9d-8be7-0ffbd7ce64e1\" (UID: \"343c42b7-22b0-4f9d-8be7-0ffbd7ce64e1\") " Oct 31 00:56:03.373796 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:03.373641 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/343c42b7-22b0-4f9d-8be7-0ffbd7ce64e1-kube-api-access-42vtx\" (UniqueName: \"kubernetes.io/projected/343c42b7-22b0-4f9d-8be7-0ffbd7ce64e1-kube-api-access-42vtx\") pod \"343c42b7-22b0-4f9d-8be7-0ffbd7ce64e1\" (UID: \"343c42b7-22b0-4f9d-8be7-0ffbd7ce64e1\") " Oct 31 00:56:03.373796 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:03.373780 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/684d2727-5c85-4374-bc7f-69634fd00e17-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/684d2727-5c85-4374-bc7f-69634fd00e17-maas-api-xks-ca\") pod \"684d2727-5c85-4374-bc7f-69634fd00e17\" (UID: \"684d2727-5c85-4374-bc7f-69634fd00e17\") " Oct 31 00:56:03.373915 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:03.373863 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/343c42b7-22b0-4f9d-8be7-0ffbd7ce64e1-maas-api-openshift-ca" pod "343c42b7-22b0-4f9d-8be7-0ffbd7ce64e1" (UID: "343c42b7-22b0-4f9d-8be7-0ffbd7ce64e1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:56:03.373915 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:03.373870 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/684d2727-5c85-4374-bc7f-69634fd00e17-maas-api-openshift-ca" pod "684d2727-5c85-4374-bc7f-69634fd00e17" (UID: "684d2727-5c85-4374-bc7f-69634fd00e17"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:56:03.373915 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:03.373898 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/343c42b7-22b0-4f9d-8be7-0ffbd7ce64e1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/343c42b7-22b0-4f9d-8be7-0ffbd7ce64e1-maas-api-xks-ca\") pod \"343c42b7-22b0-4f9d-8be7-0ffbd7ce64e1\" (UID: \"343c42b7-22b0-4f9d-8be7-0ffbd7ce64e1\") " Oct 31 00:56:03.374069 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:03.373947 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/684d2727-5c85-4374-bc7f-69634fd00e17-kube-api-access-vrhmn\" (UniqueName: \"kubernetes.io/projected/684d2727-5c85-4374-bc7f-69634fd00e17-kube-api-access-vrhmn\") pod \"684d2727-5c85-4374-bc7f-69634fd00e17\" (UID: \"684d2727-5c85-4374-bc7f-69634fd00e17\") " Oct 31 00:56:03.374407 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:03.374359 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/684d2727-5c85-4374-bc7f-69634fd00e17-maas-api-openshift-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:03.374407 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:03.374398 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/343c42b7-22b0-4f9d-8be7-0ffbd7ce64e1-maas-api-openshift-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:03.376166 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:03.376135 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/684d2727-5c85-4374-bc7f-69634fd00e17-maas-api-xks-ca" pod "684d2727-5c85-4374-bc7f-69634fd00e17" (UID: "684d2727-5c85-4374-bc7f-69634fd00e17"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:56:03.376166 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:03.376150 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/343c42b7-22b0-4f9d-8be7-0ffbd7ce64e1-maas-api-xks-ca" pod "343c42b7-22b0-4f9d-8be7-0ffbd7ce64e1" (UID: "343c42b7-22b0-4f9d-8be7-0ffbd7ce64e1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:56:03.376302 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:03.376224 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/343c42b7-22b0-4f9d-8be7-0ffbd7ce64e1-kube-api-access-42vtx" pod "343c42b7-22b0-4f9d-8be7-0ffbd7ce64e1" (UID: "343c42b7-22b0-4f9d-8be7-0ffbd7ce64e1"). InnerVolumeSpecName "kube-api-access-42vtx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:56:03.376703 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:03.376681 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/684d2727-5c85-4374-bc7f-69634fd00e17-kube-api-access-vrhmn" pod "684d2727-5c85-4374-bc7f-69634fd00e17" (UID: "684d2727-5c85-4374-bc7f-69634fd00e17"). InnerVolumeSpecName "kube-api-access-vrhmn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:56:03.475385 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:03.475345 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/343c42b7-22b0-4f9d-8be7-0ffbd7ce64e1-maas-api-xks-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:03.475575 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:03.475431 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrhmn\" (UniqueName: \"kubernetes.io/projected/684d2727-5c85-4374-bc7f-69634fd00e17-kube-api-access-vrhmn\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:03.475575 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:03.475445 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42vtx\" (UniqueName: \"kubernetes.io/projected/343c42b7-22b0-4f9d-8be7-0ffbd7ce64e1-kube-api-access-42vtx\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:03.475575 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:03.475454 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/684d2727-5c85-4374-bc7f-69634fd00e17-maas-api-xks-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:04.111559 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:04.111518 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ef266b8-vkmzd" Oct 31 00:56:04.111559 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:04.111555 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ef266b8-vkmzd" event={"ID":"684d2727-5c85-4374-bc7f-69634fd00e17","Type":"ContainerDied","Data":"9a6891fa616d6424152b4d1d0eda96e83044c4029e94213cfcb262ec80953630"} Oct 31 00:56:04.111805 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:04.111602 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9a6891fa616d6424152b4d1d0eda96e83044c4029e94213cfcb262ec80953630" Oct 31 00:56:04.113103 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:04.113084 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-162fe38d-xcsh4" Oct 31 00:56:04.113283 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:04.113106 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-162fe38d-xcsh4" event={"ID":"343c42b7-22b0-4f9d-8be7-0ffbd7ce64e1","Type":"ContainerDied","Data":"0373ec7340604008efd6f60f23a15b9cb3cf68e24c92d67d74ca1ea80591107c"} Oct 31 00:56:04.113283 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:04.113148 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0373ec7340604008efd6f60f23a15b9cb3cf68e24c92d67d74ca1ea80591107c" Oct 31 00:56:05.891210 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:56:05.891173 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 00:56:05.998516 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:05.998473 5222 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-101b9e-688fd6f5d7-7nrcm" podUID="38bd486f-e3b6-4d6e-8ca2-6bb7591b9ba3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:56:09.861621 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:09.861588 5222 scope.go:122] "RemoveContainer" containerID="066c39a7d74a193b214d15154a3d2a8f1abf5c269245146d331a9cf30c43c50b" Oct 31 00:56:09.871905 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:09.871882 5222 scope.go:122] "RemoveContainer" containerID="11a7dff0b5e0f347d74ad8a8ef7e72fb5229cc58de19ac5416cc2ad439bf177f" Oct 31 00:56:09.881727 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:09.881699 5222 scope.go:122] "RemoveContainer" containerID="2338a6eb69d1b44faecf36f9529877bf6e608849ea8187d55d0efae85bedd3b8" Oct 31 00:56:09.891570 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:09.891550 5222 scope.go:122] "RemoveContainer" containerID="126086a618ae38e81bfd9a69d5f2908a55876013eaa7adff275f8a656cdfe091" Oct 31 00:56:10.627230 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:10.627191 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a0c57b4-d46f8"] Oct 31 00:56:10.627751 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:10.627737 5222 cpu_manager.go:414] "containerMap: removing container" podUID="684d2727-5c85-4374-bc7f-69634fd00e17" containerName="revoke-keys" Oct 31 00:56:10.627810 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:10.627752 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="684d2727-5c85-4374-bc7f-69634fd00e17" containerName="revoke-keys" Oct 31 00:56:10.627810 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:10.627787 5222 cpu_manager.go:414] "containerMap: removing container" podUID="343c42b7-22b0-4f9d-8be7-0ffbd7ce64e1" containerName="revoke-keys" Oct 31 00:56:10.627810 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:10.627792 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="343c42b7-22b0-4f9d-8be7-0ffbd7ce64e1" containerName="revoke-keys" Oct 31 00:56:10.627920 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:10.627864 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="343c42b7-22b0-4f9d-8be7-0ffbd7ce64e1" containerName="revoke-keys" Oct 31 00:56:10.627920 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:10.627876 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="684d2727-5c85-4374-bc7f-69634fd00e17" containerName="revoke-keys" Oct 31 00:56:10.633189 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:10.633166 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a0c57b4-d46f8" Oct 31 00:56:10.661209 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:10.661175 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03773639-nsd5f"] Oct 31 00:56:10.665144 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:10.665123 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03773639-nsd5f" Oct 31 00:56:10.686595 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:10.686563 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a0c57b4-d46f8"] Oct 31 00:56:10.709926 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:10.709894 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:56:10.710130 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:10.709944 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:56:10.710130 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:10.709953 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfz9z\"" Oct 31 00:56:10.710130 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:10.709972 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:56:10.746693 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:10.746658 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03773639-nsd5f"] Oct 31 00:56:10.759642 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:10.759612 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca49d252-2308-40a6-a13b-96500d6c1296-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-03773639-nsd5f\" (UID: \"ca49d252-2308-40a6-a13b-96500d6c1296\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03773639-nsd5f" Oct 31 00:56:10.759812 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:10.759659 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxj9f\" (UniqueName: \"kubernetes.io/projected/2a1e9ba8-7189-4e75-a33e-028af75e3f87-kube-api-access-lxj9f\") pod \"maas-api-revoke-sub-9a0c57b4-d46f8\" (UID: \"2a1e9ba8-7189-4e75-a33e-028af75e3f87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a0c57b4-d46f8" Oct 31 00:56:10.759812 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:10.759763 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vq4x7\" (UniqueName: \"kubernetes.io/projected/ca49d252-2308-40a6-a13b-96500d6c1296-kube-api-access-vq4x7\") pod \"maas-api-revoke-sub-03773639-nsd5f\" (UID: \"ca49d252-2308-40a6-a13b-96500d6c1296\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03773639-nsd5f" Oct 31 00:56:10.759925 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:10.759871 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a1e9ba8-7189-4e75-a33e-028af75e3f87-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9a0c57b4-d46f8\" (UID: \"2a1e9ba8-7189-4e75-a33e-028af75e3f87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a0c57b4-d46f8" Oct 31 00:56:10.759966 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:10.759944 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a1e9ba8-7189-4e75-a33e-028af75e3f87-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9a0c57b4-d46f8\" (UID: \"2a1e9ba8-7189-4e75-a33e-028af75e3f87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a0c57b4-d46f8" Oct 31 00:56:10.760041 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:10.760023 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca49d252-2308-40a6-a13b-96500d6c1296-maas-api-xks-ca\") pod \"maas-api-revoke-sub-03773639-nsd5f\" (UID: \"ca49d252-2308-40a6-a13b-96500d6c1296\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03773639-nsd5f" Oct 31 00:56:10.861515 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:10.861471 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca49d252-2308-40a6-a13b-96500d6c1296-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-03773639-nsd5f\" (UID: \"ca49d252-2308-40a6-a13b-96500d6c1296\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03773639-nsd5f" Oct 31 00:56:10.861722 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:10.861539 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxj9f\" (UniqueName: \"kubernetes.io/projected/2a1e9ba8-7189-4e75-a33e-028af75e3f87-kube-api-access-lxj9f\") pod \"maas-api-revoke-sub-9a0c57b4-d46f8\" (UID: \"2a1e9ba8-7189-4e75-a33e-028af75e3f87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a0c57b4-d46f8" Oct 31 00:56:10.861722 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:10.861585 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq4x7\" (UniqueName: \"kubernetes.io/projected/ca49d252-2308-40a6-a13b-96500d6c1296-kube-api-access-vq4x7\") pod \"maas-api-revoke-sub-03773639-nsd5f\" (UID: \"ca49d252-2308-40a6-a13b-96500d6c1296\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03773639-nsd5f" Oct 31 00:56:10.861722 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:10.861643 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a1e9ba8-7189-4e75-a33e-028af75e3f87-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9a0c57b4-d46f8\" (UID: \"2a1e9ba8-7189-4e75-a33e-028af75e3f87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a0c57b4-d46f8" Oct 31 00:56:10.862113 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:10.861736 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a1e9ba8-7189-4e75-a33e-028af75e3f87-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9a0c57b4-d46f8\" (UID: \"2a1e9ba8-7189-4e75-a33e-028af75e3f87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a0c57b4-d46f8" Oct 31 00:56:10.862113 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:10.861798 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca49d252-2308-40a6-a13b-96500d6c1296-maas-api-xks-ca\") pod \"maas-api-revoke-sub-03773639-nsd5f\" (UID: \"ca49d252-2308-40a6-a13b-96500d6c1296\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03773639-nsd5f" Oct 31 00:56:10.862296 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:10.862273 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca49d252-2308-40a6-a13b-96500d6c1296-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-03773639-nsd5f\" (UID: \"ca49d252-2308-40a6-a13b-96500d6c1296\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03773639-nsd5f" Oct 31 00:56:10.862425 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:10.862400 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a1e9ba8-7189-4e75-a33e-028af75e3f87-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9a0c57b4-d46f8\" (UID: \"2a1e9ba8-7189-4e75-a33e-028af75e3f87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a0c57b4-d46f8" Oct 31 00:56:10.864236 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:10.864214 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a1e9ba8-7189-4e75-a33e-028af75e3f87-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9a0c57b4-d46f8\" (UID: \"2a1e9ba8-7189-4e75-a33e-028af75e3f87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a0c57b4-d46f8" Oct 31 00:56:10.864337 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:10.864326 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca49d252-2308-40a6-a13b-96500d6c1296-maas-api-xks-ca\") pod \"maas-api-revoke-sub-03773639-nsd5f\" (UID: \"ca49d252-2308-40a6-a13b-96500d6c1296\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03773639-nsd5f" Oct 31 00:56:10.924208 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:10.924137 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxj9f\" (UniqueName: \"kubernetes.io/projected/2a1e9ba8-7189-4e75-a33e-028af75e3f87-kube-api-access-lxj9f\") pod \"maas-api-revoke-sub-9a0c57b4-d46f8\" (UID: \"2a1e9ba8-7189-4e75-a33e-028af75e3f87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a0c57b4-d46f8" Oct 31 00:56:10.924364 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:10.924340 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq4x7\" (UniqueName: \"kubernetes.io/projected/ca49d252-2308-40a6-a13b-96500d6c1296-kube-api-access-vq4x7\") pod \"maas-api-revoke-sub-03773639-nsd5f\" (UID: \"ca49d252-2308-40a6-a13b-96500d6c1296\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03773639-nsd5f" Oct 31 00:56:10.947110 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:10.947078 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a0c57b4-d46f8" Oct 31 00:56:10.981313 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:10.981276 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03773639-nsd5f" Oct 31 00:56:11.321936 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:56:11.321883 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a1e9ba8_7189_4e75_a33e_028af75e3f87.slice/crio-b9c726b947c7261e2276e79cddc2435bbee6597c28f57397b2fcfc90b4957825 WatchSource:0}: Error finding container b9c726b947c7261e2276e79cddc2435bbee6597c28f57397b2fcfc90b4957825: Status 404 returned error can't find the container with id b9c726b947c7261e2276e79cddc2435bbee6597c28f57397b2fcfc90b4957825 Oct 31 00:56:11.325794 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:11.325758 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a0c57b4-d46f8"] Oct 31 00:56:11.371344 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:56:11.371304 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca49d252_2308_40a6_a13b_96500d6c1296.slice/crio-990a263aa276c35112dac9c9278df18d539766871738d464d6fc890e05b64b23 WatchSource:0}: Error finding container 990a263aa276c35112dac9c9278df18d539766871738d464d6fc890e05b64b23: Status 404 returned error can't find the container with id 990a263aa276c35112dac9c9278df18d539766871738d464d6fc890e05b64b23 Oct 31 00:56:11.375025 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:11.374984 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03773639-nsd5f"] Oct 31 00:56:12.168763 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:12.168729 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a0c57b4-d46f8" event={"ID":"2a1e9ba8-7189-4e75-a33e-028af75e3f87","Type":"ContainerStarted","Data":"5661ec1588b0c4cf6c3eedd761e3785334ad0921cc7db38a538235469d1d2432"} Oct 31 00:56:12.169245 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:12.168815 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a0c57b4-d46f8" event={"ID":"2a1e9ba8-7189-4e75-a33e-028af75e3f87","Type":"ContainerStarted","Data":"b9c726b947c7261e2276e79cddc2435bbee6597c28f57397b2fcfc90b4957825"} Oct 31 00:56:12.170486 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:12.170453 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03773639-nsd5f" event={"ID":"ca49d252-2308-40a6-a13b-96500d6c1296","Type":"ContainerStarted","Data":"a074f5e3d48b758450af11203cb545f3bdd3f57eeb91ac391c15227b4b9e5a8b"} Oct 31 00:56:12.170619 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:12.170502 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03773639-nsd5f" event={"ID":"ca49d252-2308-40a6-a13b-96500d6c1296","Type":"ContainerStarted","Data":"990a263aa276c35112dac9c9278df18d539766871738d464d6fc890e05b64b23"} Oct 31 00:56:12.288520 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:12.288463 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a0c57b4-d46f8" podStartSLOduration=2.288449642 podStartE2EDuration="2.288449642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:56:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:56:12.286655574 +0000 UTC m=+2464.757825323" watchObservedRunningTime="2026-10-31 00:56:12.288449642 +0000 UTC m=+2464.759619390" Oct 31 00:56:12.407942 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:12.407881 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03773639-nsd5f" podStartSLOduration=2.40786508 podStartE2EDuration="2.40786508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:56:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:56:12.404539613 +0000 UTC m=+2464.875709364" watchObservedRunningTime="2026-10-31 00:56:12.40786508 +0000 UTC m=+2464.879034829" Oct 31 00:56:13.178664 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:13.178626 5222 generic.go:356] "Generic (PLEG): container finished" podID="2a1e9ba8-7189-4e75-a33e-028af75e3f87" containerID="5661ec1588b0c4cf6c3eedd761e3785334ad0921cc7db38a538235469d1d2432" exitCode=0 Oct 31 00:56:13.179087 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:13.178710 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a0c57b4-d46f8" event={"ID":"2a1e9ba8-7189-4e75-a33e-028af75e3f87","Type":"ContainerDied","Data":"5661ec1588b0c4cf6c3eedd761e3785334ad0921cc7db38a538235469d1d2432"} Oct 31 00:56:13.180455 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:13.180351 5222 generic.go:356] "Generic (PLEG): container finished" podID="ca49d252-2308-40a6-a13b-96500d6c1296" containerID="a074f5e3d48b758450af11203cb545f3bdd3f57eeb91ac391c15227b4b9e5a8b" exitCode=0 Oct 31 00:56:13.180593 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:13.180401 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03773639-nsd5f" event={"ID":"ca49d252-2308-40a6-a13b-96500d6c1296","Type":"ContainerDied","Data":"a074f5e3d48b758450af11203cb545f3bdd3f57eeb91ac391c15227b4b9e5a8b"} Oct 31 00:56:14.349124 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:14.349094 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03773639-nsd5f" Oct 31 00:56:14.375146 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:14.375117 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a0c57b4-d46f8" Oct 31 00:56:14.504406 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:14.504274 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a1e9ba8-7189-4e75-a33e-028af75e3f87-kube-api-access-lxj9f\" (UniqueName: \"kubernetes.io/projected/2a1e9ba8-7189-4e75-a33e-028af75e3f87-kube-api-access-lxj9f\") pod \"2a1e9ba8-7189-4e75-a33e-028af75e3f87\" (UID: \"2a1e9ba8-7189-4e75-a33e-028af75e3f87\") " Oct 31 00:56:14.504406 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:14.504357 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca49d252-2308-40a6-a13b-96500d6c1296-kube-api-access-vq4x7\" (UniqueName: \"kubernetes.io/projected/ca49d252-2308-40a6-a13b-96500d6c1296-kube-api-access-vq4x7\") pod \"ca49d252-2308-40a6-a13b-96500d6c1296\" (UID: \"ca49d252-2308-40a6-a13b-96500d6c1296\") " Oct 31 00:56:14.504611 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:14.504563 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a1e9ba8-7189-4e75-a33e-028af75e3f87-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a1e9ba8-7189-4e75-a33e-028af75e3f87-maas-api-xks-ca\") pod \"2a1e9ba8-7189-4e75-a33e-028af75e3f87\" (UID: \"2a1e9ba8-7189-4e75-a33e-028af75e3f87\") " Oct 31 00:56:14.504659 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:14.504607 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a1e9ba8-7189-4e75-a33e-028af75e3f87-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a1e9ba8-7189-4e75-a33e-028af75e3f87-maas-api-openshift-ca\") pod \"2a1e9ba8-7189-4e75-a33e-028af75e3f87\" (UID: \"2a1e9ba8-7189-4e75-a33e-028af75e3f87\") " Oct 31 00:56:14.504659 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:14.504640 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca49d252-2308-40a6-a13b-96500d6c1296-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca49d252-2308-40a6-a13b-96500d6c1296-maas-api-xks-ca\") pod \"ca49d252-2308-40a6-a13b-96500d6c1296\" (UID: \"ca49d252-2308-40a6-a13b-96500d6c1296\") " Oct 31 00:56:14.504753 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:14.504701 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca49d252-2308-40a6-a13b-96500d6c1296-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca49d252-2308-40a6-a13b-96500d6c1296-maas-api-openshift-ca\") pod \"ca49d252-2308-40a6-a13b-96500d6c1296\" (UID: \"ca49d252-2308-40a6-a13b-96500d6c1296\") " Oct 31 00:56:14.505033 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:14.505001 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a1e9ba8-7189-4e75-a33e-028af75e3f87-maas-api-openshift-ca" pod "2a1e9ba8-7189-4e75-a33e-028af75e3f87" (UID: "2a1e9ba8-7189-4e75-a33e-028af75e3f87"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:56:14.505165 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:14.505077 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca49d252-2308-40a6-a13b-96500d6c1296-maas-api-openshift-ca" pod "ca49d252-2308-40a6-a13b-96500d6c1296" (UID: "ca49d252-2308-40a6-a13b-96500d6c1296"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:56:14.505445 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:14.505421 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a1e9ba8-7189-4e75-a33e-028af75e3f87-maas-api-openshift-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:14.505511 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:14.505450 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca49d252-2308-40a6-a13b-96500d6c1296-maas-api-openshift-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:14.506914 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:14.506886 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a1e9ba8-7189-4e75-a33e-028af75e3f87-kube-api-access-lxj9f" pod "2a1e9ba8-7189-4e75-a33e-028af75e3f87" (UID: "2a1e9ba8-7189-4e75-a33e-028af75e3f87"). InnerVolumeSpecName "kube-api-access-lxj9f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:56:14.507034 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:14.506920 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca49d252-2308-40a6-a13b-96500d6c1296-kube-api-access-vq4x7" pod "ca49d252-2308-40a6-a13b-96500d6c1296" (UID: "ca49d252-2308-40a6-a13b-96500d6c1296"). InnerVolumeSpecName "kube-api-access-vq4x7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:56:14.507316 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:14.507286 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca49d252-2308-40a6-a13b-96500d6c1296-maas-api-xks-ca" pod "ca49d252-2308-40a6-a13b-96500d6c1296" (UID: "ca49d252-2308-40a6-a13b-96500d6c1296"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:56:14.507448 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:14.507287 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a1e9ba8-7189-4e75-a33e-028af75e3f87-maas-api-xks-ca" pod "2a1e9ba8-7189-4e75-a33e-028af75e3f87" (UID: "2a1e9ba8-7189-4e75-a33e-028af75e3f87"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:56:14.606956 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:14.606921 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a1e9ba8-7189-4e75-a33e-028af75e3f87-maas-api-xks-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:14.606956 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:14.606954 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca49d252-2308-40a6-a13b-96500d6c1296-maas-api-xks-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:14.607181 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:14.606988 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxj9f\" (UniqueName: \"kubernetes.io/projected/2a1e9ba8-7189-4e75-a33e-028af75e3f87-kube-api-access-lxj9f\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:14.607181 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:14.607003 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vq4x7\" (UniqueName: \"kubernetes.io/projected/ca49d252-2308-40a6-a13b-96500d6c1296-kube-api-access-vq4x7\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:15.194646 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:15.194601 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a0c57b4-d46f8" Oct 31 00:56:15.194845 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:15.194605 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a0c57b4-d46f8" event={"ID":"2a1e9ba8-7189-4e75-a33e-028af75e3f87","Type":"ContainerDied","Data":"b9c726b947c7261e2276e79cddc2435bbee6597c28f57397b2fcfc90b4957825"} Oct 31 00:56:15.194845 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:15.194735 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b9c726b947c7261e2276e79cddc2435bbee6597c28f57397b2fcfc90b4957825" Oct 31 00:56:15.196059 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:15.196033 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03773639-nsd5f" Oct 31 00:56:15.196162 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:15.196077 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03773639-nsd5f" event={"ID":"ca49d252-2308-40a6-a13b-96500d6c1296","Type":"ContainerDied","Data":"990a263aa276c35112dac9c9278df18d539766871738d464d6fc890e05b64b23"} Oct 31 00:56:15.196162 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:15.196117 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="990a263aa276c35112dac9c9278df18d539766871738d464d6fc890e05b64b23" Oct 31 00:56:16.029101 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:16.029022 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-101b9e-688fd6f5d7-7nrcm" Oct 31 00:56:16.123131 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:16.123089 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38bd486f-e3b6-4d6e-8ca2-6bb7591b9ba3-kube-api-access-m5wks\" (UniqueName: \"kubernetes.io/projected/38bd486f-e3b6-4d6e-8ca2-6bb7591b9ba3-kube-api-access-m5wks\") pod \"38bd486f-e3b6-4d6e-8ca2-6bb7591b9ba3\" (UID: \"38bd486f-e3b6-4d6e-8ca2-6bb7591b9ba3\") " Oct 31 00:56:16.123345 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:16.123195 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38bd486f-e3b6-4d6e-8ca2-6bb7591b9ba3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38bd486f-e3b6-4d6e-8ca2-6bb7591b9ba3-plugins-config-volume\") pod \"38bd486f-e3b6-4d6e-8ca2-6bb7591b9ba3\" (UID: \"38bd486f-e3b6-4d6e-8ca2-6bb7591b9ba3\") " Oct 31 00:56:16.123637 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:16.123610 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38bd486f-e3b6-4d6e-8ca2-6bb7591b9ba3-plugins-config-volume" pod "38bd486f-e3b6-4d6e-8ca2-6bb7591b9ba3" (UID: "38bd486f-e3b6-4d6e-8ca2-6bb7591b9ba3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:56:16.125631 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:16.125613 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38bd486f-e3b6-4d6e-8ca2-6bb7591b9ba3-kube-api-access-m5wks" pod "38bd486f-e3b6-4d6e-8ca2-6bb7591b9ba3" (UID: "38bd486f-e3b6-4d6e-8ca2-6bb7591b9ba3"). InnerVolumeSpecName "kube-api-access-m5wks". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:56:16.203353 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:16.203320 5222 generic.go:356] "Generic (PLEG): container finished" podID="38bd486f-e3b6-4d6e-8ca2-6bb7591b9ba3" containerID="52aae2a664052461da0e54e45e312a372bc73cf5d96aeeb28fb5a40434b06f65" exitCode=0 Oct 31 00:56:16.203545 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:16.203388 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-101b9e-688fd6f5d7-7nrcm" Oct 31 00:56:16.203545 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:16.203407 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-101b9e-688fd6f5d7-7nrcm" event={"ID":"38bd486f-e3b6-4d6e-8ca2-6bb7591b9ba3","Type":"ContainerDied","Data":"52aae2a664052461da0e54e45e312a372bc73cf5d96aeeb28fb5a40434b06f65"} Oct 31 00:56:16.203545 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:16.203502 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-101b9e-688fd6f5d7-7nrcm" event={"ID":"38bd486f-e3b6-4d6e-8ca2-6bb7591b9ba3","Type":"ContainerDied","Data":"878e7588f72a91ab37d334938f568904f73a9549c4abd20e93e642b550a52017"} Oct 31 00:56:16.203545 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:16.203531 5222 scope.go:122] "RemoveContainer" containerID="52aae2a664052461da0e54e45e312a372bc73cf5d96aeeb28fb5a40434b06f65" Oct 31 00:56:16.213720 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:16.213693 5222 scope.go:122] "RemoveContainer" containerID="52aae2a664052461da0e54e45e312a372bc73cf5d96aeeb28fb5a40434b06f65" Oct 31 00:56:16.213990 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:56:16.213972 5222 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52aae2a664052461da0e54e45e312a372bc73cf5d96aeeb28fb5a40434b06f65\": container with ID starting with 52aae2a664052461da0e54e45e312a372bc73cf5d96aeeb28fb5a40434b06f65 not found: ID does not exist" containerID="52aae2a664052461da0e54e45e312a372bc73cf5d96aeeb28fb5a40434b06f65" Oct 31 00:56:16.214043 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:16.214000 5222 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52aae2a664052461da0e54e45e312a372bc73cf5d96aeeb28fb5a40434b06f65"} err="failed to get container status \"52aae2a664052461da0e54e45e312a372bc73cf5d96aeeb28fb5a40434b06f65\": rpc error: code = NotFound desc = could not find container \"52aae2a664052461da0e54e45e312a372bc73cf5d96aeeb28fb5a40434b06f65\": container with ID starting with 52aae2a664052461da0e54e45e312a372bc73cf5d96aeeb28fb5a40434b06f65 not found: ID does not exist" Oct 31 00:56:16.224731 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:16.224697 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5wks\" (UniqueName: \"kubernetes.io/projected/38bd486f-e3b6-4d6e-8ca2-6bb7591b9ba3-kube-api-access-m5wks\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:16.224731 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:16.224726 5222 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38bd486f-e3b6-4d6e-8ca2-6bb7591b9ba3-plugins-config-volume\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:16.374598 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:16.374516 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-101b9e-688fd6f5d7-7nrcm"] Oct 31 00:56:16.380248 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:16.380221 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-101b9e-688fd6f5d7-7nrcm"] Oct 31 00:56:16.998254 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:16.998209 5222 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-101b9e-688fd6f5d7-7nrcm" podUID="38bd486f-e3b6-4d6e-8ca2-6bb7591b9ba3" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.44:50052\" within 1s: context deadline exceeded" Oct 31 00:56:16.998456 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:56:16.998292 5222 logging.go:55] [core] [Channel #7796 SubChannel #7797]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.44:50052", ServerName: "10.134.0.44:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.44:50052: operation was canceled" Oct 31 00:56:17.655264 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:17.655227 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38bd486f-e3b6-4d6e-8ca2-6bb7591b9ba3" path="/var/lib/kubelet/pods/38bd486f-e3b6-4d6e-8ca2-6bb7591b9ba3/volumes" Oct 31 00:56:18.874646 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:18.874608 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-79tk8"] Oct 31 00:56:25.050981 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:25.050941 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73e94f2f-vnx94"] Oct 31 00:56:25.101796 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:25.101754 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73e94f2f-vnx94"] Oct 31 00:56:25.104348 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:25.104335 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72cb1008-qp222"] Oct 31 00:56:25.106747 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:25.106724 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72cb1008-qp222"] Oct 31 00:56:25.657862 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:25.657824 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ddbb670-bb0c-45b6-ac6f-be091e71e08f" path="/var/lib/kubelet/pods/8ddbb670-bb0c-45b6-ac6f-be091e71e08f/volumes" Oct 31 00:56:25.658858 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:25.658834 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d113faa4-aa17-4a24-bf7c-bfe3e01d9ee0" path="/var/lib/kubelet/pods/d113faa4-aa17-4a24-bf7c-bfe3e01d9ee0/volumes" Oct 31 00:56:35.890537 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:56:35.890500 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 00:56:36.813808 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:36.813767 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-rf2wq"] Oct 31 00:56:36.814301 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:36.814284 5222 cpu_manager.go:414] "containerMap: removing container" podUID="2a1e9ba8-7189-4e75-a33e-028af75e3f87" containerName="revoke-keys" Oct 31 00:56:36.814356 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:36.814302 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a1e9ba8-7189-4e75-a33e-028af75e3f87" containerName="revoke-keys" Oct 31 00:56:36.814356 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:36.814337 5222 cpu_manager.go:414] "containerMap: removing container" podUID="ca49d252-2308-40a6-a13b-96500d6c1296" containerName="revoke-keys" Oct 31 00:56:36.814356 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:36.814343 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca49d252-2308-40a6-a13b-96500d6c1296" containerName="revoke-keys" Oct 31 00:56:36.814356 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:36.814351 5222 cpu_manager.go:414] "containerMap: removing container" podUID="38bd486f-e3b6-4d6e-8ca2-6bb7591b9ba3" containerName="payload-pre-processing" Oct 31 00:56:36.814356 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:36.814356 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38bd486f-e3b6-4d6e-8ca2-6bb7591b9ba3" containerName="payload-pre-processing" Oct 31 00:56:36.814545 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:36.814480 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca49d252-2308-40a6-a13b-96500d6c1296" containerName="revoke-keys" Oct 31 00:56:36.814545 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:36.814491 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="38bd486f-e3b6-4d6e-8ca2-6bb7591b9ba3" containerName="payload-pre-processing" Oct 31 00:56:36.814545 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:36.814502 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a1e9ba8-7189-4e75-a33e-028af75e3f87" containerName="revoke-keys" Oct 31 00:56:36.818014 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:36.817980 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-rf2wq" Oct 31 00:56:36.857623 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:36.857582 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c2a7c21-8b15-47fc-aa57-25b19399fd60-istio-podinfo\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-rf2wq\" (UID: \"6c2a7c21-8b15-47fc-aa57-25b19399fd60\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-rf2wq" Oct 31 00:56:36.857623 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:36.857619 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c2a7c21-8b15-47fc-aa57-25b19399fd60-credential-socket\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-rf2wq\" (UID: \"6c2a7c21-8b15-47fc-aa57-25b19399fd60\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-rf2wq" Oct 31 00:56:36.857859 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:36.857766 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c2a7c21-8b15-47fc-aa57-25b19399fd60-workload-certs\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-rf2wq\" (UID: \"6c2a7c21-8b15-47fc-aa57-25b19399fd60\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-rf2wq" Oct 31 00:56:36.857859 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:36.857801 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c2a7c21-8b15-47fc-aa57-25b19399fd60-istio-envoy\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-rf2wq\" (UID: \"6c2a7c21-8b15-47fc-aa57-25b19399fd60\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-rf2wq" Oct 31 00:56:36.857859 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:36.857844 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c2a7c21-8b15-47fc-aa57-25b19399fd60-istio-token\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-rf2wq\" (UID: \"6c2a7c21-8b15-47fc-aa57-25b19399fd60\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-rf2wq" Oct 31 00:56:36.857988 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:36.857940 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c2a7c21-8b15-47fc-aa57-25b19399fd60-workload-socket\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-rf2wq\" (UID: \"6c2a7c21-8b15-47fc-aa57-25b19399fd60\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-rf2wq" Oct 31 00:56:36.858031 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:36.858005 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c2a7c21-8b15-47fc-aa57-25b19399fd60-istiod-ca-cert\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-rf2wq\" (UID: \"6c2a7c21-8b15-47fc-aa57-25b19399fd60\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-rf2wq" Oct 31 00:56:36.858069 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:36.858031 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lf7wf\" (UniqueName: \"kubernetes.io/projected/6c2a7c21-8b15-47fc-aa57-25b19399fd60-kube-api-access-lf7wf\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-rf2wq\" (UID: \"6c2a7c21-8b15-47fc-aa57-25b19399fd60\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-rf2wq" Oct 31 00:56:36.858069 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:36.858051 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c2a7c21-8b15-47fc-aa57-25b19399fd60-istio-data\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-rf2wq\" (UID: \"6c2a7c21-8b15-47fc-aa57-25b19399fd60\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-rf2wq" Oct 31 00:56:36.865082 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:36.865050 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-xnr5x"] Oct 31 00:56:36.869167 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:36.869144 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-xnr5x" Oct 31 00:56:36.875417 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:36.875385 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-rf2wq"] Oct 31 00:56:36.913742 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:36.913697 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-101b9e-openshift-default-dockercfg-8l66m\"" Oct 31 00:56:36.927107 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:36.927075 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-xnr5x"] Oct 31 00:56:36.959402 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:36.959329 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dbc245a-0bc5-4a0e-83fb-a49841984b74-workload-socket\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-xnr5x\" (UID: \"5dbc245a-0bc5-4a0e-83fb-a49841984b74\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-xnr5x" Oct 31 00:56:36.959623 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:36.959410 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c2a7c21-8b15-47fc-aa57-25b19399fd60-istio-token\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-rf2wq\" (UID: \"6c2a7c21-8b15-47fc-aa57-25b19399fd60\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-rf2wq" Oct 31 00:56:36.959623 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:36.959505 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dbc245a-0bc5-4a0e-83fb-a49841984b74-credential-socket\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-xnr5x\" (UID: \"5dbc245a-0bc5-4a0e-83fb-a49841984b74\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-xnr5x" Oct 31 00:56:36.959623 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:36.959545 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dbc245a-0bc5-4a0e-83fb-a49841984b74-workload-certs\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-xnr5x\" (UID: \"5dbc245a-0bc5-4a0e-83fb-a49841984b74\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-xnr5x" Oct 31 00:56:36.959796 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:36.959661 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dbc245a-0bc5-4a0e-83fb-a49841984b74-istio-podinfo\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-xnr5x\" (UID: \"5dbc245a-0bc5-4a0e-83fb-a49841984b74\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-xnr5x" Oct 31 00:56:36.959796 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:36.959689 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dbc245a-0bc5-4a0e-83fb-a49841984b74-istio-data\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-xnr5x\" (UID: \"5dbc245a-0bc5-4a0e-83fb-a49841984b74\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-xnr5x" Oct 31 00:56:36.959796 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:36.959759 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c2a7c21-8b15-47fc-aa57-25b19399fd60-workload-socket\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-rf2wq\" (UID: \"6c2a7c21-8b15-47fc-aa57-25b19399fd60\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-rf2wq" Oct 31 00:56:36.959961 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:36.959814 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dbc245a-0bc5-4a0e-83fb-a49841984b74-istiod-ca-cert\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-xnr5x\" (UID: \"5dbc245a-0bc5-4a0e-83fb-a49841984b74\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-xnr5x" Oct 31 00:56:36.959961 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:36.959912 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dbc245a-0bc5-4a0e-83fb-a49841984b74-istio-envoy\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-xnr5x\" (UID: \"5dbc245a-0bc5-4a0e-83fb-a49841984b74\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-xnr5x" Oct 31 00:56:36.959961 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:36.959945 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c2a7c21-8b15-47fc-aa57-25b19399fd60-istiod-ca-cert\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-rf2wq\" (UID: \"6c2a7c21-8b15-47fc-aa57-25b19399fd60\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-rf2wq" Oct 31 00:56:36.960124 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:36.959970 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lf7wf\" (UniqueName: \"kubernetes.io/projected/6c2a7c21-8b15-47fc-aa57-25b19399fd60-kube-api-access-lf7wf\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-rf2wq\" (UID: \"6c2a7c21-8b15-47fc-aa57-25b19399fd60\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-rf2wq" Oct 31 00:56:36.960124 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:36.960000 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c2a7c21-8b15-47fc-aa57-25b19399fd60-istio-data\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-rf2wq\" (UID: \"6c2a7c21-8b15-47fc-aa57-25b19399fd60\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-rf2wq" Oct 31 00:56:36.960124 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:36.960026 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c2a7c21-8b15-47fc-aa57-25b19399fd60-istio-podinfo\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-rf2wq\" (UID: \"6c2a7c21-8b15-47fc-aa57-25b19399fd60\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-rf2wq" Oct 31 00:56:36.960124 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:36.960052 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c2a7c21-8b15-47fc-aa57-25b19399fd60-credential-socket\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-rf2wq\" (UID: \"6c2a7c21-8b15-47fc-aa57-25b19399fd60\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-rf2wq" Oct 31 00:56:36.960124 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:36.960114 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c2a7c21-8b15-47fc-aa57-25b19399fd60-workload-socket\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-rf2wq\" (UID: \"6c2a7c21-8b15-47fc-aa57-25b19399fd60\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-rf2wq" Oct 31 00:56:36.960408 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:36.960128 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5dbc245a-0bc5-4a0e-83fb-a49841984b74-istio-token\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-xnr5x\" (UID: \"5dbc245a-0bc5-4a0e-83fb-a49841984b74\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-xnr5x" Oct 31 00:56:36.960408 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:36.960180 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c2a7c21-8b15-47fc-aa57-25b19399fd60-workload-certs\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-rf2wq\" (UID: \"6c2a7c21-8b15-47fc-aa57-25b19399fd60\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-rf2wq" Oct 31 00:56:36.960408 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:36.960222 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c2a7c21-8b15-47fc-aa57-25b19399fd60-istio-envoy\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-rf2wq\" (UID: \"6c2a7c21-8b15-47fc-aa57-25b19399fd60\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-rf2wq" Oct 31 00:56:36.960408 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:36.960335 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c2a7c21-8b15-47fc-aa57-25b19399fd60-istio-data\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-rf2wq\" (UID: \"6c2a7c21-8b15-47fc-aa57-25b19399fd60\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-rf2wq" Oct 31 00:56:36.960408 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:36.960352 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c2a7c21-8b15-47fc-aa57-25b19399fd60-credential-socket\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-rf2wq\" (UID: \"6c2a7c21-8b15-47fc-aa57-25b19399fd60\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-rf2wq" Oct 31 00:56:36.960695 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:36.960424 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zg4g\" (UniqueName: \"kubernetes.io/projected/5dbc245a-0bc5-4a0e-83fb-a49841984b74-kube-api-access-9zg4g\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-xnr5x\" (UID: \"5dbc245a-0bc5-4a0e-83fb-a49841984b74\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-xnr5x" Oct 31 00:56:36.960695 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:36.960658 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c2a7c21-8b15-47fc-aa57-25b19399fd60-workload-certs\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-rf2wq\" (UID: \"6c2a7c21-8b15-47fc-aa57-25b19399fd60\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-rf2wq" Oct 31 00:56:36.960695 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:36.960682 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c2a7c21-8b15-47fc-aa57-25b19399fd60-istiod-ca-cert\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-rf2wq\" (UID: \"6c2a7c21-8b15-47fc-aa57-25b19399fd60\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-rf2wq" Oct 31 00:56:36.962689 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:36.962664 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c2a7c21-8b15-47fc-aa57-25b19399fd60-istio-envoy\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-rf2wq\" (UID: \"6c2a7c21-8b15-47fc-aa57-25b19399fd60\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-rf2wq" Oct 31 00:56:36.962793 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:36.962745 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c2a7c21-8b15-47fc-aa57-25b19399fd60-istio-podinfo\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-rf2wq\" (UID: \"6c2a7c21-8b15-47fc-aa57-25b19399fd60\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-rf2wq" Oct 31 00:56:37.023565 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:37.023524 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c2a7c21-8b15-47fc-aa57-25b19399fd60-istio-token\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-rf2wq\" (UID: \"6c2a7c21-8b15-47fc-aa57-25b19399fd60\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-rf2wq" Oct 31 00:56:37.027361 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:37.027330 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lf7wf\" (UniqueName: \"kubernetes.io/projected/6c2a7c21-8b15-47fc-aa57-25b19399fd60-kube-api-access-lf7wf\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-rf2wq\" (UID: \"6c2a7c21-8b15-47fc-aa57-25b19399fd60\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-rf2wq" Oct 31 00:56:37.061217 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:37.061178 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dbc245a-0bc5-4a0e-83fb-a49841984b74-istiod-ca-cert\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-xnr5x\" (UID: \"5dbc245a-0bc5-4a0e-83fb-a49841984b74\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-xnr5x" Oct 31 00:56:37.061444 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:37.061241 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dbc245a-0bc5-4a0e-83fb-a49841984b74-istio-envoy\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-xnr5x\" (UID: \"5dbc245a-0bc5-4a0e-83fb-a49841984b74\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-xnr5x" Oct 31 00:56:37.061444 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:37.061285 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5dbc245a-0bc5-4a0e-83fb-a49841984b74-istio-token\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-xnr5x\" (UID: \"5dbc245a-0bc5-4a0e-83fb-a49841984b74\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-xnr5x" Oct 31 00:56:37.061444 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:37.061320 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zg4g\" (UniqueName: \"kubernetes.io/projected/5dbc245a-0bc5-4a0e-83fb-a49841984b74-kube-api-access-9zg4g\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-xnr5x\" (UID: \"5dbc245a-0bc5-4a0e-83fb-a49841984b74\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-xnr5x" Oct 31 00:56:37.061444 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:37.061358 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dbc245a-0bc5-4a0e-83fb-a49841984b74-workload-socket\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-xnr5x\" (UID: \"5dbc245a-0bc5-4a0e-83fb-a49841984b74\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-xnr5x" Oct 31 00:56:37.061444 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:37.061428 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dbc245a-0bc5-4a0e-83fb-a49841984b74-credential-socket\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-xnr5x\" (UID: \"5dbc245a-0bc5-4a0e-83fb-a49841984b74\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-xnr5x" Oct 31 00:56:37.061691 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:37.061452 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dbc245a-0bc5-4a0e-83fb-a49841984b74-workload-certs\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-xnr5x\" (UID: \"5dbc245a-0bc5-4a0e-83fb-a49841984b74\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-xnr5x" Oct 31 00:56:37.061691 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:37.061496 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dbc245a-0bc5-4a0e-83fb-a49841984b74-istio-podinfo\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-xnr5x\" (UID: \"5dbc245a-0bc5-4a0e-83fb-a49841984b74\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-xnr5x" Oct 31 00:56:37.061691 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:37.061520 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dbc245a-0bc5-4a0e-83fb-a49841984b74-istio-data\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-xnr5x\" (UID: \"5dbc245a-0bc5-4a0e-83fb-a49841984b74\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-xnr5x" Oct 31 00:56:37.061856 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:37.061832 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dbc245a-0bc5-4a0e-83fb-a49841984b74-workload-certs\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-xnr5x\" (UID: \"5dbc245a-0bc5-4a0e-83fb-a49841984b74\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-xnr5x" Oct 31 00:56:37.061952 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:37.061929 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dbc245a-0bc5-4a0e-83fb-a49841984b74-istio-data\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-xnr5x\" (UID: \"5dbc245a-0bc5-4a0e-83fb-a49841984b74\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-xnr5x" Oct 31 00:56:37.062014 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:37.061969 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dbc245a-0bc5-4a0e-83fb-a49841984b74-workload-socket\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-xnr5x\" (UID: \"5dbc245a-0bc5-4a0e-83fb-a49841984b74\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-xnr5x" Oct 31 00:56:37.062014 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:37.062003 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dbc245a-0bc5-4a0e-83fb-a49841984b74-credential-socket\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-xnr5x\" (UID: \"5dbc245a-0bc5-4a0e-83fb-a49841984b74\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-xnr5x" Oct 31 00:56:37.062139 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:37.062045 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dbc245a-0bc5-4a0e-83fb-a49841984b74-istiod-ca-cert\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-xnr5x\" (UID: \"5dbc245a-0bc5-4a0e-83fb-a49841984b74\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-xnr5x" Oct 31 00:56:37.064064 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:37.064005 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dbc245a-0bc5-4a0e-83fb-a49841984b74-istio-envoy\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-xnr5x\" (UID: \"5dbc245a-0bc5-4a0e-83fb-a49841984b74\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-xnr5x" Oct 31 00:56:37.064184 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:37.064102 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dbc245a-0bc5-4a0e-83fb-a49841984b74-istio-podinfo\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-xnr5x\" (UID: \"5dbc245a-0bc5-4a0e-83fb-a49841984b74\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-xnr5x" Oct 31 00:56:37.126239 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:37.126207 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5dbc245a-0bc5-4a0e-83fb-a49841984b74-istio-token\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-xnr5x\" (UID: \"5dbc245a-0bc5-4a0e-83fb-a49841984b74\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-xnr5x" Oct 31 00:56:37.126476 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:37.126359 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zg4g\" (UniqueName: \"kubernetes.io/projected/5dbc245a-0bc5-4a0e-83fb-a49841984b74-kube-api-access-9zg4g\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-xnr5x\" (UID: \"5dbc245a-0bc5-4a0e-83fb-a49841984b74\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-xnr5x" Oct 31 00:56:37.132162 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:37.132130 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-rf2wq" Oct 31 00:56:37.187130 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:37.187102 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-xnr5x" Oct 31 00:56:37.447247 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:37.447161 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-j4tb5"] Oct 31 00:56:37.447441 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:37.447401 5222 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-j4tb5" podUID="d359a4aa-8996-4615-9b33-fccc33762a6a" containerName="istio-proxy" containerID="cri-o://5d7a265874f1ee0b61e4e566f8de0fce151565afb36530586afa5e7b0790de22" gracePeriod=30 Oct 31 00:56:37.603994 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:56:37.603963 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c2a7c21_8b15_47fc_aa57_25b19399fd60.slice/crio-e8b1cc9d48e53165a20fa110741136d111182c68e497ffd04ce23b0a37c430f4 WatchSource:0}: Error finding container e8b1cc9d48e53165a20fa110741136d111182c68e497ffd04ce23b0a37c430f4: Status 404 returned error can't find the container with id e8b1cc9d48e53165a20fa110741136d111182c68e497ffd04ce23b0a37c430f4 Oct 31 00:56:37.606634 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:37.606593 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:56:37.606737 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:37.606685 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:56:37.606737 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:37.606726 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:56:37.607541 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:37.607493 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-rf2wq"] Oct 31 00:56:37.620801 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:56:37.620758 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dbc245a_0bc5_4a0e_83fb_a49841984b74.slice/crio-77b842f447c6a1dc9d4927c13056a91c49622f4615375cc06389ef3c81fd6971 WatchSource:0}: Error finding container 77b842f447c6a1dc9d4927c13056a91c49622f4615375cc06389ef3c81fd6971: Status 404 returned error can't find the container with id 77b842f447c6a1dc9d4927c13056a91c49622f4615375cc06389ef3c81fd6971 Oct 31 00:56:37.626101 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:37.626055 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:56:37.626192 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:37.626163 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:56:37.626241 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:37.626211 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:56:37.628259 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:37.628234 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-xnr5x"] Oct 31 00:56:37.856923 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:37.856884 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-101b9e-4s8tx"] Oct 31 00:56:37.865473 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:37.865439 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-101b9e-4s8tx"] Oct 31 00:56:38.381413 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:38.381345 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-xnr5x" event={"ID":"5dbc245a-0bc5-4a0e-83fb-a49841984b74","Type":"ContainerStarted","Data":"6b89c4f76e335a5cd993d59356a76304dfdf589575910daf8579a616243ffb89"} Oct 31 00:56:38.382078 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:38.382051 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-xnr5x" event={"ID":"5dbc245a-0bc5-4a0e-83fb-a49841984b74","Type":"ContainerStarted","Data":"77b842f447c6a1dc9d4927c13056a91c49622f4615375cc06389ef3c81fd6971"} Oct 31 00:56:38.385823 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:38.385790 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-rf2wq" event={"ID":"6c2a7c21-8b15-47fc-aa57-25b19399fd60","Type":"ContainerStarted","Data":"02fb429afcd7fd04a736c4259c64abdcc58ed1ebe8dcb0cae268d73ab73d5692"} Oct 31 00:56:38.386137 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:38.386097 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-rf2wq" event={"ID":"6c2a7c21-8b15-47fc-aa57-25b19399fd60","Type":"ContainerStarted","Data":"e8b1cc9d48e53165a20fa110741136d111182c68e497ffd04ce23b0a37c430f4"} Oct 31 00:56:38.510707 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:38.510653 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-xnr5x" podStartSLOduration=2.5106396010000003 podStartE2EDuration="2.510639601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:56:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:56:38.508814216 +0000 UTC m=+2490.979983967" watchObservedRunningTime="2026-10-31 00:56:38.510639601 +0000 UTC m=+2490.981809350" Oct 31 00:56:38.655480 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:38.646057 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-rf2wq" podStartSLOduration=2.646038016 podStartE2EDuration="2.646038016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:56:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:56:38.645446335 +0000 UTC m=+2491.116616083" watchObservedRunningTime="2026-10-31 00:56:38.646038016 +0000 UTC m=+2491.117207764" Oct 31 00:56:39.132580 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:39.132542 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-rf2wq" Oct 31 00:56:39.134396 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:39.134353 5222 patch_prober.go:29] interesting pod/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-rf2wq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" start-of-body= Oct 31 00:56:39.134553 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:39.134466 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-rf2wq" podUID="6c2a7c21-8b15-47fc-aa57-25b19399fd60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" Oct 31 00:56:39.188177 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:39.187838 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-xnr5x" Oct 31 00:56:39.189558 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:39.189528 5222 patch_prober.go:29] interesting pod/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-xnr5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" start-of-body= Oct 31 00:56:39.189638 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:39.189594 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-xnr5x" podUID="5dbc245a-0bc5-4a0e-83fb-a49841984b74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" Oct 31 00:56:39.675396 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:39.672043 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6eefacd4-395f-42f5-922d-fb2d971401c5" path="/var/lib/kubelet/pods/6eefacd4-395f-42f5-922d-fb2d971401c5/volumes" Oct 31 00:56:40.133547 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:40.133505 5222 patch_prober.go:29] interesting pod/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-rf2wq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" start-of-body= Oct 31 00:56:40.133714 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:40.133560 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-rf2wq" podUID="6c2a7c21-8b15-47fc-aa57-25b19399fd60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" Oct 31 00:56:40.187957 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:40.187916 5222 patch_prober.go:29] interesting pod/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-xnr5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" start-of-body= Oct 31 00:56:40.188239 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:40.188209 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-xnr5x" podUID="5dbc245a-0bc5-4a0e-83fb-a49841984b74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" Oct 31 00:56:41.133628 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:41.133589 5222 patch_prober.go:29] interesting pod/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-rf2wq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" start-of-body= Oct 31 00:56:41.135985 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:41.133645 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-rf2wq" podUID="6c2a7c21-8b15-47fc-aa57-25b19399fd60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" Oct 31 00:56:41.188203 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:41.188159 5222 patch_prober.go:29] interesting pod/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-xnr5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" start-of-body= Oct 31 00:56:41.188522 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:41.188488 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-xnr5x" podUID="5dbc245a-0bc5-4a0e-83fb-a49841984b74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" Oct 31 00:56:42.132625 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:42.132581 5222 patch_prober.go:29] interesting pod/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-rf2wq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" start-of-body= Oct 31 00:56:42.132813 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:42.132654 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-rf2wq" podUID="6c2a7c21-8b15-47fc-aa57-25b19399fd60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" Oct 31 00:56:42.188933 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:42.188890 5222 patch_prober.go:29] interesting pod/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-xnr5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" start-of-body= Oct 31 00:56:42.189345 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:42.188965 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-xnr5x" podUID="5dbc245a-0bc5-4a0e-83fb-a49841984b74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" Oct 31 00:56:42.905209 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:42.905186 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-j4tb5" Oct 31 00:56:42.983840 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:42.983785 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d359a4aa-8996-4615-9b33-fccc33762a6a-istio-token\" (UniqueName: \"kubernetes.io/projected/d359a4aa-8996-4615-9b33-fccc33762a6a-istio-token\") pod \"d359a4aa-8996-4615-9b33-fccc33762a6a\" (UID: \"d359a4aa-8996-4615-9b33-fccc33762a6a\") " Oct 31 00:56:42.984625 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:42.984599 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d359a4aa-8996-4615-9b33-fccc33762a6a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d359a4aa-8996-4615-9b33-fccc33762a6a-credential-socket\") pod \"d359a4aa-8996-4615-9b33-fccc33762a6a\" (UID: \"d359a4aa-8996-4615-9b33-fccc33762a6a\") " Oct 31 00:56:42.985108 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:42.985092 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d359a4aa-8996-4615-9b33-fccc33762a6a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d359a4aa-8996-4615-9b33-fccc33762a6a-istio-envoy\") pod \"d359a4aa-8996-4615-9b33-fccc33762a6a\" (UID: \"d359a4aa-8996-4615-9b33-fccc33762a6a\") " Oct 31 00:56:42.985282 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:42.985266 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d359a4aa-8996-4615-9b33-fccc33762a6a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d359a4aa-8996-4615-9b33-fccc33762a6a-istiod-ca-cert\") pod \"d359a4aa-8996-4615-9b33-fccc33762a6a\" (UID: \"d359a4aa-8996-4615-9b33-fccc33762a6a\") " Oct 31 00:56:42.985493 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:42.985475 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d359a4aa-8996-4615-9b33-fccc33762a6a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d359a4aa-8996-4615-9b33-fccc33762a6a-workload-socket\") pod \"d359a4aa-8996-4615-9b33-fccc33762a6a\" (UID: \"d359a4aa-8996-4615-9b33-fccc33762a6a\") " Oct 31 00:56:42.985712 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:42.985695 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d359a4aa-8996-4615-9b33-fccc33762a6a-kube-api-access-74htg\" (UniqueName: \"kubernetes.io/projected/d359a4aa-8996-4615-9b33-fccc33762a6a-kube-api-access-74htg\") pod \"d359a4aa-8996-4615-9b33-fccc33762a6a\" (UID: \"d359a4aa-8996-4615-9b33-fccc33762a6a\") " Oct 31 00:56:42.985878 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:42.985864 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d359a4aa-8996-4615-9b33-fccc33762a6a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d359a4aa-8996-4615-9b33-fccc33762a6a-istio-data\") pod \"d359a4aa-8996-4615-9b33-fccc33762a6a\" (UID: \"d359a4aa-8996-4615-9b33-fccc33762a6a\") " Oct 31 00:56:42.986060 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:42.986046 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d359a4aa-8996-4615-9b33-fccc33762a6a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d359a4aa-8996-4615-9b33-fccc33762a6a-workload-certs\") pod \"d359a4aa-8996-4615-9b33-fccc33762a6a\" (UID: \"d359a4aa-8996-4615-9b33-fccc33762a6a\") " Oct 31 00:56:42.986178 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:42.986164 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d359a4aa-8996-4615-9b33-fccc33762a6a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d359a4aa-8996-4615-9b33-fccc33762a6a-istio-podinfo\") pod \"d359a4aa-8996-4615-9b33-fccc33762a6a\" (UID: \"d359a4aa-8996-4615-9b33-fccc33762a6a\") " Oct 31 00:56:42.987475 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:42.985040 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d359a4aa-8996-4615-9b33-fccc33762a6a-credential-socket" pod "d359a4aa-8996-4615-9b33-fccc33762a6a" (UID: "d359a4aa-8996-4615-9b33-fccc33762a6a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:42.987475 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:42.987184 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d359a4aa-8996-4615-9b33-fccc33762a6a-workload-socket" pod "d359a4aa-8996-4615-9b33-fccc33762a6a" (UID: "d359a4aa-8996-4615-9b33-fccc33762a6a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:42.988514 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:42.988477 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d359a4aa-8996-4615-9b33-fccc33762a6a-workload-certs" pod "d359a4aa-8996-4615-9b33-fccc33762a6a" (UID: "d359a4aa-8996-4615-9b33-fccc33762a6a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:42.989000 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:42.988978 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d359a4aa-8996-4615-9b33-fccc33762a6a-istio-data" pod "d359a4aa-8996-4615-9b33-fccc33762a6a" (UID: "d359a4aa-8996-4615-9b33-fccc33762a6a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:42.989281 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:42.989261 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d359a4aa-8996-4615-9b33-fccc33762a6a-istiod-ca-cert" pod "d359a4aa-8996-4615-9b33-fccc33762a6a" (UID: "d359a4aa-8996-4615-9b33-fccc33762a6a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:56:42.991120 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:42.991081 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d359a4aa-8996-4615-9b33-fccc33762a6a-istio-token" pod "d359a4aa-8996-4615-9b33-fccc33762a6a" (UID: "d359a4aa-8996-4615-9b33-fccc33762a6a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:56:42.991351 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:42.991317 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d359a4aa-8996-4615-9b33-fccc33762a6a-istio-envoy" pod "d359a4aa-8996-4615-9b33-fccc33762a6a" (UID: "d359a4aa-8996-4615-9b33-fccc33762a6a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:42.992443 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:42.992412 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d359a4aa-8996-4615-9b33-fccc33762a6a-istio-podinfo" pod "d359a4aa-8996-4615-9b33-fccc33762a6a" (UID: "d359a4aa-8996-4615-9b33-fccc33762a6a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:56:42.992682 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:42.992657 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d359a4aa-8996-4615-9b33-fccc33762a6a-kube-api-access-74htg" pod "d359a4aa-8996-4615-9b33-fccc33762a6a" (UID: "d359a4aa-8996-4615-9b33-fccc33762a6a"). InnerVolumeSpecName "kube-api-access-74htg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:56:43.091604 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:43.091568 5222 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d359a4aa-8996-4615-9b33-fccc33762a6a-istiod-ca-cert\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:43.091860 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:43.091841 5222 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d359a4aa-8996-4615-9b33-fccc33762a6a-workload-socket\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:43.091985 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:43.091969 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74htg\" (UniqueName: \"kubernetes.io/projected/d359a4aa-8996-4615-9b33-fccc33762a6a-kube-api-access-74htg\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:43.092082 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:43.092071 5222 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d359a4aa-8996-4615-9b33-fccc33762a6a-istio-data\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:43.092177 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:43.092163 5222 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d359a4aa-8996-4615-9b33-fccc33762a6a-workload-certs\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:43.092283 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:43.092269 5222 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d359a4aa-8996-4615-9b33-fccc33762a6a-istio-podinfo\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:43.092451 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:43.092434 5222 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d359a4aa-8996-4615-9b33-fccc33762a6a-istio-token\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:43.092573 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:43.092559 5222 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d359a4aa-8996-4615-9b33-fccc33762a6a-credential-socket\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:43.092667 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:43.092656 5222 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d359a4aa-8996-4615-9b33-fccc33762a6a-istio-envoy\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:43.133228 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:43.132729 5222 patch_prober.go:29] interesting pod/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-rf2wq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" start-of-body= Oct 31 00:56:43.133228 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:43.132778 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-rf2wq" podUID="6c2a7c21-8b15-47fc-aa57-25b19399fd60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" Oct 31 00:56:43.191051 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:43.190702 5222 patch_prober.go:29] interesting pod/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-xnr5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" start-of-body= Oct 31 00:56:43.191051 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:43.190768 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-xnr5x" podUID="5dbc245a-0bc5-4a0e-83fb-a49841984b74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" Oct 31 00:56:43.426691 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:43.426592 5222 generic.go:356] "Generic (PLEG): container finished" podID="d359a4aa-8996-4615-9b33-fccc33762a6a" containerID="5d7a265874f1ee0b61e4e566f8de0fce151565afb36530586afa5e7b0790de22" exitCode=0 Oct 31 00:56:43.426853 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:43.426708 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-j4tb5" Oct 31 00:56:43.426853 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:43.426744 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-j4tb5" event={"ID":"d359a4aa-8996-4615-9b33-fccc33762a6a","Type":"ContainerDied","Data":"5d7a265874f1ee0b61e4e566f8de0fce151565afb36530586afa5e7b0790de22"} Oct 31 00:56:43.426853 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:43.426782 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-j4tb5" event={"ID":"d359a4aa-8996-4615-9b33-fccc33762a6a","Type":"ContainerDied","Data":"0b015d3eff42faf1a44f398a3aa0bb686ca4ae687ee23c68ed225263f98d571f"} Oct 31 00:56:43.426853 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:43.426802 5222 scope.go:122] "RemoveContainer" containerID="5d7a265874f1ee0b61e4e566f8de0fce151565afb36530586afa5e7b0790de22" Oct 31 00:56:43.454591 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:43.452556 5222 scope.go:122] "RemoveContainer" containerID="5d7a265874f1ee0b61e4e566f8de0fce151565afb36530586afa5e7b0790de22" Oct 31 00:56:43.477396 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:56:43.464573 5222 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d7a265874f1ee0b61e4e566f8de0fce151565afb36530586afa5e7b0790de22\": container with ID starting with 5d7a265874f1ee0b61e4e566f8de0fce151565afb36530586afa5e7b0790de22 not found: ID does not exist" containerID="5d7a265874f1ee0b61e4e566f8de0fce151565afb36530586afa5e7b0790de22" Oct 31 00:56:43.477396 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:43.464629 5222 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d7a265874f1ee0b61e4e566f8de0fce151565afb36530586afa5e7b0790de22"} err="failed to get container status \"5d7a265874f1ee0b61e4e566f8de0fce151565afb36530586afa5e7b0790de22\": rpc error: code = NotFound desc = could not find container \"5d7a265874f1ee0b61e4e566f8de0fce151565afb36530586afa5e7b0790de22\": container with ID starting with 5d7a265874f1ee0b61e4e566f8de0fce151565afb36530586afa5e7b0790de22 not found: ID does not exist" Oct 31 00:56:43.624291 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:43.624263 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-j4tb5"] Oct 31 00:56:43.626880 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:43.626859 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-j4tb5"] Oct 31 00:56:43.692285 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:43.692199 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d359a4aa-8996-4615-9b33-fccc33762a6a" path="/var/lib/kubelet/pods/d359a4aa-8996-4615-9b33-fccc33762a6a/volumes" Oct 31 00:56:44.133697 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:44.133654 5222 patch_prober.go:29] interesting pod/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-rf2wq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" start-of-body= Oct 31 00:56:44.133901 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:44.133724 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-rf2wq" podUID="6c2a7c21-8b15-47fc-aa57-25b19399fd60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" Oct 31 00:56:44.188986 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:44.188943 5222 patch_prober.go:29] interesting pod/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-xnr5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" start-of-body= Oct 31 00:56:44.189188 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:44.189015 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-xnr5x" podUID="5dbc245a-0bc5-4a0e-83fb-a49841984b74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" Oct 31 00:56:45.137700 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:45.137667 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-rf2wq" Oct 31 00:56:45.138103 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:45.138064 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-rf2wq" Oct 31 00:56:45.138938 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:45.138900 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-rf2wq" Oct 31 00:56:45.193215 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:45.193175 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-xnr5x" Oct 31 00:56:45.193573 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:45.193551 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-xnr5x" Oct 31 00:56:45.194459 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:45.194428 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-xnr5x" Oct 31 00:56:45.604039 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:45.604005 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-rf2wq"] Oct 31 00:56:45.616406 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:45.616360 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-xnr5x"] Oct 31 00:56:46.448457 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:46.448427 5222 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-xnr5x" secret="" err="secret \"e2e-worker-w0-101b9e-openshift-default-dockercfg-8l66m\" not found" Oct 31 00:56:46.448877 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:46.448562 5222 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-rf2wq" secret="" err="secret \"e2e-worker-w0-101b9e-openshift-default-dockercfg-8l66m\" not found" Oct 31 00:56:47.474242 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:47.474196 5222 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-xnr5x" podUID="5dbc245a-0bc5-4a0e-83fb-a49841984b74" containerName="istio-proxy" containerID="cri-o://6b89c4f76e335a5cd993d59356a76304dfdf589575910daf8579a616243ffb89" gracePeriod=30 Oct 31 00:56:47.474958 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:47.474262 5222 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-rf2wq" podUID="6c2a7c21-8b15-47fc-aa57-25b19399fd60" containerName="istio-proxy" containerID="cri-o://02fb429afcd7fd04a736c4259c64abdcc58ed1ebe8dcb0cae268d73ab73d5692" gracePeriod=30 Oct 31 00:56:49.331783 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:49.331748 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-412743-hdbhc"] Oct 31 00:56:49.382707 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:49.382660 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-412743-hdbhc"] Oct 31 00:56:49.655792 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:49.655711 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2721a0cc-1697-4f49-8c2b-b72f84186b8b" path="/var/lib/kubelet/pods/2721a0cc-1697-4f49-8c2b-b72f84186b8b/volumes" Oct 31 00:56:52.811202 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:52.811170 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-rf2wq" Oct 31 00:56:52.832432 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:52.832395 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-xnr5x" Oct 31 00:56:52.846667 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:52.846637 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c2a7c21-8b15-47fc-aa57-25b19399fd60-kube-api-access-lf7wf\" (UniqueName: \"kubernetes.io/projected/6c2a7c21-8b15-47fc-aa57-25b19399fd60-kube-api-access-lf7wf\") pod \"6c2a7c21-8b15-47fc-aa57-25b19399fd60\" (UID: \"6c2a7c21-8b15-47fc-aa57-25b19399fd60\") " Oct 31 00:56:52.846824 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:52.846675 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5dbc245a-0bc5-4a0e-83fb-a49841984b74-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dbc245a-0bc5-4a0e-83fb-a49841984b74-credential-socket\") pod \"5dbc245a-0bc5-4a0e-83fb-a49841984b74\" (UID: \"5dbc245a-0bc5-4a0e-83fb-a49841984b74\") " Oct 31 00:56:52.846824 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:52.846718 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5dbc245a-0bc5-4a0e-83fb-a49841984b74-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dbc245a-0bc5-4a0e-83fb-a49841984b74-workload-certs\") pod \"5dbc245a-0bc5-4a0e-83fb-a49841984b74\" (UID: \"5dbc245a-0bc5-4a0e-83fb-a49841984b74\") " Oct 31 00:56:52.846824 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:52.846749 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c2a7c21-8b15-47fc-aa57-25b19399fd60-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c2a7c21-8b15-47fc-aa57-25b19399fd60-istio-envoy\") pod \"6c2a7c21-8b15-47fc-aa57-25b19399fd60\" (UID: \"6c2a7c21-8b15-47fc-aa57-25b19399fd60\") " Oct 31 00:56:52.846824 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:52.846800 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c2a7c21-8b15-47fc-aa57-25b19399fd60-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c2a7c21-8b15-47fc-aa57-25b19399fd60-istiod-ca-cert\") pod \"6c2a7c21-8b15-47fc-aa57-25b19399fd60\" (UID: \"6c2a7c21-8b15-47fc-aa57-25b19399fd60\") " Oct 31 00:56:52.847061 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:52.846836 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c2a7c21-8b15-47fc-aa57-25b19399fd60-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c2a7c21-8b15-47fc-aa57-25b19399fd60-workload-socket\") pod \"6c2a7c21-8b15-47fc-aa57-25b19399fd60\" (UID: \"6c2a7c21-8b15-47fc-aa57-25b19399fd60\") " Oct 31 00:56:52.847061 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:52.846870 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c2a7c21-8b15-47fc-aa57-25b19399fd60-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c2a7c21-8b15-47fc-aa57-25b19399fd60-workload-certs\") pod \"6c2a7c21-8b15-47fc-aa57-25b19399fd60\" (UID: \"6c2a7c21-8b15-47fc-aa57-25b19399fd60\") " Oct 31 00:56:52.847061 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:52.846897 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5dbc245a-0bc5-4a0e-83fb-a49841984b74-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dbc245a-0bc5-4a0e-83fb-a49841984b74-istio-envoy\") pod \"5dbc245a-0bc5-4a0e-83fb-a49841984b74\" (UID: \"5dbc245a-0bc5-4a0e-83fb-a49841984b74\") " Oct 31 00:56:52.847061 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:52.846978 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5dbc245a-0bc5-4a0e-83fb-a49841984b74-kube-api-access-9zg4g\" (UniqueName: \"kubernetes.io/projected/5dbc245a-0bc5-4a0e-83fb-a49841984b74-kube-api-access-9zg4g\") pod \"5dbc245a-0bc5-4a0e-83fb-a49841984b74\" (UID: \"5dbc245a-0bc5-4a0e-83fb-a49841984b74\") " Oct 31 00:56:52.847061 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:52.847010 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c2a7c21-8b15-47fc-aa57-25b19399fd60-istio-token\" (UniqueName: \"kubernetes.io/projected/6c2a7c21-8b15-47fc-aa57-25b19399fd60-istio-token\") pod \"6c2a7c21-8b15-47fc-aa57-25b19399fd60\" (UID: \"6c2a7c21-8b15-47fc-aa57-25b19399fd60\") " Oct 31 00:56:52.847061 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:52.847010 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dbc245a-0bc5-4a0e-83fb-a49841984b74-credential-socket" pod "5dbc245a-0bc5-4a0e-83fb-a49841984b74" (UID: "5dbc245a-0bc5-4a0e-83fb-a49841984b74"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:52.847061 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:52.847053 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6c2a7c21-8b15-47fc-aa57-25b19399fd60-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c2a7c21-8b15-47fc-aa57-25b19399fd60-istio-podinfo\") pod \"6c2a7c21-8b15-47fc-aa57-25b19399fd60\" (UID: \"6c2a7c21-8b15-47fc-aa57-25b19399fd60\") " Oct 31 00:56:52.847518 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:52.847049 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dbc245a-0bc5-4a0e-83fb-a49841984b74-workload-certs" pod "5dbc245a-0bc5-4a0e-83fb-a49841984b74" (UID: "5dbc245a-0bc5-4a0e-83fb-a49841984b74"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:52.847518 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:52.847159 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c2a7c21-8b15-47fc-aa57-25b19399fd60-workload-certs" pod "6c2a7c21-8b15-47fc-aa57-25b19399fd60" (UID: "6c2a7c21-8b15-47fc-aa57-25b19399fd60"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:52.847518 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:52.847277 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c2a7c21-8b15-47fc-aa57-25b19399fd60-workload-socket" pod "6c2a7c21-8b15-47fc-aa57-25b19399fd60" (UID: "6c2a7c21-8b15-47fc-aa57-25b19399fd60"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:52.847518 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:52.847483 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c2a7c21-8b15-47fc-aa57-25b19399fd60-istiod-ca-cert" pod "6c2a7c21-8b15-47fc-aa57-25b19399fd60" (UID: "6c2a7c21-8b15-47fc-aa57-25b19399fd60"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:56:52.847908 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:52.847540 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c2a7c21-8b15-47fc-aa57-25b19399fd60-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c2a7c21-8b15-47fc-aa57-25b19399fd60-istio-data\") pod \"6c2a7c21-8b15-47fc-aa57-25b19399fd60\" (UID: \"6c2a7c21-8b15-47fc-aa57-25b19399fd60\") " Oct 31 00:56:52.847908 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:52.847654 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5dbc245a-0bc5-4a0e-83fb-a49841984b74-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dbc245a-0bc5-4a0e-83fb-a49841984b74-istio-data\") pod \"5dbc245a-0bc5-4a0e-83fb-a49841984b74\" (UID: \"5dbc245a-0bc5-4a0e-83fb-a49841984b74\") " Oct 31 00:56:52.847908 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:52.847794 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c2a7c21-8b15-47fc-aa57-25b19399fd60-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c2a7c21-8b15-47fc-aa57-25b19399fd60-credential-socket\") pod \"6c2a7c21-8b15-47fc-aa57-25b19399fd60\" (UID: \"6c2a7c21-8b15-47fc-aa57-25b19399fd60\") " Oct 31 00:56:52.847908 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:52.847883 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5dbc245a-0bc5-4a0e-83fb-a49841984b74-istio-token\" (UniqueName: \"kubernetes.io/projected/5dbc245a-0bc5-4a0e-83fb-a49841984b74-istio-token\") pod \"5dbc245a-0bc5-4a0e-83fb-a49841984b74\" (UID: \"5dbc245a-0bc5-4a0e-83fb-a49841984b74\") " Oct 31 00:56:52.848137 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:52.847938 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5dbc245a-0bc5-4a0e-83fb-a49841984b74-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dbc245a-0bc5-4a0e-83fb-a49841984b74-istio-podinfo\") pod \"5dbc245a-0bc5-4a0e-83fb-a49841984b74\" (UID: \"5dbc245a-0bc5-4a0e-83fb-a49841984b74\") " Oct 31 00:56:52.848137 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:52.847977 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5dbc245a-0bc5-4a0e-83fb-a49841984b74-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dbc245a-0bc5-4a0e-83fb-a49841984b74-istiod-ca-cert\") pod \"5dbc245a-0bc5-4a0e-83fb-a49841984b74\" (UID: \"5dbc245a-0bc5-4a0e-83fb-a49841984b74\") " Oct 31 00:56:52.848137 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:52.848022 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5dbc245a-0bc5-4a0e-83fb-a49841984b74-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dbc245a-0bc5-4a0e-83fb-a49841984b74-workload-socket\") pod \"5dbc245a-0bc5-4a0e-83fb-a49841984b74\" (UID: \"5dbc245a-0bc5-4a0e-83fb-a49841984b74\") " Oct 31 00:56:52.849329 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:52.848551 5222 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dbc245a-0bc5-4a0e-83fb-a49841984b74-credential-socket\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:52.849329 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:52.848597 5222 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dbc245a-0bc5-4a0e-83fb-a49841984b74-workload-certs\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:52.849329 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:52.848613 5222 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c2a7c21-8b15-47fc-aa57-25b19399fd60-istiod-ca-cert\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:52.849329 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:52.848680 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c2a7c21-8b15-47fc-aa57-25b19399fd60-credential-socket" pod "6c2a7c21-8b15-47fc-aa57-25b19399fd60" (UID: "6c2a7c21-8b15-47fc-aa57-25b19399fd60"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:52.849329 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:52.848697 5222 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c2a7c21-8b15-47fc-aa57-25b19399fd60-workload-socket\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:52.849329 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:52.848712 5222 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c2a7c21-8b15-47fc-aa57-25b19399fd60-workload-certs\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:52.849329 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:52.848867 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c2a7c21-8b15-47fc-aa57-25b19399fd60-istio-data" pod "6c2a7c21-8b15-47fc-aa57-25b19399fd60" (UID: "6c2a7c21-8b15-47fc-aa57-25b19399fd60"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:52.849329 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:52.848899 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dbc245a-0bc5-4a0e-83fb-a49841984b74-workload-socket" pod "5dbc245a-0bc5-4a0e-83fb-a49841984b74" (UID: "5dbc245a-0bc5-4a0e-83fb-a49841984b74"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:52.849329 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:52.849074 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dbc245a-0bc5-4a0e-83fb-a49841984b74-istio-data" pod "5dbc245a-0bc5-4a0e-83fb-a49841984b74" (UID: "5dbc245a-0bc5-4a0e-83fb-a49841984b74"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:52.849907 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:52.849800 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c2a7c21-8b15-47fc-aa57-25b19399fd60-istio-envoy" pod "6c2a7c21-8b15-47fc-aa57-25b19399fd60" (UID: "6c2a7c21-8b15-47fc-aa57-25b19399fd60"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:52.849907 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:52.849835 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dbc245a-0bc5-4a0e-83fb-a49841984b74-istio-envoy" pod "5dbc245a-0bc5-4a0e-83fb-a49841984b74" (UID: "5dbc245a-0bc5-4a0e-83fb-a49841984b74"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:52.849907 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:52.849873 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dbc245a-0bc5-4a0e-83fb-a49841984b74-kube-api-access-9zg4g" pod "5dbc245a-0bc5-4a0e-83fb-a49841984b74" (UID: "5dbc245a-0bc5-4a0e-83fb-a49841984b74"). InnerVolumeSpecName "kube-api-access-9zg4g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:56:52.850152 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:52.850124 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c2a7c21-8b15-47fc-aa57-25b19399fd60-istio-token" pod "6c2a7c21-8b15-47fc-aa57-25b19399fd60" (UID: "6c2a7c21-8b15-47fc-aa57-25b19399fd60"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:56:52.850152 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:52.850130 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5dbc245a-0bc5-4a0e-83fb-a49841984b74-istiod-ca-cert" pod "5dbc245a-0bc5-4a0e-83fb-a49841984b74" (UID: "5dbc245a-0bc5-4a0e-83fb-a49841984b74"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:56:52.850242 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:52.850193 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6c2a7c21-8b15-47fc-aa57-25b19399fd60-istio-podinfo" pod "6c2a7c21-8b15-47fc-aa57-25b19399fd60" (UID: "6c2a7c21-8b15-47fc-aa57-25b19399fd60"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:56:52.850324 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:52.850299 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c2a7c21-8b15-47fc-aa57-25b19399fd60-kube-api-access-lf7wf" pod "6c2a7c21-8b15-47fc-aa57-25b19399fd60" (UID: "6c2a7c21-8b15-47fc-aa57-25b19399fd60"). InnerVolumeSpecName "kube-api-access-lf7wf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:56:52.851031 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:52.851014 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dbc245a-0bc5-4a0e-83fb-a49841984b74-istio-token" pod "5dbc245a-0bc5-4a0e-83fb-a49841984b74" (UID: "5dbc245a-0bc5-4a0e-83fb-a49841984b74"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:56:52.851446 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:52.851424 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5dbc245a-0bc5-4a0e-83fb-a49841984b74-istio-podinfo" pod "5dbc245a-0bc5-4a0e-83fb-a49841984b74" (UID: "5dbc245a-0bc5-4a0e-83fb-a49841984b74"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:56:52.949715 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:52.949673 5222 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dbc245a-0bc5-4a0e-83fb-a49841984b74-istio-podinfo\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:52.949715 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:52.949707 5222 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dbc245a-0bc5-4a0e-83fb-a49841984b74-istiod-ca-cert\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:52.949715 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:52.949718 5222 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dbc245a-0bc5-4a0e-83fb-a49841984b74-workload-socket\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:52.949966 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:52.949728 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lf7wf\" (UniqueName: \"kubernetes.io/projected/6c2a7c21-8b15-47fc-aa57-25b19399fd60-kube-api-access-lf7wf\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:52.949966 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:52.949738 5222 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c2a7c21-8b15-47fc-aa57-25b19399fd60-istio-envoy\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:52.949966 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:52.949748 5222 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dbc245a-0bc5-4a0e-83fb-a49841984b74-istio-envoy\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:52.949966 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:52.949758 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zg4g\" (UniqueName: \"kubernetes.io/projected/5dbc245a-0bc5-4a0e-83fb-a49841984b74-kube-api-access-9zg4g\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:52.949966 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:52.949766 5222 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c2a7c21-8b15-47fc-aa57-25b19399fd60-istio-token\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:52.949966 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:52.949775 5222 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c2a7c21-8b15-47fc-aa57-25b19399fd60-istio-podinfo\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:52.949966 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:52.949783 5222 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c2a7c21-8b15-47fc-aa57-25b19399fd60-istio-data\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:52.949966 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:52.949790 5222 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dbc245a-0bc5-4a0e-83fb-a49841984b74-istio-data\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:52.949966 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:52.949798 5222 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c2a7c21-8b15-47fc-aa57-25b19399fd60-credential-socket\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:52.949966 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:52.949807 5222 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5dbc245a-0bc5-4a0e-83fb-a49841984b74-istio-token\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:53.515298 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:53.515266 5222 generic.go:356] "Generic (PLEG): container finished" podID="5dbc245a-0bc5-4a0e-83fb-a49841984b74" containerID="6b89c4f76e335a5cd993d59356a76304dfdf589575910daf8579a616243ffb89" exitCode=0 Oct 31 00:56:53.515488 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:53.515311 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-xnr5x" Oct 31 00:56:53.515488 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:53.515347 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-xnr5x" event={"ID":"5dbc245a-0bc5-4a0e-83fb-a49841984b74","Type":"ContainerDied","Data":"6b89c4f76e335a5cd993d59356a76304dfdf589575910daf8579a616243ffb89"} Oct 31 00:56:53.515488 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:53.515422 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-xnr5x" event={"ID":"5dbc245a-0bc5-4a0e-83fb-a49841984b74","Type":"ContainerDied","Data":"77b842f447c6a1dc9d4927c13056a91c49622f4615375cc06389ef3c81fd6971"} Oct 31 00:56:53.515488 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:53.515448 5222 scope.go:122] "RemoveContainer" containerID="6b89c4f76e335a5cd993d59356a76304dfdf589575910daf8579a616243ffb89" Oct 31 00:56:53.517096 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:53.517073 5222 generic.go:356] "Generic (PLEG): container finished" podID="6c2a7c21-8b15-47fc-aa57-25b19399fd60" containerID="02fb429afcd7fd04a736c4259c64abdcc58ed1ebe8dcb0cae268d73ab73d5692" exitCode=0 Oct 31 00:56:53.517245 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:53.517119 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-rf2wq" Oct 31 00:56:53.517245 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:53.517128 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-rf2wq" event={"ID":"6c2a7c21-8b15-47fc-aa57-25b19399fd60","Type":"ContainerDied","Data":"02fb429afcd7fd04a736c4259c64abdcc58ed1ebe8dcb0cae268d73ab73d5692"} Oct 31 00:56:53.517245 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:53.517162 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-rf2wq" event={"ID":"6c2a7c21-8b15-47fc-aa57-25b19399fd60","Type":"ContainerDied","Data":"e8b1cc9d48e53165a20fa110741136d111182c68e497ffd04ce23b0a37c430f4"} Oct 31 00:56:53.530312 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:53.530295 5222 scope.go:122] "RemoveContainer" containerID="6b89c4f76e335a5cd993d59356a76304dfdf589575910daf8579a616243ffb89" Oct 31 00:56:53.530718 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:56:53.530687 5222 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b89c4f76e335a5cd993d59356a76304dfdf589575910daf8579a616243ffb89\": container with ID starting with 6b89c4f76e335a5cd993d59356a76304dfdf589575910daf8579a616243ffb89 not found: ID does not exist" containerID="6b89c4f76e335a5cd993d59356a76304dfdf589575910daf8579a616243ffb89" Oct 31 00:56:53.530840 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:53.530723 5222 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b89c4f76e335a5cd993d59356a76304dfdf589575910daf8579a616243ffb89"} err="failed to get container status \"6b89c4f76e335a5cd993d59356a76304dfdf589575910daf8579a616243ffb89\": rpc error: code = NotFound desc = could not find container \"6b89c4f76e335a5cd993d59356a76304dfdf589575910daf8579a616243ffb89\": container with ID starting with 6b89c4f76e335a5cd993d59356a76304dfdf589575910daf8579a616243ffb89 not found: ID does not exist" Oct 31 00:56:53.530840 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:53.530742 5222 scope.go:122] "RemoveContainer" containerID="02fb429afcd7fd04a736c4259c64abdcc58ed1ebe8dcb0cae268d73ab73d5692" Oct 31 00:56:53.540839 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:53.540806 5222 scope.go:122] "RemoveContainer" containerID="02fb429afcd7fd04a736c4259c64abdcc58ed1ebe8dcb0cae268d73ab73d5692" Oct 31 00:56:53.541150 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:56:53.541128 5222 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02fb429afcd7fd04a736c4259c64abdcc58ed1ebe8dcb0cae268d73ab73d5692\": container with ID starting with 02fb429afcd7fd04a736c4259c64abdcc58ed1ebe8dcb0cae268d73ab73d5692 not found: ID does not exist" containerID="02fb429afcd7fd04a736c4259c64abdcc58ed1ebe8dcb0cae268d73ab73d5692" Oct 31 00:56:53.541224 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:53.541158 5222 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02fb429afcd7fd04a736c4259c64abdcc58ed1ebe8dcb0cae268d73ab73d5692"} err="failed to get container status \"02fb429afcd7fd04a736c4259c64abdcc58ed1ebe8dcb0cae268d73ab73d5692\": rpc error: code = NotFound desc = could not find container \"02fb429afcd7fd04a736c4259c64abdcc58ed1ebe8dcb0cae268d73ab73d5692\": container with ID starting with 02fb429afcd7fd04a736c4259c64abdcc58ed1ebe8dcb0cae268d73ab73d5692 not found: ID does not exist" Oct 31 00:56:53.685821 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:53.685788 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-rf2wq"] Oct 31 00:56:53.735975 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:53.735939 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-rf2wq"] Oct 31 00:56:53.801615 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:53.801584 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-xnr5x"] Oct 31 00:56:53.805457 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:53.805431 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-xnr5x"] Oct 31 00:56:55.655630 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:55.655591 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5dbc245a-0bc5-4a0e-83fb-a49841984b74" path="/var/lib/kubelet/pods/5dbc245a-0bc5-4a0e-83fb-a49841984b74/volumes" Oct 31 00:56:55.656083 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:55.656070 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c2a7c21-8b15-47fc-aa57-25b19399fd60" path="/var/lib/kubelet/pods/6c2a7c21-8b15-47fc-aa57-25b19399fd60/volumes" Oct 31 00:56:56.141003 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:56.140965 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-412743-openshift-default-7bddc8b5df-thfvg"] Oct 31 00:56:56.141252 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:56:56.141225 5222 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-412743-openshift-default-7bddc8b5df-thfvg" podUID="ede619b3-54d8-4ba1-9c2e-237e54461390" containerName="istio-proxy" containerID="cri-o://7e82c1d645c963553ad4634ab5649719ff344bbf6ef1096bf3960fea44b98eec" gracePeriod=30 Oct 31 00:57:01.399050 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:57:01.399019 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-412743-openshift-default-7bddc8b5df-thfvg" Oct 31 00:57:01.454546 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:57:01.454448 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ede619b3-54d8-4ba1-9c2e-237e54461390-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ede619b3-54d8-4ba1-9c2e-237e54461390-istio-podinfo\") pod \"ede619b3-54d8-4ba1-9c2e-237e54461390\" (UID: \"ede619b3-54d8-4ba1-9c2e-237e54461390\") " Oct 31 00:57:01.454715 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:57:01.454547 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ede619b3-54d8-4ba1-9c2e-237e54461390-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ede619b3-54d8-4ba1-9c2e-237e54461390-workload-certs\") pod \"ede619b3-54d8-4ba1-9c2e-237e54461390\" (UID: \"ede619b3-54d8-4ba1-9c2e-237e54461390\") " Oct 31 00:57:01.454715 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:57:01.454594 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ede619b3-54d8-4ba1-9c2e-237e54461390-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ede619b3-54d8-4ba1-9c2e-237e54461390-istio-envoy\") pod \"ede619b3-54d8-4ba1-9c2e-237e54461390\" (UID: \"ede619b3-54d8-4ba1-9c2e-237e54461390\") " Oct 31 00:57:01.454715 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:57:01.454639 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ede619b3-54d8-4ba1-9c2e-237e54461390-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ede619b3-54d8-4ba1-9c2e-237e54461390-istio-data\") pod \"ede619b3-54d8-4ba1-9c2e-237e54461390\" (UID: \"ede619b3-54d8-4ba1-9c2e-237e54461390\") " Oct 31 00:57:01.454715 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:57:01.454673 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ede619b3-54d8-4ba1-9c2e-237e54461390-kube-api-access-ph4lj\" (UniqueName: \"kubernetes.io/projected/ede619b3-54d8-4ba1-9c2e-237e54461390-kube-api-access-ph4lj\") pod \"ede619b3-54d8-4ba1-9c2e-237e54461390\" (UID: \"ede619b3-54d8-4ba1-9c2e-237e54461390\") " Oct 31 00:57:01.454953 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:57:01.454730 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ede619b3-54d8-4ba1-9c2e-237e54461390-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ede619b3-54d8-4ba1-9c2e-237e54461390-credential-socket\") pod \"ede619b3-54d8-4ba1-9c2e-237e54461390\" (UID: \"ede619b3-54d8-4ba1-9c2e-237e54461390\") " Oct 31 00:57:01.454953 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:57:01.454786 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ede619b3-54d8-4ba1-9c2e-237e54461390-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ede619b3-54d8-4ba1-9c2e-237e54461390-istiod-ca-cert\") pod \"ede619b3-54d8-4ba1-9c2e-237e54461390\" (UID: \"ede619b3-54d8-4ba1-9c2e-237e54461390\") " Oct 31 00:57:01.454953 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:57:01.454857 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ede619b3-54d8-4ba1-9c2e-237e54461390-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ede619b3-54d8-4ba1-9c2e-237e54461390-workload-socket\") pod \"ede619b3-54d8-4ba1-9c2e-237e54461390\" (UID: \"ede619b3-54d8-4ba1-9c2e-237e54461390\") " Oct 31 00:57:01.454953 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:57:01.454916 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ede619b3-54d8-4ba1-9c2e-237e54461390-istio-token\" (UniqueName: \"kubernetes.io/projected/ede619b3-54d8-4ba1-9c2e-237e54461390-istio-token\") pod \"ede619b3-54d8-4ba1-9c2e-237e54461390\" (UID: \"ede619b3-54d8-4ba1-9c2e-237e54461390\") " Oct 31 00:57:01.455312 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:57:01.454907 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ede619b3-54d8-4ba1-9c2e-237e54461390-workload-certs" pod "ede619b3-54d8-4ba1-9c2e-237e54461390" (UID: "ede619b3-54d8-4ba1-9c2e-237e54461390"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:57:01.455312 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:57:01.454979 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ede619b3-54d8-4ba1-9c2e-237e54461390-istio-data" pod "ede619b3-54d8-4ba1-9c2e-237e54461390" (UID: "ede619b3-54d8-4ba1-9c2e-237e54461390"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:57:01.455312 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:57:01.455200 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ede619b3-54d8-4ba1-9c2e-237e54461390-workload-socket" pod "ede619b3-54d8-4ba1-9c2e-237e54461390" (UID: "ede619b3-54d8-4ba1-9c2e-237e54461390"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:57:01.455312 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:57:01.455229 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ede619b3-54d8-4ba1-9c2e-237e54461390-istiod-ca-cert" pod "ede619b3-54d8-4ba1-9c2e-237e54461390" (UID: "ede619b3-54d8-4ba1-9c2e-237e54461390"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:57:01.455630 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:57:01.455599 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ede619b3-54d8-4ba1-9c2e-237e54461390-credential-socket" pod "ede619b3-54d8-4ba1-9c2e-237e54461390" (UID: "ede619b3-54d8-4ba1-9c2e-237e54461390"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:57:01.456995 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:57:01.456968 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ede619b3-54d8-4ba1-9c2e-237e54461390-istio-podinfo" pod "ede619b3-54d8-4ba1-9c2e-237e54461390" (UID: "ede619b3-54d8-4ba1-9c2e-237e54461390"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:57:01.457105 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:57:01.456985 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ede619b3-54d8-4ba1-9c2e-237e54461390-istio-envoy" pod "ede619b3-54d8-4ba1-9c2e-237e54461390" (UID: "ede619b3-54d8-4ba1-9c2e-237e54461390"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:57:01.457105 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:57:01.457026 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ede619b3-54d8-4ba1-9c2e-237e54461390-kube-api-access-ph4lj" pod "ede619b3-54d8-4ba1-9c2e-237e54461390" (UID: "ede619b3-54d8-4ba1-9c2e-237e54461390"). InnerVolumeSpecName "kube-api-access-ph4lj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:57:01.457255 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:57:01.457224 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ede619b3-54d8-4ba1-9c2e-237e54461390-istio-token" pod "ede619b3-54d8-4ba1-9c2e-237e54461390" (UID: "ede619b3-54d8-4ba1-9c2e-237e54461390"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:57:01.556813 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:57:01.556771 5222 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ede619b3-54d8-4ba1-9c2e-237e54461390-workload-socket\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:57:01.556813 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:57:01.556805 5222 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ede619b3-54d8-4ba1-9c2e-237e54461390-istio-token\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:57:01.556813 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:57:01.556819 5222 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ede619b3-54d8-4ba1-9c2e-237e54461390-istio-podinfo\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:57:01.557085 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:57:01.556833 5222 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ede619b3-54d8-4ba1-9c2e-237e54461390-workload-certs\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:57:01.557085 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:57:01.556844 5222 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ede619b3-54d8-4ba1-9c2e-237e54461390-istio-envoy\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:57:01.557085 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:57:01.556857 5222 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ede619b3-54d8-4ba1-9c2e-237e54461390-istio-data\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:57:01.557085 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:57:01.556869 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ph4lj\" (UniqueName: \"kubernetes.io/projected/ede619b3-54d8-4ba1-9c2e-237e54461390-kube-api-access-ph4lj\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:57:01.557085 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:57:01.556882 5222 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ede619b3-54d8-4ba1-9c2e-237e54461390-credential-socket\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:57:01.557085 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:57:01.556897 5222 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ede619b3-54d8-4ba1-9c2e-237e54461390-istiod-ca-cert\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:57:01.575409 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:57:01.575355 5222 generic.go:356] "Generic (PLEG): container finished" podID="ede619b3-54d8-4ba1-9c2e-237e54461390" containerID="7e82c1d645c963553ad4634ab5649719ff344bbf6ef1096bf3960fea44b98eec" exitCode=0 Oct 31 00:57:01.575409 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:57:01.575400 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-412743-openshift-default-7bddc8b5df-thfvg" event={"ID":"ede619b3-54d8-4ba1-9c2e-237e54461390","Type":"ContainerDied","Data":"7e82c1d645c963553ad4634ab5649719ff344bbf6ef1096bf3960fea44b98eec"} Oct 31 00:57:01.575647 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:57:01.575417 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-412743-openshift-default-7bddc8b5df-thfvg" Oct 31 00:57:01.575647 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:57:01.575488 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-412743-openshift-default-7bddc8b5df-thfvg" event={"ID":"ede619b3-54d8-4ba1-9c2e-237e54461390","Type":"ContainerDied","Data":"a2f1d7f37f593fffc225daf3fe7daacb82e32fb8ceaaecce0ab0c6d08aec4cea"} Oct 31 00:57:01.575647 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:57:01.575511 5222 scope.go:122] "RemoveContainer" containerID="7e82c1d645c963553ad4634ab5649719ff344bbf6ef1096bf3960fea44b98eec" Oct 31 00:57:01.586256 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:57:01.586236 5222 scope.go:122] "RemoveContainer" containerID="7e82c1d645c963553ad4634ab5649719ff344bbf6ef1096bf3960fea44b98eec" Oct 31 00:57:01.586599 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:57:01.586575 5222 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e82c1d645c963553ad4634ab5649719ff344bbf6ef1096bf3960fea44b98eec\": container with ID starting with 7e82c1d645c963553ad4634ab5649719ff344bbf6ef1096bf3960fea44b98eec not found: ID does not exist" containerID="7e82c1d645c963553ad4634ab5649719ff344bbf6ef1096bf3960fea44b98eec" Oct 31 00:57:01.586692 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:57:01.586607 5222 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e82c1d645c963553ad4634ab5649719ff344bbf6ef1096bf3960fea44b98eec"} err="failed to get container status \"7e82c1d645c963553ad4634ab5649719ff344bbf6ef1096bf3960fea44b98eec\": rpc error: code = NotFound desc = could not find container \"7e82c1d645c963553ad4634ab5649719ff344bbf6ef1096bf3960fea44b98eec\": container with ID starting with 7e82c1d645c963553ad4634ab5649719ff344bbf6ef1096bf3960fea44b98eec not found: ID does not exist" Oct 31 00:57:01.745970 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:57:01.745888 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-412743-openshift-default-7bddc8b5df-thfvg"] Oct 31 00:57:01.751068 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:57:01.751029 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-412743-openshift-default-7bddc8b5df-thfvg"] Oct 31 00:57:02.820935 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:57:02.820902 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-79tk8"] Oct 31 00:57:03.655713 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:57:03.655681 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ede619b3-54d8-4ba1-9c2e-237e54461390" path="/var/lib/kubelet/pods/ede619b3-54d8-4ba1-9c2e-237e54461390/volumes" Oct 31 00:57:05.890337 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:57:05.890300 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 00:57:09.974635 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:57:09.974603 5222 scope.go:122] "RemoveContainer" containerID="d1ab91eb128938b620d59f10568d2acf317af134a639088519edcaf215b4c721" Oct 31 00:57:09.984255 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:57:09.984226 5222 scope.go:122] "RemoveContainer" containerID="53a6874622882ef8b0f3692d4fafd3f8c48effb925ff98d449556b45fa93f1b9" Oct 31 00:57:09.993654 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:57:09.993629 5222 scope.go:122] "RemoveContainer" containerID="0fe56ba038af95dfab8d305b69a236e62d86a479a246be2bcbd21818a410f28a" Oct 31 00:57:20.050671 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:57:20.050579 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bdfc3f86-6vskf"] Oct 31 00:57:20.101877 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:57:20.101841 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bdfc3f86-6vskf"] Oct 31 00:57:21.655927 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:57:21.655889 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f697756-a33a-4883-a075-a97f84e3d3c4" path="/var/lib/kubelet/pods/7f697756-a33a-4883-a075-a97f84e3d3c4/volumes" Oct 31 00:57:31.890946 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:57:31.890910 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-79tk8"] Oct 31 00:57:35.890997 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:57:35.890955 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 00:57:38.059411 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:57:38.059334 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-116b593b-h778h"] Oct 31 00:57:38.110281 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:57:38.110252 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aaf18069-4hwxp"] Oct 31 00:57:38.112471 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:57:38.112457 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-116b593b-h778h"] Oct 31 00:57:38.114384 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:57:38.114350 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aaf18069-4hwxp"] Oct 31 00:57:39.654981 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:57:39.654945 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0aeab769-a2dc-473e-ba35-441312562df8" path="/var/lib/kubelet/pods/0aeab769-a2dc-473e-ba35-441312562df8/volumes" Oct 31 00:57:39.655399 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:57:39.655361 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4399764-d07a-4838-ad80-57801a53c83c" path="/var/lib/kubelet/pods/e4399764-d07a-4838-ad80-57801a53c83c/volumes" Oct 31 00:57:58.391734 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:57:58.391692 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f85ed32c-mz22p"] Oct 31 00:57:58.392234 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:57:58.392164 5222 cpu_manager.go:414] "containerMap: removing container" podUID="6c2a7c21-8b15-47fc-aa57-25b19399fd60" containerName="istio-proxy" Oct 31 00:57:58.392234 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:57:58.392176 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c2a7c21-8b15-47fc-aa57-25b19399fd60" containerName="istio-proxy" Oct 31 00:57:58.392234 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:57:58.392204 5222 cpu_manager.go:414] "containerMap: removing container" podUID="d359a4aa-8996-4615-9b33-fccc33762a6a" containerName="istio-proxy" Oct 31 00:57:58.392234 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:57:58.392209 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d359a4aa-8996-4615-9b33-fccc33762a6a" containerName="istio-proxy" Oct 31 00:57:58.392234 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:57:58.392217 5222 cpu_manager.go:414] "containerMap: removing container" podUID="5dbc245a-0bc5-4a0e-83fb-a49841984b74" containerName="istio-proxy" Oct 31 00:57:58.392234 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:57:58.392222 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5dbc245a-0bc5-4a0e-83fb-a49841984b74" containerName="istio-proxy" Oct 31 00:57:58.392469 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:57:58.392239 5222 cpu_manager.go:414] "containerMap: removing container" podUID="ede619b3-54d8-4ba1-9c2e-237e54461390" containerName="istio-proxy" Oct 31 00:57:58.392469 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:57:58.392244 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ede619b3-54d8-4ba1-9c2e-237e54461390" containerName="istio-proxy" Oct 31 00:57:58.392469 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:57:58.392304 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="d359a4aa-8996-4615-9b33-fccc33762a6a" containerName="istio-proxy" Oct 31 00:57:58.392469 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:57:58.392312 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="5dbc245a-0bc5-4a0e-83fb-a49841984b74" containerName="istio-proxy" Oct 31 00:57:58.392469 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:57:58.392321 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="ede619b3-54d8-4ba1-9c2e-237e54461390" containerName="istio-proxy" Oct 31 00:57:58.392469 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:57:58.392328 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c2a7c21-8b15-47fc-aa57-25b19399fd60" containerName="istio-proxy" Oct 31 00:57:58.396831 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:57:58.396814 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f85ed32c-mz22p" Oct 31 00:57:58.459669 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:57:58.459629 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f85ed32c-mz22p"] Oct 31 00:57:58.481101 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:57:58.481062 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24d0a0fc-07f1-4f6b-b7c8-8a89a97d5548-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f85ed32c-mz22p\" (UID: \"24d0a0fc-07f1-4f6b-b7c8-8a89a97d5548\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f85ed32c-mz22p" Oct 31 00:57:58.481278 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:57:58.481122 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8rzb\" (UniqueName: \"kubernetes.io/projected/24d0a0fc-07f1-4f6b-b7c8-8a89a97d5548-kube-api-access-x8rzb\") pod \"maas-api-revoke-sub-f85ed32c-mz22p\" (UID: \"24d0a0fc-07f1-4f6b-b7c8-8a89a97d5548\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f85ed32c-mz22p" Oct 31 00:57:58.481322 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:57:58.481265 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24d0a0fc-07f1-4f6b-b7c8-8a89a97d5548-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f85ed32c-mz22p\" (UID: \"24d0a0fc-07f1-4f6b-b7c8-8a89a97d5548\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f85ed32c-mz22p" Oct 31 00:57:58.507428 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:57:58.507364 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:57:58.507428 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:57:58.507431 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:57:58.507682 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:57:58.507487 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:57:58.507682 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:57:58.507512 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfz9z\"" Oct 31 00:57:58.582102 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:57:58.582058 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24d0a0fc-07f1-4f6b-b7c8-8a89a97d5548-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f85ed32c-mz22p\" (UID: \"24d0a0fc-07f1-4f6b-b7c8-8a89a97d5548\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f85ed32c-mz22p" Oct 31 00:57:58.582318 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:57:58.582130 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8rzb\" (UniqueName: \"kubernetes.io/projected/24d0a0fc-07f1-4f6b-b7c8-8a89a97d5548-kube-api-access-x8rzb\") pod \"maas-api-revoke-sub-f85ed32c-mz22p\" (UID: \"24d0a0fc-07f1-4f6b-b7c8-8a89a97d5548\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f85ed32c-mz22p" Oct 31 00:57:58.582318 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:57:58.582186 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24d0a0fc-07f1-4f6b-b7c8-8a89a97d5548-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f85ed32c-mz22p\" (UID: \"24d0a0fc-07f1-4f6b-b7c8-8a89a97d5548\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f85ed32c-mz22p" Oct 31 00:57:58.582862 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:57:58.582838 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24d0a0fc-07f1-4f6b-b7c8-8a89a97d5548-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f85ed32c-mz22p\" (UID: \"24d0a0fc-07f1-4f6b-b7c8-8a89a97d5548\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f85ed32c-mz22p" Oct 31 00:57:58.584621 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:57:58.584602 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24d0a0fc-07f1-4f6b-b7c8-8a89a97d5548-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f85ed32c-mz22p\" (UID: \"24d0a0fc-07f1-4f6b-b7c8-8a89a97d5548\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f85ed32c-mz22p" Oct 31 00:57:58.645527 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:57:58.645452 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8rzb\" (UniqueName: \"kubernetes.io/projected/24d0a0fc-07f1-4f6b-b7c8-8a89a97d5548-kube-api-access-x8rzb\") pod \"maas-api-revoke-sub-f85ed32c-mz22p\" (UID: \"24d0a0fc-07f1-4f6b-b7c8-8a89a97d5548\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f85ed32c-mz22p" Oct 31 00:57:58.709769 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:57:58.709731 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f85ed32c-mz22p" Oct 31 00:57:59.086158 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:57:59.086111 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24d0a0fc_07f1_4f6b_b7c8_8a89a97d5548.slice/crio-f9ad2f70eb6c4dd41d56f9341821fbce14ab2961bffc2a98a72df0fabb939911 WatchSource:0}: Error finding container f9ad2f70eb6c4dd41d56f9341821fbce14ab2961bffc2a98a72df0fabb939911: Status 404 returned error can't find the container with id f9ad2f70eb6c4dd41d56f9341821fbce14ab2961bffc2a98a72df0fabb939911 Oct 31 00:57:59.089515 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:57:59.089490 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f85ed32c-mz22p"] Oct 31 00:57:59.944154 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:57:59.944119 5222 generic.go:356] "Generic (PLEG): container finished" podID="24d0a0fc-07f1-4f6b-b7c8-8a89a97d5548" containerID="898a2269efdfc6a79c62282b99124defed8473fb32c16488971a9833fedafdcd" exitCode=0 Oct 31 00:57:59.944617 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:57:59.944202 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f85ed32c-mz22p" event={"ID":"24d0a0fc-07f1-4f6b-b7c8-8a89a97d5548","Type":"ContainerDied","Data":"898a2269efdfc6a79c62282b99124defed8473fb32c16488971a9833fedafdcd"} Oct 31 00:57:59.944617 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:57:59.944249 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f85ed32c-mz22p" event={"ID":"24d0a0fc-07f1-4f6b-b7c8-8a89a97d5548","Type":"ContainerStarted","Data":"f9ad2f70eb6c4dd41d56f9341821fbce14ab2961bffc2a98a72df0fabb939911"} Oct 31 00:58:01.094429 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:01.094403 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f85ed32c-mz22p" Oct 31 00:58:01.110469 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:01.110437 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24d0a0fc-07f1-4f6b-b7c8-8a89a97d5548-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24d0a0fc-07f1-4f6b-b7c8-8a89a97d5548-maas-api-openshift-ca\") pod \"24d0a0fc-07f1-4f6b-b7c8-8a89a97d5548\" (UID: \"24d0a0fc-07f1-4f6b-b7c8-8a89a97d5548\") " Oct 31 00:58:01.110639 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:01.110510 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24d0a0fc-07f1-4f6b-b7c8-8a89a97d5548-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24d0a0fc-07f1-4f6b-b7c8-8a89a97d5548-maas-api-xks-ca\") pod \"24d0a0fc-07f1-4f6b-b7c8-8a89a97d5548\" (UID: \"24d0a0fc-07f1-4f6b-b7c8-8a89a97d5548\") " Oct 31 00:58:01.110639 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:01.110533 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24d0a0fc-07f1-4f6b-b7c8-8a89a97d5548-kube-api-access-x8rzb\" (UniqueName: \"kubernetes.io/projected/24d0a0fc-07f1-4f6b-b7c8-8a89a97d5548-kube-api-access-x8rzb\") pod \"24d0a0fc-07f1-4f6b-b7c8-8a89a97d5548\" (UID: \"24d0a0fc-07f1-4f6b-b7c8-8a89a97d5548\") " Oct 31 00:58:01.110868 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:01.110842 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24d0a0fc-07f1-4f6b-b7c8-8a89a97d5548-maas-api-openshift-ca" pod "24d0a0fc-07f1-4f6b-b7c8-8a89a97d5548" (UID: "24d0a0fc-07f1-4f6b-b7c8-8a89a97d5548"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:58:01.112801 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:01.112768 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24d0a0fc-07f1-4f6b-b7c8-8a89a97d5548-maas-api-xks-ca" pod "24d0a0fc-07f1-4f6b-b7c8-8a89a97d5548" (UID: "24d0a0fc-07f1-4f6b-b7c8-8a89a97d5548"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:58:01.112921 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:01.112835 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24d0a0fc-07f1-4f6b-b7c8-8a89a97d5548-kube-api-access-x8rzb" pod "24d0a0fc-07f1-4f6b-b7c8-8a89a97d5548" (UID: "24d0a0fc-07f1-4f6b-b7c8-8a89a97d5548"). InnerVolumeSpecName "kube-api-access-x8rzb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:58:01.212691 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:01.212613 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24d0a0fc-07f1-4f6b-b7c8-8a89a97d5548-maas-api-openshift-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:58:01.212691 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:01.212654 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24d0a0fc-07f1-4f6b-b7c8-8a89a97d5548-maas-api-xks-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:58:01.212691 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:01.212670 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8rzb\" (UniqueName: \"kubernetes.io/projected/24d0a0fc-07f1-4f6b-b7c8-8a89a97d5548-kube-api-access-x8rzb\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:58:01.956840 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:01.956759 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f85ed32c-mz22p" Oct 31 00:58:01.956840 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:01.956804 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f85ed32c-mz22p" event={"ID":"24d0a0fc-07f1-4f6b-b7c8-8a89a97d5548","Type":"ContainerDied","Data":"f9ad2f70eb6c4dd41d56f9341821fbce14ab2961bffc2a98a72df0fabb939911"} Oct 31 00:58:01.957037 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:01.956850 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f9ad2f70eb6c4dd41d56f9341821fbce14ab2961bffc2a98a72df0fabb939911" Oct 31 00:58:05.889407 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:58:05.889338 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 00:58:10.053077 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:10.053036 5222 scope.go:122] "RemoveContainer" containerID="7485db9836ea874ebdab72d00635eb84682a4926b031c8e572201471d1ee085a" Oct 31 00:58:10.063842 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:10.063817 5222 scope.go:122] "RemoveContainer" containerID="778f9e0a89a273cdbacdd720189692f13d142a178a8874b6fd6e6fc2106bc63e" Oct 31 00:58:10.076591 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:10.076568 5222 scope.go:122] "RemoveContainer" containerID="884319de0e2dfa6a978a236b9ce62d0a0b0cf94856a63f687ab766a5563fc1d6" Oct 31 00:58:10.566438 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:10.566393 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-4ab8e5/test-model-4ab8e5-kserve-78dd57fd69-h64c4"] Oct 31 00:58:11.618419 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:11.618382 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6764d1c-ljhrm"] Oct 31 00:58:11.618882 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:11.618867 5222 cpu_manager.go:414] "containerMap: removing container" podUID="24d0a0fc-07f1-4f6b-b7c8-8a89a97d5548" containerName="revoke-keys" Oct 31 00:58:11.618928 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:11.618884 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24d0a0fc-07f1-4f6b-b7c8-8a89a97d5548" containerName="revoke-keys" Oct 31 00:58:11.618979 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:11.618970 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="24d0a0fc-07f1-4f6b-b7c8-8a89a97d5548" containerName="revoke-keys" Oct 31 00:58:11.623905 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:11.623882 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6764d1c-ljhrm" Oct 31 00:58:11.676594 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:11.676557 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6764d1c-ljhrm"] Oct 31 00:58:11.724998 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:11.724967 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:58:11.725200 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:11.725030 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfz9z\"" Oct 31 00:58:11.729612 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:11.729585 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07ace044-4dbc-43cd-a259-81de418b36ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f6764d1c-ljhrm\" (UID: \"07ace044-4dbc-43cd-a259-81de418b36ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6764d1c-ljhrm" Oct 31 00:58:11.729720 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:11.729646 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07ace044-4dbc-43cd-a259-81de418b36ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f6764d1c-ljhrm\" (UID: \"07ace044-4dbc-43cd-a259-81de418b36ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6764d1c-ljhrm" Oct 31 00:58:11.729964 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:11.729948 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tpsf\" (UniqueName: \"kubernetes.io/projected/07ace044-4dbc-43cd-a259-81de418b36ab-kube-api-access-7tpsf\") pod \"maas-api-revoke-sub-f6764d1c-ljhrm\" (UID: \"07ace044-4dbc-43cd-a259-81de418b36ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6764d1c-ljhrm" Oct 31 00:58:11.776242 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:11.776211 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:58:11.776460 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:11.776292 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:58:11.831489 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:11.831450 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07ace044-4dbc-43cd-a259-81de418b36ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f6764d1c-ljhrm\" (UID: \"07ace044-4dbc-43cd-a259-81de418b36ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6764d1c-ljhrm" Oct 31 00:58:11.831688 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:11.831529 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07ace044-4dbc-43cd-a259-81de418b36ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f6764d1c-ljhrm\" (UID: \"07ace044-4dbc-43cd-a259-81de418b36ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6764d1c-ljhrm" Oct 31 00:58:11.831688 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:11.831594 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tpsf\" (UniqueName: \"kubernetes.io/projected/07ace044-4dbc-43cd-a259-81de418b36ab-kube-api-access-7tpsf\") pod \"maas-api-revoke-sub-f6764d1c-ljhrm\" (UID: \"07ace044-4dbc-43cd-a259-81de418b36ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6764d1c-ljhrm" Oct 31 00:58:11.832269 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:11.832246 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07ace044-4dbc-43cd-a259-81de418b36ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f6764d1c-ljhrm\" (UID: \"07ace044-4dbc-43cd-a259-81de418b36ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6764d1c-ljhrm" Oct 31 00:58:11.834195 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:11.834171 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07ace044-4dbc-43cd-a259-81de418b36ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f6764d1c-ljhrm\" (UID: \"07ace044-4dbc-43cd-a259-81de418b36ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6764d1c-ljhrm" Oct 31 00:58:11.894582 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:11.894504 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tpsf\" (UniqueName: \"kubernetes.io/projected/07ace044-4dbc-43cd-a259-81de418b36ab-kube-api-access-7tpsf\") pod \"maas-api-revoke-sub-f6764d1c-ljhrm\" (UID: \"07ace044-4dbc-43cd-a259-81de418b36ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6764d1c-ljhrm" Oct 31 00:58:11.937520 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:11.937479 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6764d1c-ljhrm" Oct 31 00:58:12.358581 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:58:12.358553 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07ace044_4dbc_43cd_a259_81de418b36ab.slice/crio-4b5510a5130d0ee1f52305006704906edcbbc6cd4185b9eda2aaa1184a301c05 WatchSource:0}: Error finding container 4b5510a5130d0ee1f52305006704906edcbbc6cd4185b9eda2aaa1184a301c05: Status 404 returned error can't find the container with id 4b5510a5130d0ee1f52305006704906edcbbc6cd4185b9eda2aaa1184a301c05 Oct 31 00:58:12.360970 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:12.360942 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6764d1c-ljhrm"] Oct 31 00:58:13.030974 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:13.030930 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6764d1c-ljhrm" event={"ID":"07ace044-4dbc-43cd-a259-81de418b36ab","Type":"ContainerStarted","Data":"688574db526f610ab2349927ae41443fb03891d025ef268f8544a109c4099ee1"} Oct 31 00:58:13.031506 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:13.030988 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6764d1c-ljhrm" event={"ID":"07ace044-4dbc-43cd-a259-81de418b36ab","Type":"ContainerStarted","Data":"4b5510a5130d0ee1f52305006704906edcbbc6cd4185b9eda2aaa1184a301c05"} Oct 31 00:58:13.151926 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:13.151870 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6764d1c-ljhrm" podStartSLOduration=2.151856026 podStartE2EDuration="2.151856026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:58:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:58:13.149850367 +0000 UTC m=+2585.621020118" watchObservedRunningTime="2026-10-31 00:58:13.151856026 +0000 UTC m=+2585.623025989" Oct 31 00:58:14.037961 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:14.037924 5222 generic.go:356] "Generic (PLEG): container finished" podID="07ace044-4dbc-43cd-a259-81de418b36ab" containerID="688574db526f610ab2349927ae41443fb03891d025ef268f8544a109c4099ee1" exitCode=0 Oct 31 00:58:14.038423 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:14.038011 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6764d1c-ljhrm" event={"ID":"07ace044-4dbc-43cd-a259-81de418b36ab","Type":"ContainerDied","Data":"688574db526f610ab2349927ae41443fb03891d025ef268f8544a109c4099ee1"} Oct 31 00:58:15.194595 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:15.194571 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6764d1c-ljhrm" Oct 31 00:58:15.269315 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:15.269270 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07ace044-4dbc-43cd-a259-81de418b36ab-kube-api-access-7tpsf\" (UniqueName: \"kubernetes.io/projected/07ace044-4dbc-43cd-a259-81de418b36ab-kube-api-access-7tpsf\") pod \"07ace044-4dbc-43cd-a259-81de418b36ab\" (UID: \"07ace044-4dbc-43cd-a259-81de418b36ab\") " Oct 31 00:58:15.269537 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:15.269425 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07ace044-4dbc-43cd-a259-81de418b36ab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07ace044-4dbc-43cd-a259-81de418b36ab-maas-api-xks-ca\") pod \"07ace044-4dbc-43cd-a259-81de418b36ab\" (UID: \"07ace044-4dbc-43cd-a259-81de418b36ab\") " Oct 31 00:58:15.269537 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:15.269497 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07ace044-4dbc-43cd-a259-81de418b36ab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07ace044-4dbc-43cd-a259-81de418b36ab-maas-api-openshift-ca\") pod \"07ace044-4dbc-43cd-a259-81de418b36ab\" (UID: \"07ace044-4dbc-43cd-a259-81de418b36ab\") " Oct 31 00:58:15.269892 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:15.269849 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07ace044-4dbc-43cd-a259-81de418b36ab-maas-api-openshift-ca" pod "07ace044-4dbc-43cd-a259-81de418b36ab" (UID: "07ace044-4dbc-43cd-a259-81de418b36ab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:58:15.271644 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:15.271623 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07ace044-4dbc-43cd-a259-81de418b36ab-maas-api-xks-ca" pod "07ace044-4dbc-43cd-a259-81de418b36ab" (UID: "07ace044-4dbc-43cd-a259-81de418b36ab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:58:15.271747 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:15.271683 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07ace044-4dbc-43cd-a259-81de418b36ab-kube-api-access-7tpsf" pod "07ace044-4dbc-43cd-a259-81de418b36ab" (UID: "07ace044-4dbc-43cd-a259-81de418b36ab"). InnerVolumeSpecName "kube-api-access-7tpsf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:58:15.371170 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:15.371086 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7tpsf\" (UniqueName: \"kubernetes.io/projected/07ace044-4dbc-43cd-a259-81de418b36ab-kube-api-access-7tpsf\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:58:15.371170 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:15.371120 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07ace044-4dbc-43cd-a259-81de418b36ab-maas-api-xks-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:58:15.371170 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:15.371130 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07ace044-4dbc-43cd-a259-81de418b36ab-maas-api-openshift-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:58:16.051909 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:16.051881 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6764d1c-ljhrm" Oct 31 00:58:16.052122 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:16.051881 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6764d1c-ljhrm" event={"ID":"07ace044-4dbc-43cd-a259-81de418b36ab","Type":"ContainerDied","Data":"4b5510a5130d0ee1f52305006704906edcbbc6cd4185b9eda2aaa1184a301c05"} Oct 31 00:58:16.052122 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:16.052005 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b5510a5130d0ee1f52305006704906edcbbc6cd4185b9eda2aaa1184a301c05" Oct 31 00:58:16.933758 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:16.933719 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-79tk8"] Oct 31 00:58:25.588062 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:25.588013 5222 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-4ab8e5/test-model-4ab8e5-kserve-78dd57fd69-h64c4" podUID="c05360e3-dc37-4ee8-8589-9d32cb58a814" containerName="main" containerID="cri-o://d5917ea2f01324648c6e288197658969633db19fc68c4e24a8eaaa08d3d61a65" gracePeriod=45 Oct 31 00:58:25.850898 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:25.850833 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-4ab8e5/test-model-4ab8e5-kserve-78dd57fd69-h64c4" Oct 31 00:58:25.990276 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:25.990236 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c05360e3-dc37-4ee8-8589-9d32cb58a814-kube-api-access-k9krk\" (UniqueName: \"kubernetes.io/projected/c05360e3-dc37-4ee8-8589-9d32cb58a814-kube-api-access-k9krk\") pod \"c05360e3-dc37-4ee8-8589-9d32cb58a814\" (UID: \"c05360e3-dc37-4ee8-8589-9d32cb58a814\") " Oct 31 00:58:25.990519 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:25.990307 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c05360e3-dc37-4ee8-8589-9d32cb58a814-home\" (UniqueName: \"kubernetes.io/empty-dir/c05360e3-dc37-4ee8-8589-9d32cb58a814-home\") pod \"c05360e3-dc37-4ee8-8589-9d32cb58a814\" (UID: \"c05360e3-dc37-4ee8-8589-9d32cb58a814\") " Oct 31 00:58:25.990519 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:25.990335 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c05360e3-dc37-4ee8-8589-9d32cb58a814-tls-certs\" (UniqueName: \"kubernetes.io/secret/c05360e3-dc37-4ee8-8589-9d32cb58a814-tls-certs\") pod \"c05360e3-dc37-4ee8-8589-9d32cb58a814\" (UID: \"c05360e3-dc37-4ee8-8589-9d32cb58a814\") " Oct 31 00:58:25.990519 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:25.990430 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c05360e3-dc37-4ee8-8589-9d32cb58a814-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c05360e3-dc37-4ee8-8589-9d32cb58a814-tmp-dir\") pod \"c05360e3-dc37-4ee8-8589-9d32cb58a814\" (UID: \"c05360e3-dc37-4ee8-8589-9d32cb58a814\") " Oct 31 00:58:25.990519 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:25.990454 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c05360e3-dc37-4ee8-8589-9d32cb58a814-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c05360e3-dc37-4ee8-8589-9d32cb58a814-model-cache\") pod \"c05360e3-dc37-4ee8-8589-9d32cb58a814\" (UID: \"c05360e3-dc37-4ee8-8589-9d32cb58a814\") " Oct 31 00:58:25.990519 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:25.990491 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c05360e3-dc37-4ee8-8589-9d32cb58a814-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c05360e3-dc37-4ee8-8589-9d32cb58a814-dshm\") pod \"c05360e3-dc37-4ee8-8589-9d32cb58a814\" (UID: \"c05360e3-dc37-4ee8-8589-9d32cb58a814\") " Oct 31 00:58:25.990817 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:25.990640 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c05360e3-dc37-4ee8-8589-9d32cb58a814-home" pod "c05360e3-dc37-4ee8-8589-9d32cb58a814" (UID: "c05360e3-dc37-4ee8-8589-9d32cb58a814"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:58:25.990884 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:25.990826 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c05360e3-dc37-4ee8-8589-9d32cb58a814-tmp-dir" pod "c05360e3-dc37-4ee8-8589-9d32cb58a814" (UID: "c05360e3-dc37-4ee8-8589-9d32cb58a814"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:58:25.990997 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:25.990979 5222 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c05360e3-dc37-4ee8-8589-9d32cb58a814-tmp-dir\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:58:25.991173 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:25.991162 5222 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c05360e3-dc37-4ee8-8589-9d32cb58a814-home\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:58:25.991288 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:25.990977 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c05360e3-dc37-4ee8-8589-9d32cb58a814-model-cache" pod "c05360e3-dc37-4ee8-8589-9d32cb58a814" (UID: "c05360e3-dc37-4ee8-8589-9d32cb58a814"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:58:25.993063 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:25.993041 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c05360e3-dc37-4ee8-8589-9d32cb58a814-dshm" pod "c05360e3-dc37-4ee8-8589-9d32cb58a814" (UID: "c05360e3-dc37-4ee8-8589-9d32cb58a814"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:58:25.993308 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:25.993276 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c05360e3-dc37-4ee8-8589-9d32cb58a814-tls-certs" pod "c05360e3-dc37-4ee8-8589-9d32cb58a814" (UID: "c05360e3-dc37-4ee8-8589-9d32cb58a814"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:58:25.993450 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:25.993322 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c05360e3-dc37-4ee8-8589-9d32cb58a814-kube-api-access-k9krk" pod "c05360e3-dc37-4ee8-8589-9d32cb58a814" (UID: "c05360e3-dc37-4ee8-8589-9d32cb58a814"). InnerVolumeSpecName "kube-api-access-k9krk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:58:26.093127 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:26.092531 5222 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c05360e3-dc37-4ee8-8589-9d32cb58a814-model-cache\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:58:26.093127 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:26.092570 5222 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c05360e3-dc37-4ee8-8589-9d32cb58a814-dshm\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:58:26.093127 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:26.092587 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9krk\" (UniqueName: \"kubernetes.io/projected/c05360e3-dc37-4ee8-8589-9d32cb58a814-kube-api-access-k9krk\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:58:26.093127 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:26.092602 5222 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c05360e3-dc37-4ee8-8589-9d32cb58a814-tls-certs\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:58:26.124829 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:26.124740 5222 generic.go:356] "Generic (PLEG): container finished" podID="c05360e3-dc37-4ee8-8589-9d32cb58a814" containerID="d5917ea2f01324648c6e288197658969633db19fc68c4e24a8eaaa08d3d61a65" exitCode=0 Oct 31 00:58:26.124829 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:26.124785 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-4ab8e5/test-model-4ab8e5-kserve-78dd57fd69-h64c4" Oct 31 00:58:26.125048 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:26.124786 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-4ab8e5/test-model-4ab8e5-kserve-78dd57fd69-h64c4" event={"ID":"c05360e3-dc37-4ee8-8589-9d32cb58a814","Type":"ContainerDied","Data":"d5917ea2f01324648c6e288197658969633db19fc68c4e24a8eaaa08d3d61a65"} Oct 31 00:58:26.125048 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:26.124905 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-4ab8e5/test-model-4ab8e5-kserve-78dd57fd69-h64c4" event={"ID":"c05360e3-dc37-4ee8-8589-9d32cb58a814","Type":"ContainerDied","Data":"bd91276dfecd28a8b6f7d78650a22273023d752d2d2608eb0d3cf12a261aecfe"} Oct 31 00:58:26.125048 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:26.124930 5222 scope.go:122] "RemoveContainer" containerID="d5917ea2f01324648c6e288197658969633db19fc68c4e24a8eaaa08d3d61a65" Oct 31 00:58:26.137891 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:26.137870 5222 scope.go:122] "RemoveContainer" containerID="d5917ea2f01324648c6e288197658969633db19fc68c4e24a8eaaa08d3d61a65" Oct 31 00:58:26.138219 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:58:26.138196 5222 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5917ea2f01324648c6e288197658969633db19fc68c4e24a8eaaa08d3d61a65\": container with ID starting with d5917ea2f01324648c6e288197658969633db19fc68c4e24a8eaaa08d3d61a65 not found: ID does not exist" containerID="d5917ea2f01324648c6e288197658969633db19fc68c4e24a8eaaa08d3d61a65" Oct 31 00:58:26.138291 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:26.138232 5222 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5917ea2f01324648c6e288197658969633db19fc68c4e24a8eaaa08d3d61a65"} err="failed to get container status \"d5917ea2f01324648c6e288197658969633db19fc68c4e24a8eaaa08d3d61a65\": rpc error: code = NotFound desc = could not find container \"d5917ea2f01324648c6e288197658969633db19fc68c4e24a8eaaa08d3d61a65\": container with ID starting with d5917ea2f01324648c6e288197658969633db19fc68c4e24a8eaaa08d3d61a65 not found: ID does not exist" Oct 31 00:58:26.296175 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:26.296140 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-4ab8e5/test-model-4ab8e5-kserve-78dd57fd69-h64c4"] Oct 31 00:58:26.299966 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:26.299941 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-4ab8e5/test-model-4ab8e5-kserve-78dd57fd69-h64c4"] Oct 31 00:58:27.654963 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:27.654928 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c05360e3-dc37-4ee8-8589-9d32cb58a814" path="/var/lib/kubelet/pods/c05360e3-dc37-4ee8-8589-9d32cb58a814/volumes" Oct 31 00:58:35.890014 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:58:35.889973 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 00:58:36.131757 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:36.131727 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-79tk8"] Oct 31 00:58:37.059820 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:37.059784 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-bc2453/test-model-bc2453-kserve-d4955dd5c-59x8p"] Oct 31 00:58:37.646283 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:37.646243 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4ab8e5-b7m9x"] Oct 31 00:58:37.646729 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:37.646714 5222 cpu_manager.go:414] "containerMap: removing container" podUID="07ace044-4dbc-43cd-a259-81de418b36ab" containerName="revoke-keys" Oct 31 00:58:37.646791 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:37.646730 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07ace044-4dbc-43cd-a259-81de418b36ab" containerName="revoke-keys" Oct 31 00:58:37.646791 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:37.646773 5222 cpu_manager.go:414] "containerMap: removing container" podUID="c05360e3-dc37-4ee8-8589-9d32cb58a814" containerName="main" Oct 31 00:58:37.646791 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:37.646779 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c05360e3-dc37-4ee8-8589-9d32cb58a814" containerName="main" Oct 31 00:58:37.646903 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:37.646854 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="c05360e3-dc37-4ee8-8589-9d32cb58a814" containerName="main" Oct 31 00:58:37.646903 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:37.646868 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="07ace044-4dbc-43cd-a259-81de418b36ab" containerName="revoke-keys" Oct 31 00:58:37.652208 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:37.652179 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4ab8e5-b7m9x" Oct 31 00:58:37.705811 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:37.705770 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4ab8e5-b7m9x"] Oct 31 00:58:37.711486 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:37.711442 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:58:37.711486 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:37.711468 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:58:37.711486 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:37.711468 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:58:37.744247 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:37.744212 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3952d123-2319-48d7-b9fd-23e5c1eceb00-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-4ab8e5-b7m9x\" (UID: \"3952d123-2319-48d7-b9fd-23e5c1eceb00\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4ab8e5-b7m9x" Oct 31 00:58:37.744462 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:37.744285 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxf9m\" (UniqueName: \"kubernetes.io/projected/3952d123-2319-48d7-b9fd-23e5c1eceb00-kube-api-access-cxf9m\") pod \"maas-api-revoke-keys-e2e-inf-a-4ab8e5-b7m9x\" (UID: \"3952d123-2319-48d7-b9fd-23e5c1eceb00\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4ab8e5-b7m9x" Oct 31 00:58:37.744509 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:37.744472 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3952d123-2319-48d7-b9fd-23e5c1eceb00-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-4ab8e5-b7m9x\" (UID: \"3952d123-2319-48d7-b9fd-23e5c1eceb00\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4ab8e5-b7m9x" Oct 31 00:58:37.753581 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:37.753552 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfz9z\"" Oct 31 00:58:37.845896 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:37.845850 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxf9m\" (UniqueName: \"kubernetes.io/projected/3952d123-2319-48d7-b9fd-23e5c1eceb00-kube-api-access-cxf9m\") pod \"maas-api-revoke-keys-e2e-inf-a-4ab8e5-b7m9x\" (UID: \"3952d123-2319-48d7-b9fd-23e5c1eceb00\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4ab8e5-b7m9x" Oct 31 00:58:37.846128 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:37.845920 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3952d123-2319-48d7-b9fd-23e5c1eceb00-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-4ab8e5-b7m9x\" (UID: \"3952d123-2319-48d7-b9fd-23e5c1eceb00\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4ab8e5-b7m9x" Oct 31 00:58:37.846128 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:37.846026 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3952d123-2319-48d7-b9fd-23e5c1eceb00-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-4ab8e5-b7m9x\" (UID: \"3952d123-2319-48d7-b9fd-23e5c1eceb00\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4ab8e5-b7m9x" Oct 31 00:58:37.846644 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:37.846619 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3952d123-2319-48d7-b9fd-23e5c1eceb00-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-4ab8e5-b7m9x\" (UID: \"3952d123-2319-48d7-b9fd-23e5c1eceb00\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4ab8e5-b7m9x" Oct 31 00:58:37.848643 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:37.848617 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3952d123-2319-48d7-b9fd-23e5c1eceb00-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-4ab8e5-b7m9x\" (UID: \"3952d123-2319-48d7-b9fd-23e5c1eceb00\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4ab8e5-b7m9x" Oct 31 00:58:37.909592 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:37.909512 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxf9m\" (UniqueName: \"kubernetes.io/projected/3952d123-2319-48d7-b9fd-23e5c1eceb00-kube-api-access-cxf9m\") pod \"maas-api-revoke-keys-e2e-inf-a-4ab8e5-b7m9x\" (UID: \"3952d123-2319-48d7-b9fd-23e5c1eceb00\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4ab8e5-b7m9x" Oct 31 00:58:37.965430 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:37.965365 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4ab8e5-b7m9x" Oct 31 00:58:38.349160 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:58:38.349124 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3952d123_2319_48d7_b9fd_23e5c1eceb00.slice/crio-2d869b70058171ce0a04301239b2b8983c4af3a84f9680164ed0496865188565 WatchSource:0}: Error finding container 2d869b70058171ce0a04301239b2b8983c4af3a84f9680164ed0496865188565: Status 404 returned error can't find the container with id 2d869b70058171ce0a04301239b2b8983c4af3a84f9680164ed0496865188565 Oct 31 00:58:38.351430 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:38.351405 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4ab8e5-b7m9x"] Oct 31 00:58:39.223486 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:39.223451 5222 generic.go:356] "Generic (PLEG): container finished" podID="3952d123-2319-48d7-b9fd-23e5c1eceb00" containerID="73dbb9cb87b4b3eedb8a1e7f218c3c5f28c69f635a38d966f1acbf5eb83d3326" exitCode=0 Oct 31 00:58:39.223694 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:39.223537 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4ab8e5-b7m9x" event={"ID":"3952d123-2319-48d7-b9fd-23e5c1eceb00","Type":"ContainerDied","Data":"73dbb9cb87b4b3eedb8a1e7f218c3c5f28c69f635a38d966f1acbf5eb83d3326"} Oct 31 00:58:39.223694 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:39.223593 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4ab8e5-b7m9x" event={"ID":"3952d123-2319-48d7-b9fd-23e5c1eceb00","Type":"ContainerStarted","Data":"2d869b70058171ce0a04301239b2b8983c4af3a84f9680164ed0496865188565"} Oct 31 00:58:40.379046 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:40.379010 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4ab8e5-b7m9x" Oct 31 00:58:40.473650 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:40.473614 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3952d123-2319-48d7-b9fd-23e5c1eceb00-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3952d123-2319-48d7-b9fd-23e5c1eceb00-maas-api-openshift-ca\") pod \"3952d123-2319-48d7-b9fd-23e5c1eceb00\" (UID: \"3952d123-2319-48d7-b9fd-23e5c1eceb00\") " Oct 31 00:58:40.473650 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:40.473658 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3952d123-2319-48d7-b9fd-23e5c1eceb00-kube-api-access-cxf9m\" (UniqueName: \"kubernetes.io/projected/3952d123-2319-48d7-b9fd-23e5c1eceb00-kube-api-access-cxf9m\") pod \"3952d123-2319-48d7-b9fd-23e5c1eceb00\" (UID: \"3952d123-2319-48d7-b9fd-23e5c1eceb00\") " Oct 31 00:58:40.473926 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:40.473712 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3952d123-2319-48d7-b9fd-23e5c1eceb00-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3952d123-2319-48d7-b9fd-23e5c1eceb00-maas-api-xks-ca\") pod \"3952d123-2319-48d7-b9fd-23e5c1eceb00\" (UID: \"3952d123-2319-48d7-b9fd-23e5c1eceb00\") " Oct 31 00:58:40.474019 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:40.473994 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3952d123-2319-48d7-b9fd-23e5c1eceb00-maas-api-openshift-ca" pod "3952d123-2319-48d7-b9fd-23e5c1eceb00" (UID: "3952d123-2319-48d7-b9fd-23e5c1eceb00"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:58:40.475996 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:40.475967 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3952d123-2319-48d7-b9fd-23e5c1eceb00-maas-api-xks-ca" pod "3952d123-2319-48d7-b9fd-23e5c1eceb00" (UID: "3952d123-2319-48d7-b9fd-23e5c1eceb00"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:58:40.476201 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:40.476185 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3952d123-2319-48d7-b9fd-23e5c1eceb00-kube-api-access-cxf9m" pod "3952d123-2319-48d7-b9fd-23e5c1eceb00" (UID: "3952d123-2319-48d7-b9fd-23e5c1eceb00"). InnerVolumeSpecName "kube-api-access-cxf9m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:58:40.574968 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:40.574921 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3952d123-2319-48d7-b9fd-23e5c1eceb00-maas-api-openshift-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:58:40.574968 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:40.574960 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxf9m\" (UniqueName: \"kubernetes.io/projected/3952d123-2319-48d7-b9fd-23e5c1eceb00-kube-api-access-cxf9m\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:58:40.574968 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:40.574975 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3952d123-2319-48d7-b9fd-23e5c1eceb00-maas-api-xks-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:58:41.237327 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:41.237298 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4ab8e5-b7m9x" Oct 31 00:58:41.237532 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:41.237330 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4ab8e5-b7m9x" event={"ID":"3952d123-2319-48d7-b9fd-23e5c1eceb00","Type":"ContainerDied","Data":"2d869b70058171ce0a04301239b2b8983c4af3a84f9680164ed0496865188565"} Oct 31 00:58:41.237532 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:41.237406 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2d869b70058171ce0a04301239b2b8983c4af3a84f9680164ed0496865188565" Oct 31 00:58:49.128733 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:49.128642 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4ab8e5-b77879c55-qptzw"] Oct 31 00:58:49.240359 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:49.240328 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4ab8e5-79cc98db87-r95fp"] Oct 31 00:58:50.910969 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:50.910935 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-79tk8"] Oct 31 00:58:52.080772 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:52.080723 5222 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-bc2453/test-model-bc2453-kserve-d4955dd5c-59x8p" podUID="74eae8f3-ee20-417d-ada2-d300274b2038" containerName="main" containerID="cri-o://f45c11c0cf2dc1d65b1e9e07d64670ae52595e09b81c462ec56f180594c0899f" gracePeriod=45 Oct 31 00:58:52.306130 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:52.306097 5222 generic.go:356] "Generic (PLEG): container finished" podID="74eae8f3-ee20-417d-ada2-d300274b2038" containerID="f45c11c0cf2dc1d65b1e9e07d64670ae52595e09b81c462ec56f180594c0899f" exitCode=0 Oct 31 00:58:52.306335 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:52.306169 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-bc2453/test-model-bc2453-kserve-d4955dd5c-59x8p" event={"ID":"74eae8f3-ee20-417d-ada2-d300274b2038","Type":"ContainerDied","Data":"f45c11c0cf2dc1d65b1e9e07d64670ae52595e09b81c462ec56f180594c0899f"} Oct 31 00:58:52.347118 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:52.347055 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-bc2453/test-model-bc2453-kserve-d4955dd5c-59x8p" Oct 31 00:58:52.495829 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:52.495795 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74eae8f3-ee20-417d-ada2-d300274b2038-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74eae8f3-ee20-417d-ada2-d300274b2038-tmp-dir\") pod \"74eae8f3-ee20-417d-ada2-d300274b2038\" (UID: \"74eae8f3-ee20-417d-ada2-d300274b2038\") " Oct 31 00:58:52.496018 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:52.495856 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74eae8f3-ee20-417d-ada2-d300274b2038-home\" (UniqueName: \"kubernetes.io/empty-dir/74eae8f3-ee20-417d-ada2-d300274b2038-home\") pod \"74eae8f3-ee20-417d-ada2-d300274b2038\" (UID: \"74eae8f3-ee20-417d-ada2-d300274b2038\") " Oct 31 00:58:52.496018 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:52.495931 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74eae8f3-ee20-417d-ada2-d300274b2038-tls-certs\" (UniqueName: \"kubernetes.io/secret/74eae8f3-ee20-417d-ada2-d300274b2038-tls-certs\") pod \"74eae8f3-ee20-417d-ada2-d300274b2038\" (UID: \"74eae8f3-ee20-417d-ada2-d300274b2038\") " Oct 31 00:58:52.496018 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:52.496002 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74eae8f3-ee20-417d-ada2-d300274b2038-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74eae8f3-ee20-417d-ada2-d300274b2038-model-cache\") pod \"74eae8f3-ee20-417d-ada2-d300274b2038\" (UID: \"74eae8f3-ee20-417d-ada2-d300274b2038\") " Oct 31 00:58:52.496148 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:52.496074 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74eae8f3-ee20-417d-ada2-d300274b2038-kube-api-access-w6g7p\" (UniqueName: \"kubernetes.io/projected/74eae8f3-ee20-417d-ada2-d300274b2038-kube-api-access-w6g7p\") pod \"74eae8f3-ee20-417d-ada2-d300274b2038\" (UID: \"74eae8f3-ee20-417d-ada2-d300274b2038\") " Oct 31 00:58:52.496148 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:52.496111 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74eae8f3-ee20-417d-ada2-d300274b2038-tmp-dir" pod "74eae8f3-ee20-417d-ada2-d300274b2038" (UID: "74eae8f3-ee20-417d-ada2-d300274b2038"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:58:52.496257 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:52.496158 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74eae8f3-ee20-417d-ada2-d300274b2038-home" pod "74eae8f3-ee20-417d-ada2-d300274b2038" (UID: "74eae8f3-ee20-417d-ada2-d300274b2038"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:58:52.496257 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:52.496170 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74eae8f3-ee20-417d-ada2-d300274b2038-dshm\" (UniqueName: \"kubernetes.io/empty-dir/74eae8f3-ee20-417d-ada2-d300274b2038-dshm\") pod \"74eae8f3-ee20-417d-ada2-d300274b2038\" (UID: \"74eae8f3-ee20-417d-ada2-d300274b2038\") " Oct 31 00:58:52.496406 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:52.496275 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74eae8f3-ee20-417d-ada2-d300274b2038-model-cache" pod "74eae8f3-ee20-417d-ada2-d300274b2038" (UID: "74eae8f3-ee20-417d-ada2-d300274b2038"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:58:52.496597 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:52.496577 5222 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74eae8f3-ee20-417d-ada2-d300274b2038-model-cache\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:58:52.496597 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:52.496596 5222 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74eae8f3-ee20-417d-ada2-d300274b2038-tmp-dir\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:58:52.496936 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:52.496612 5222 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74eae8f3-ee20-417d-ada2-d300274b2038-home\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:58:52.498462 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:52.498435 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74eae8f3-ee20-417d-ada2-d300274b2038-tls-certs" pod "74eae8f3-ee20-417d-ada2-d300274b2038" (UID: "74eae8f3-ee20-417d-ada2-d300274b2038"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:58:52.498462 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:52.498440 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74eae8f3-ee20-417d-ada2-d300274b2038-dshm" pod "74eae8f3-ee20-417d-ada2-d300274b2038" (UID: "74eae8f3-ee20-417d-ada2-d300274b2038"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:58:52.498611 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:52.498539 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74eae8f3-ee20-417d-ada2-d300274b2038-kube-api-access-w6g7p" pod "74eae8f3-ee20-417d-ada2-d300274b2038" (UID: "74eae8f3-ee20-417d-ada2-d300274b2038"). InnerVolumeSpecName "kube-api-access-w6g7p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:58:52.598197 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:52.598111 5222 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74eae8f3-ee20-417d-ada2-d300274b2038-dshm\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:58:52.598197 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:52.598141 5222 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74eae8f3-ee20-417d-ada2-d300274b2038-tls-certs\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:58:52.598197 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:52.598153 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6g7p\" (UniqueName: \"kubernetes.io/projected/74eae8f3-ee20-417d-ada2-d300274b2038-kube-api-access-w6g7p\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:58:53.315269 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:53.315237 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-bc2453/test-model-bc2453-kserve-d4955dd5c-59x8p" Oct 31 00:58:53.315817 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:53.315227 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-bc2453/test-model-bc2453-kserve-d4955dd5c-59x8p" event={"ID":"74eae8f3-ee20-417d-ada2-d300274b2038","Type":"ContainerDied","Data":"f6af5b9e64c94e3927927209e16a28ee7ac8b40d0d235de2b083e8e21c5dea1f"} Oct 31 00:58:53.315817 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:53.315362 5222 scope.go:122] "RemoveContainer" containerID="f45c11c0cf2dc1d65b1e9e07d64670ae52595e09b81c462ec56f180594c0899f" Oct 31 00:58:53.490816 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:53.490780 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-bc2453/test-model-bc2453-kserve-d4955dd5c-59x8p"] Oct 31 00:58:53.542194 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:53.542157 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-bc2453/test-model-bc2453-kserve-d4955dd5c-59x8p"] Oct 31 00:58:53.656851 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:53.656771 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74eae8f3-ee20-417d-ada2-d300274b2038" path="/var/lib/kubelet/pods/74eae8f3-ee20-417d-ada2-d300274b2038/volumes" Oct 31 00:58:54.129149 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:54.129073 5222 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-4ab8e5-b77879c55-qptzw" podUID="81a85359-4e26-4ae9-8251-11e56381e3bd" containerName="payload-processing" containerID="cri-o://d52f3b5a282eff0303b6cbf6bb425d4ff66ce65fedff6b076dc6dab8aafdf9b8" gracePeriod=25 Oct 31 00:58:54.241651 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:58:54.241586 5222 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4ab8e5-79cc98db87-r95fp" podUID="3132fd63-bf3d-4386-a31c-3c2ff47efd20" containerName="payload-pre-processing" containerID="cri-o://3c703c647fd747d85b363db7d63fc973555d90660597e3d8c2d54dff6fa6890e" gracePeriod=25 Oct 31 00:59:01.085444 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:01.085399 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bef057b-gv7fm"] Oct 31 00:59:01.135795 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:01.135757 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bef057b-gv7fm"] Oct 31 00:59:01.654858 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:01.654807 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="602dd633-f02d-4891-a476-cc693443d0a9" path="/var/lib/kubelet/pods/602dd633-f02d-4891-a476-cc693443d0a9/volumes" Oct 31 00:59:02.767070 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:02.767025 5222 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4ab8e5-79cc98db87-r95fp" podUID="3132fd63-bf3d-4386-a31c-3c2ff47efd20" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:59:02.767070 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:02.767058 5222 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-4ab8e5-b77879c55-qptzw" podUID="81a85359-4e26-4ae9-8251-11e56381e3bd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:59:05.889900 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:59:05.889862 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 00:59:10.110101 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:10.110070 5222 scope.go:122] "RemoveContainer" containerID="401c36cad00d2004be52c4e07dfd34e09674bdebbf699456249b221926fb0470" Oct 31 00:59:10.119869 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:10.119846 5222 scope.go:122] "RemoveContainer" containerID="7918809807c83036acf97d6ec3bf7b7452a3bb45b9d8e9d828932c336a63d344" Oct 31 00:59:10.128841 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:10.128819 5222 scope.go:122] "RemoveContainer" containerID="40bf70fd874c54561704082986718d734650ca8c777c78094d3731aa7d590e50" Oct 31 00:59:10.137713 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:10.137695 5222 scope.go:122] "RemoveContainer" containerID="e34bc7b8fe27c5c92d02a809c201a47a18499a5cef9c7fceeae43a301adf5c17" Oct 31 00:59:12.767706 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:12.767655 5222 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4ab8e5-79cc98db87-r95fp" podUID="3132fd63-bf3d-4386-a31c-3c2ff47efd20" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:59:12.767706 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:12.767686 5222 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-4ab8e5-b77879c55-qptzw" podUID="81a85359-4e26-4ae9-8251-11e56381e3bd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:59:14.397522 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:14.397443 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-4ab8e5-b77879c55-qptzw" Oct 31 00:59:14.420991 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:14.420955 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81a85359-4e26-4ae9-8251-11e56381e3bd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81a85359-4e26-4ae9-8251-11e56381e3bd-plugins-config-volume\") pod \"81a85359-4e26-4ae9-8251-11e56381e3bd\" (UID: \"81a85359-4e26-4ae9-8251-11e56381e3bd\") " Oct 31 00:59:14.421152 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:14.421097 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81a85359-4e26-4ae9-8251-11e56381e3bd-kube-api-access-8t7mp\" (UniqueName: \"kubernetes.io/projected/81a85359-4e26-4ae9-8251-11e56381e3bd-kube-api-access-8t7mp\") pod \"81a85359-4e26-4ae9-8251-11e56381e3bd\" (UID: \"81a85359-4e26-4ae9-8251-11e56381e3bd\") " Oct 31 00:59:14.421651 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:14.421626 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81a85359-4e26-4ae9-8251-11e56381e3bd-plugins-config-volume" pod "81a85359-4e26-4ae9-8251-11e56381e3bd" (UID: "81a85359-4e26-4ae9-8251-11e56381e3bd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:59:14.423612 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:14.423587 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81a85359-4e26-4ae9-8251-11e56381e3bd-kube-api-access-8t7mp" pod "81a85359-4e26-4ae9-8251-11e56381e3bd" (UID: "81a85359-4e26-4ae9-8251-11e56381e3bd"). InnerVolumeSpecName "kube-api-access-8t7mp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:59:14.444242 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:14.444200 5222 generic.go:356] "Generic (PLEG): container finished" podID="81a85359-4e26-4ae9-8251-11e56381e3bd" containerID="d52f3b5a282eff0303b6cbf6bb425d4ff66ce65fedff6b076dc6dab8aafdf9b8" exitCode=0 Oct 31 00:59:14.444467 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:14.444247 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-4ab8e5-b77879c55-qptzw" Oct 31 00:59:14.444467 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:14.444282 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-4ab8e5-b77879c55-qptzw" event={"ID":"81a85359-4e26-4ae9-8251-11e56381e3bd","Type":"ContainerDied","Data":"d52f3b5a282eff0303b6cbf6bb425d4ff66ce65fedff6b076dc6dab8aafdf9b8"} Oct 31 00:59:14.444467 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:14.444356 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-4ab8e5-b77879c55-qptzw" event={"ID":"81a85359-4e26-4ae9-8251-11e56381e3bd","Type":"ContainerDied","Data":"d746f722099ad1b04b64a27477416c8318c5ff5fa18dfec01280e6db6dd4faa4"} Oct 31 00:59:14.444467 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:14.444403 5222 scope.go:122] "RemoveContainer" containerID="d52f3b5a282eff0303b6cbf6bb425d4ff66ce65fedff6b076dc6dab8aafdf9b8" Oct 31 00:59:14.446692 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:14.446668 5222 generic.go:356] "Generic (PLEG): container finished" podID="3132fd63-bf3d-4386-a31c-3c2ff47efd20" containerID="3c703c647fd747d85b363db7d63fc973555d90660597e3d8c2d54dff6fa6890e" exitCode=0 Oct 31 00:59:14.446801 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:14.446772 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4ab8e5-79cc98db87-r95fp" event={"ID":"3132fd63-bf3d-4386-a31c-3c2ff47efd20","Type":"ContainerDied","Data":"3c703c647fd747d85b363db7d63fc973555d90660597e3d8c2d54dff6fa6890e"} Oct 31 00:59:14.457185 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:14.457157 5222 scope.go:122] "RemoveContainer" containerID="d52f3b5a282eff0303b6cbf6bb425d4ff66ce65fedff6b076dc6dab8aafdf9b8" Oct 31 00:59:14.457576 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:59:14.457554 5222 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d52f3b5a282eff0303b6cbf6bb425d4ff66ce65fedff6b076dc6dab8aafdf9b8\": container with ID starting with d52f3b5a282eff0303b6cbf6bb425d4ff66ce65fedff6b076dc6dab8aafdf9b8 not found: ID does not exist" containerID="d52f3b5a282eff0303b6cbf6bb425d4ff66ce65fedff6b076dc6dab8aafdf9b8" Oct 31 00:59:14.457673 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:14.457584 5222 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d52f3b5a282eff0303b6cbf6bb425d4ff66ce65fedff6b076dc6dab8aafdf9b8"} err="failed to get container status \"d52f3b5a282eff0303b6cbf6bb425d4ff66ce65fedff6b076dc6dab8aafdf9b8\": rpc error: code = NotFound desc = could not find container \"d52f3b5a282eff0303b6cbf6bb425d4ff66ce65fedff6b076dc6dab8aafdf9b8\": container with ID starting with d52f3b5a282eff0303b6cbf6bb425d4ff66ce65fedff6b076dc6dab8aafdf9b8 not found: ID does not exist" Oct 31 00:59:14.479758 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:14.479730 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4ab8e5-79cc98db87-r95fp" Oct 31 00:59:14.522732 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:14.522700 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3132fd63-bf3d-4386-a31c-3c2ff47efd20-kube-api-access-krfp7\" (UniqueName: \"kubernetes.io/projected/3132fd63-bf3d-4386-a31c-3c2ff47efd20-kube-api-access-krfp7\") pod \"3132fd63-bf3d-4386-a31c-3c2ff47efd20\" (UID: \"3132fd63-bf3d-4386-a31c-3c2ff47efd20\") " Oct 31 00:59:14.522929 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:14.522810 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3132fd63-bf3d-4386-a31c-3c2ff47efd20-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3132fd63-bf3d-4386-a31c-3c2ff47efd20-plugins-config-volume\") pod \"3132fd63-bf3d-4386-a31c-3c2ff47efd20\" (UID: \"3132fd63-bf3d-4386-a31c-3c2ff47efd20\") " Oct 31 00:59:14.523000 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:14.522978 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8t7mp\" (UniqueName: \"kubernetes.io/projected/81a85359-4e26-4ae9-8251-11e56381e3bd-kube-api-access-8t7mp\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:59:14.523000 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:14.522993 5222 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81a85359-4e26-4ae9-8251-11e56381e3bd-plugins-config-volume\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:59:14.523180 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:14.523155 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3132fd63-bf3d-4386-a31c-3c2ff47efd20-plugins-config-volume" pod "3132fd63-bf3d-4386-a31c-3c2ff47efd20" (UID: "3132fd63-bf3d-4386-a31c-3c2ff47efd20"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:59:14.525098 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:14.525071 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3132fd63-bf3d-4386-a31c-3c2ff47efd20-kube-api-access-krfp7" pod "3132fd63-bf3d-4386-a31c-3c2ff47efd20" (UID: "3132fd63-bf3d-4386-a31c-3c2ff47efd20"). InnerVolumeSpecName "kube-api-access-krfp7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:59:14.624283 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:14.624246 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krfp7\" (UniqueName: \"kubernetes.io/projected/3132fd63-bf3d-4386-a31c-3c2ff47efd20-kube-api-access-krfp7\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:59:14.624283 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:14.624277 5222 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3132fd63-bf3d-4386-a31c-3c2ff47efd20-plugins-config-volume\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:59:14.792194 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:14.792159 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4ab8e5-b77879c55-qptzw"] Oct 31 00:59:14.801655 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:14.801618 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4ab8e5-b77879c55-qptzw"] Oct 31 00:59:15.455564 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:15.455482 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4ab8e5-79cc98db87-r95fp" Oct 31 00:59:15.456041 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:15.455485 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4ab8e5-79cc98db87-r95fp" event={"ID":"3132fd63-bf3d-4386-a31c-3c2ff47efd20","Type":"ContainerDied","Data":"99a3e8360803ebf10da834ae17ba93d4376bddf2c216eb3ef51bcfbd5b6b7c59"} Oct 31 00:59:15.456041 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:15.455618 5222 scope.go:122] "RemoveContainer" containerID="3c703c647fd747d85b363db7d63fc973555d90660597e3d8c2d54dff6fa6890e" Oct 31 00:59:15.627314 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:15.627276 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4ab8e5-79cc98db87-r95fp"] Oct 31 00:59:15.631862 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:15.631837 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4ab8e5-79cc98db87-r95fp"] Oct 31 00:59:15.655610 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:15.655576 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3132fd63-bf3d-4386-a31c-3c2ff47efd20" path="/var/lib/kubelet/pods/3132fd63-bf3d-4386-a31c-3c2ff47efd20/volumes" Oct 31 00:59:15.655974 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:15.655962 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81a85359-4e26-4ae9-8251-11e56381e3bd" path="/var/lib/kubelet/pods/81a85359-4e26-4ae9-8251-11e56381e3bd/volumes" Oct 31 00:59:17.917308 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:17.917266 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4ab8e5-b7m9x"] Oct 31 00:59:17.968170 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:17.968132 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4ab8e5-b7m9x"] Oct 31 00:59:19.655604 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:19.655560 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3952d123-2319-48d7-b9fd-23e5c1eceb00" path="/var/lib/kubelet/pods/3952d123-2319-48d7-b9fd-23e5c1eceb00/volumes" Oct 31 00:59:21.534715 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:21.534677 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-79tk8"] Oct 31 00:59:25.202868 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:25.202830 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-4ab8e5-openshift-default-6c89ffbf45-mtckf"] Oct 31 00:59:25.203402 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:25.203045 5222 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-4ab8e5-openshift-default-6c89ffbf45-mtckf" podUID="c46a2765-b445-4cd6-9b3a-1aef0a68b895" containerName="istio-proxy" containerID="cri-o://b6ff4242e6b0992cd4acc1bd25bb6984ff8beed4ae1f8953d333d3f0d7f1f5ac" gracePeriod=30 Oct 31 00:59:26.130697 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:26.130657 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bc2453-2zdjs"] Oct 31 00:59:26.131237 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:26.131217 5222 cpu_manager.go:414] "containerMap: removing container" podUID="3132fd63-bf3d-4386-a31c-3c2ff47efd20" containerName="payload-pre-processing" Oct 31 00:59:26.131354 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:26.131238 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3132fd63-bf3d-4386-a31c-3c2ff47efd20" containerName="payload-pre-processing" Oct 31 00:59:26.131354 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:26.131270 5222 cpu_manager.go:414] "containerMap: removing container" podUID="74eae8f3-ee20-417d-ada2-d300274b2038" containerName="main" Oct 31 00:59:26.131354 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:26.131279 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74eae8f3-ee20-417d-ada2-d300274b2038" containerName="main" Oct 31 00:59:26.131354 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:26.131293 5222 cpu_manager.go:414] "containerMap: removing container" podUID="81a85359-4e26-4ae9-8251-11e56381e3bd" containerName="payload-processing" Oct 31 00:59:26.131354 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:26.131300 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81a85359-4e26-4ae9-8251-11e56381e3bd" containerName="payload-processing" Oct 31 00:59:26.131354 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:26.131330 5222 cpu_manager.go:414] "containerMap: removing container" podUID="3952d123-2319-48d7-b9fd-23e5c1eceb00" containerName="revoke-keys" Oct 31 00:59:26.131354 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:26.131337 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3952d123-2319-48d7-b9fd-23e5c1eceb00" containerName="revoke-keys" Oct 31 00:59:26.131750 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:26.131466 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="74eae8f3-ee20-417d-ada2-d300274b2038" containerName="main" Oct 31 00:59:26.131750 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:26.131482 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="81a85359-4e26-4ae9-8251-11e56381e3bd" containerName="payload-processing" Oct 31 00:59:26.131750 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:26.131497 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="3952d123-2319-48d7-b9fd-23e5c1eceb00" containerName="revoke-keys" Oct 31 00:59:26.131750 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:26.131510 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="3132fd63-bf3d-4386-a31c-3c2ff47efd20" containerName="payload-pre-processing" Oct 31 00:59:26.136601 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:26.136578 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bc2453-2zdjs" Oct 31 00:59:26.245111 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:26.245062 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:59:26.245660 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:26.245214 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:59:26.245660 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:26.245237 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:59:26.245660 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:26.245268 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfz9z\"" Oct 31 00:59:26.247844 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:26.247819 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bc2453-2zdjs"] Oct 31 00:59:26.266556 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:26.266519 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a63ce376-cdca-443e-8ada-6c3959ff63e4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-bc2453-2zdjs\" (UID: \"a63ce376-cdca-443e-8ada-6c3959ff63e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bc2453-2zdjs" Oct 31 00:59:26.266741 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:26.266646 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a63ce376-cdca-443e-8ada-6c3959ff63e4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-bc2453-2zdjs\" (UID: \"a63ce376-cdca-443e-8ada-6c3959ff63e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bc2453-2zdjs" Oct 31 00:59:26.266741 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:26.266704 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxz99\" (UniqueName: \"kubernetes.io/projected/a63ce376-cdca-443e-8ada-6c3959ff63e4-kube-api-access-pxz99\") pod \"maas-api-revoke-keys-e2e-inf-b-bc2453-2zdjs\" (UID: \"a63ce376-cdca-443e-8ada-6c3959ff63e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bc2453-2zdjs" Oct 31 00:59:26.368378 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:26.368337 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a63ce376-cdca-443e-8ada-6c3959ff63e4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-bc2453-2zdjs\" (UID: \"a63ce376-cdca-443e-8ada-6c3959ff63e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bc2453-2zdjs" Oct 31 00:59:26.368575 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:26.368494 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a63ce376-cdca-443e-8ada-6c3959ff63e4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-bc2453-2zdjs\" (UID: \"a63ce376-cdca-443e-8ada-6c3959ff63e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bc2453-2zdjs" Oct 31 00:59:26.368575 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:26.368529 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxz99\" (UniqueName: \"kubernetes.io/projected/a63ce376-cdca-443e-8ada-6c3959ff63e4-kube-api-access-pxz99\") pod \"maas-api-revoke-keys-e2e-inf-b-bc2453-2zdjs\" (UID: \"a63ce376-cdca-443e-8ada-6c3959ff63e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bc2453-2zdjs" Oct 31 00:59:26.369141 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:26.369116 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a63ce376-cdca-443e-8ada-6c3959ff63e4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-bc2453-2zdjs\" (UID: \"a63ce376-cdca-443e-8ada-6c3959ff63e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bc2453-2zdjs" Oct 31 00:59:26.370971 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:26.370949 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a63ce376-cdca-443e-8ada-6c3959ff63e4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-bc2453-2zdjs\" (UID: \"a63ce376-cdca-443e-8ada-6c3959ff63e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bc2453-2zdjs" Oct 31 00:59:26.432319 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:26.432238 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxz99\" (UniqueName: \"kubernetes.io/projected/a63ce376-cdca-443e-8ada-6c3959ff63e4-kube-api-access-pxz99\") pod \"maas-api-revoke-keys-e2e-inf-b-bc2453-2zdjs\" (UID: \"a63ce376-cdca-443e-8ada-6c3959ff63e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bc2453-2zdjs" Oct 31 00:59:26.451002 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:26.450965 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bc2453-2zdjs" Oct 31 00:59:26.830383 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:59:26.830339 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda63ce376_cdca_443e_8ada_6c3959ff63e4.slice/crio-dce8259ea2d1fce43f7a5500f62bca3fc7cadca5a908e49451184055a6f7879c WatchSource:0}: Error finding container dce8259ea2d1fce43f7a5500f62bca3fc7cadca5a908e49451184055a6f7879c: Status 404 returned error can't find the container with id dce8259ea2d1fce43f7a5500f62bca3fc7cadca5a908e49451184055a6f7879c Oct 31 00:59:26.833163 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:26.833133 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bc2453-2zdjs"] Oct 31 00:59:27.531443 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:27.531397 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bc2453-2zdjs" event={"ID":"a63ce376-cdca-443e-8ada-6c3959ff63e4","Type":"ContainerStarted","Data":"799b7351b0255223fd1530ba1515955e8ab19d3e77c3e979dbde54567137df7d"} Oct 31 00:59:27.531851 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:27.531464 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bc2453-2zdjs" event={"ID":"a63ce376-cdca-443e-8ada-6c3959ff63e4","Type":"ContainerStarted","Data":"dce8259ea2d1fce43f7a5500f62bca3fc7cadca5a908e49451184055a6f7879c"} Oct 31 00:59:27.652234 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:27.652166 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bc2453-2zdjs" podStartSLOduration=1.652147284 podStartE2EDuration="1.652147284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:59:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:59:27.649737187 +0000 UTC m=+2660.120906938" watchObservedRunningTime="2026-10-31 00:59:27.652147284 +0000 UTC m=+2660.123317037" Oct 31 00:59:28.538678 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:28.538639 5222 generic.go:356] "Generic (PLEG): container finished" podID="a63ce376-cdca-443e-8ada-6c3959ff63e4" containerID="799b7351b0255223fd1530ba1515955e8ab19d3e77c3e979dbde54567137df7d" exitCode=0 Oct 31 00:59:28.539093 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:28.538718 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bc2453-2zdjs" event={"ID":"a63ce376-cdca-443e-8ada-6c3959ff63e4","Type":"ContainerDied","Data":"799b7351b0255223fd1530ba1515955e8ab19d3e77c3e979dbde54567137df7d"} Oct 31 00:59:29.687971 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:29.687937 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bc2453-2zdjs" Oct 31 00:59:29.806262 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:29.806210 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a63ce376-cdca-443e-8ada-6c3959ff63e4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a63ce376-cdca-443e-8ada-6c3959ff63e4-maas-api-xks-ca\") pod \"a63ce376-cdca-443e-8ada-6c3959ff63e4\" (UID: \"a63ce376-cdca-443e-8ada-6c3959ff63e4\") " Oct 31 00:59:29.806492 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:29.806284 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a63ce376-cdca-443e-8ada-6c3959ff63e4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a63ce376-cdca-443e-8ada-6c3959ff63e4-maas-api-openshift-ca\") pod \"a63ce376-cdca-443e-8ada-6c3959ff63e4\" (UID: \"a63ce376-cdca-443e-8ada-6c3959ff63e4\") " Oct 31 00:59:29.806492 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:29.806315 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a63ce376-cdca-443e-8ada-6c3959ff63e4-kube-api-access-pxz99\" (UniqueName: \"kubernetes.io/projected/a63ce376-cdca-443e-8ada-6c3959ff63e4-kube-api-access-pxz99\") pod \"a63ce376-cdca-443e-8ada-6c3959ff63e4\" (UID: \"a63ce376-cdca-443e-8ada-6c3959ff63e4\") " Oct 31 00:59:29.806712 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:29.806673 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a63ce376-cdca-443e-8ada-6c3959ff63e4-maas-api-openshift-ca" pod "a63ce376-cdca-443e-8ada-6c3959ff63e4" (UID: "a63ce376-cdca-443e-8ada-6c3959ff63e4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:59:29.808689 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:29.808653 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a63ce376-cdca-443e-8ada-6c3959ff63e4-maas-api-xks-ca" pod "a63ce376-cdca-443e-8ada-6c3959ff63e4" (UID: "a63ce376-cdca-443e-8ada-6c3959ff63e4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:59:29.808827 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:29.808757 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a63ce376-cdca-443e-8ada-6c3959ff63e4-kube-api-access-pxz99" pod "a63ce376-cdca-443e-8ada-6c3959ff63e4" (UID: "a63ce376-cdca-443e-8ada-6c3959ff63e4"). InnerVolumeSpecName "kube-api-access-pxz99". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:59:29.907659 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:29.907626 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a63ce376-cdca-443e-8ada-6c3959ff63e4-maas-api-xks-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:59:29.907659 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:29.907656 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a63ce376-cdca-443e-8ada-6c3959ff63e4-maas-api-openshift-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:59:29.907659 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:29.907666 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxz99\" (UniqueName: \"kubernetes.io/projected/a63ce376-cdca-443e-8ada-6c3959ff63e4-kube-api-access-pxz99\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:59:30.448790 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:30.448763 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4ab8e5-openshift-default-6c89ffbf45-mtckf" Oct 31 00:59:30.552427 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:30.552385 5222 generic.go:356] "Generic (PLEG): container finished" podID="c46a2765-b445-4cd6-9b3a-1aef0a68b895" containerID="b6ff4242e6b0992cd4acc1bd25bb6984ff8beed4ae1f8953d333d3f0d7f1f5ac" exitCode=0 Oct 31 00:59:30.552427 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:30.552422 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4ab8e5-openshift-default-6c89ffbf45-mtckf" event={"ID":"c46a2765-b445-4cd6-9b3a-1aef0a68b895","Type":"ContainerDied","Data":"b6ff4242e6b0992cd4acc1bd25bb6984ff8beed4ae1f8953d333d3f0d7f1f5ac"} Oct 31 00:59:30.552687 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:30.552458 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4ab8e5-openshift-default-6c89ffbf45-mtckf" Oct 31 00:59:30.552687 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:30.552493 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4ab8e5-openshift-default-6c89ffbf45-mtckf" event={"ID":"c46a2765-b445-4cd6-9b3a-1aef0a68b895","Type":"ContainerDied","Data":"c143921677f0e227a7738500706e8b2bba156f895a540c22426ce5d0921242de"} Oct 31 00:59:30.552687 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:30.552524 5222 scope.go:122] "RemoveContainer" containerID="b6ff4242e6b0992cd4acc1bd25bb6984ff8beed4ae1f8953d333d3f0d7f1f5ac" Oct 31 00:59:30.554011 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:30.553995 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bc2453-2zdjs" Oct 31 00:59:30.554113 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:30.554036 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bc2453-2zdjs" event={"ID":"a63ce376-cdca-443e-8ada-6c3959ff63e4","Type":"ContainerDied","Data":"dce8259ea2d1fce43f7a5500f62bca3fc7cadca5a908e49451184055a6f7879c"} Oct 31 00:59:30.554162 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:30.554118 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dce8259ea2d1fce43f7a5500f62bca3fc7cadca5a908e49451184055a6f7879c" Oct 31 00:59:30.566863 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:30.566834 5222 scope.go:122] "RemoveContainer" containerID="ffd419175db9ab219d8aa6bb9ebd361568b3e73b48ccb04f0f9d7ff01734c8bd" Oct 31 00:59:30.576558 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:30.576538 5222 scope.go:122] "RemoveContainer" containerID="b6ff4242e6b0992cd4acc1bd25bb6984ff8beed4ae1f8953d333d3f0d7f1f5ac" Oct 31 00:59:30.576884 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:59:30.576864 5222 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6ff4242e6b0992cd4acc1bd25bb6984ff8beed4ae1f8953d333d3f0d7f1f5ac\": container with ID starting with b6ff4242e6b0992cd4acc1bd25bb6984ff8beed4ae1f8953d333d3f0d7f1f5ac not found: ID does not exist" containerID="b6ff4242e6b0992cd4acc1bd25bb6984ff8beed4ae1f8953d333d3f0d7f1f5ac" Oct 31 00:59:30.576932 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:30.576895 5222 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6ff4242e6b0992cd4acc1bd25bb6984ff8beed4ae1f8953d333d3f0d7f1f5ac"} err="failed to get container status \"b6ff4242e6b0992cd4acc1bd25bb6984ff8beed4ae1f8953d333d3f0d7f1f5ac\": rpc error: code = NotFound desc = could not find container \"b6ff4242e6b0992cd4acc1bd25bb6984ff8beed4ae1f8953d333d3f0d7f1f5ac\": container with ID starting with b6ff4242e6b0992cd4acc1bd25bb6984ff8beed4ae1f8953d333d3f0d7f1f5ac not found: ID does not exist" Oct 31 00:59:30.576932 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:30.576916 5222 scope.go:122] "RemoveContainer" containerID="ffd419175db9ab219d8aa6bb9ebd361568b3e73b48ccb04f0f9d7ff01734c8bd" Oct 31 00:59:30.577184 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:59:30.577168 5222 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffd419175db9ab219d8aa6bb9ebd361568b3e73b48ccb04f0f9d7ff01734c8bd\": container with ID starting with ffd419175db9ab219d8aa6bb9ebd361568b3e73b48ccb04f0f9d7ff01734c8bd not found: ID does not exist" containerID="ffd419175db9ab219d8aa6bb9ebd361568b3e73b48ccb04f0f9d7ff01734c8bd" Oct 31 00:59:30.577253 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:30.577186 5222 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffd419175db9ab219d8aa6bb9ebd361568b3e73b48ccb04f0f9d7ff01734c8bd"} err="failed to get container status \"ffd419175db9ab219d8aa6bb9ebd361568b3e73b48ccb04f0f9d7ff01734c8bd\": rpc error: code = NotFound desc = could not find container \"ffd419175db9ab219d8aa6bb9ebd361568b3e73b48ccb04f0f9d7ff01734c8bd\": container with ID starting with ffd419175db9ab219d8aa6bb9ebd361568b3e73b48ccb04f0f9d7ff01734c8bd not found: ID does not exist" Oct 31 00:59:30.614508 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:30.614447 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c46a2765-b445-4cd6-9b3a-1aef0a68b895-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c46a2765-b445-4cd6-9b3a-1aef0a68b895-istio-data\") pod \"c46a2765-b445-4cd6-9b3a-1aef0a68b895\" (UID: \"c46a2765-b445-4cd6-9b3a-1aef0a68b895\") " Oct 31 00:59:30.614508 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:30.614504 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c46a2765-b445-4cd6-9b3a-1aef0a68b895-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c46a2765-b445-4cd6-9b3a-1aef0a68b895-credential-socket\") pod \"c46a2765-b445-4cd6-9b3a-1aef0a68b895\" (UID: \"c46a2765-b445-4cd6-9b3a-1aef0a68b895\") " Oct 31 00:59:30.614730 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:30.614540 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c46a2765-b445-4cd6-9b3a-1aef0a68b895-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c46a2765-b445-4cd6-9b3a-1aef0a68b895-istio-envoy\") pod \"c46a2765-b445-4cd6-9b3a-1aef0a68b895\" (UID: \"c46a2765-b445-4cd6-9b3a-1aef0a68b895\") " Oct 31 00:59:30.614730 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:30.614567 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c46a2765-b445-4cd6-9b3a-1aef0a68b895-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c46a2765-b445-4cd6-9b3a-1aef0a68b895-istio-podinfo\") pod \"c46a2765-b445-4cd6-9b3a-1aef0a68b895\" (UID: \"c46a2765-b445-4cd6-9b3a-1aef0a68b895\") " Oct 31 00:59:30.614730 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:30.614610 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c46a2765-b445-4cd6-9b3a-1aef0a68b895-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c46a2765-b445-4cd6-9b3a-1aef0a68b895-workload-certs\") pod \"c46a2765-b445-4cd6-9b3a-1aef0a68b895\" (UID: \"c46a2765-b445-4cd6-9b3a-1aef0a68b895\") " Oct 31 00:59:30.614730 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:30.614662 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c46a2765-b445-4cd6-9b3a-1aef0a68b895-istio-token\" (UniqueName: \"kubernetes.io/projected/c46a2765-b445-4cd6-9b3a-1aef0a68b895-istio-token\") pod \"c46a2765-b445-4cd6-9b3a-1aef0a68b895\" (UID: \"c46a2765-b445-4cd6-9b3a-1aef0a68b895\") " Oct 31 00:59:30.614929 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:30.614781 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c46a2765-b445-4cd6-9b3a-1aef0a68b895-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c46a2765-b445-4cd6-9b3a-1aef0a68b895-workload-socket\") pod \"c46a2765-b445-4cd6-9b3a-1aef0a68b895\" (UID: \"c46a2765-b445-4cd6-9b3a-1aef0a68b895\") " Oct 31 00:59:30.614929 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:30.614802 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c46a2765-b445-4cd6-9b3a-1aef0a68b895-credential-socket" pod "c46a2765-b445-4cd6-9b3a-1aef0a68b895" (UID: "c46a2765-b445-4cd6-9b3a-1aef0a68b895"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:59:30.614929 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:30.614818 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c46a2765-b445-4cd6-9b3a-1aef0a68b895-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c46a2765-b445-4cd6-9b3a-1aef0a68b895-istiod-ca-cert\") pod \"c46a2765-b445-4cd6-9b3a-1aef0a68b895\" (UID: \"c46a2765-b445-4cd6-9b3a-1aef0a68b895\") " Oct 31 00:59:30.614929 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:30.614810 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c46a2765-b445-4cd6-9b3a-1aef0a68b895-istio-data" pod "c46a2765-b445-4cd6-9b3a-1aef0a68b895" (UID: "c46a2765-b445-4cd6-9b3a-1aef0a68b895"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:59:30.614929 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:30.614844 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c46a2765-b445-4cd6-9b3a-1aef0a68b895-kube-api-access-6dzcm\" (UniqueName: \"kubernetes.io/projected/c46a2765-b445-4cd6-9b3a-1aef0a68b895-kube-api-access-6dzcm\") pod \"c46a2765-b445-4cd6-9b3a-1aef0a68b895\" (UID: \"c46a2765-b445-4cd6-9b3a-1aef0a68b895\") " Oct 31 00:59:30.615201 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:30.615029 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c46a2765-b445-4cd6-9b3a-1aef0a68b895-workload-socket" pod "c46a2765-b445-4cd6-9b3a-1aef0a68b895" (UID: "c46a2765-b445-4cd6-9b3a-1aef0a68b895"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:59:30.615201 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:30.615060 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c46a2765-b445-4cd6-9b3a-1aef0a68b895-workload-certs" pod "c46a2765-b445-4cd6-9b3a-1aef0a68b895" (UID: "c46a2765-b445-4cd6-9b3a-1aef0a68b895"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:59:30.615201 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:30.615141 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c46a2765-b445-4cd6-9b3a-1aef0a68b895-istiod-ca-cert" pod "c46a2765-b445-4cd6-9b3a-1aef0a68b895" (UID: "c46a2765-b445-4cd6-9b3a-1aef0a68b895"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:59:30.615567 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:30.615274 5222 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c46a2765-b445-4cd6-9b3a-1aef0a68b895-istio-data\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:59:30.615567 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:30.615289 5222 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c46a2765-b445-4cd6-9b3a-1aef0a68b895-credential-socket\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:59:30.615567 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:30.615299 5222 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c46a2765-b445-4cd6-9b3a-1aef0a68b895-workload-certs\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:59:30.615567 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:30.615308 5222 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c46a2765-b445-4cd6-9b3a-1aef0a68b895-workload-socket\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:59:30.615567 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:30.615316 5222 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c46a2765-b445-4cd6-9b3a-1aef0a68b895-istiod-ca-cert\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:59:30.617025 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:30.616999 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c46a2765-b445-4cd6-9b3a-1aef0a68b895-istio-podinfo" pod "c46a2765-b445-4cd6-9b3a-1aef0a68b895" (UID: "c46a2765-b445-4cd6-9b3a-1aef0a68b895"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:59:30.617132 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:30.617108 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c46a2765-b445-4cd6-9b3a-1aef0a68b895-kube-api-access-6dzcm" pod "c46a2765-b445-4cd6-9b3a-1aef0a68b895" (UID: "c46a2765-b445-4cd6-9b3a-1aef0a68b895"). InnerVolumeSpecName "kube-api-access-6dzcm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:59:30.617497 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:30.617477 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c46a2765-b445-4cd6-9b3a-1aef0a68b895-istio-envoy" pod "c46a2765-b445-4cd6-9b3a-1aef0a68b895" (UID: "c46a2765-b445-4cd6-9b3a-1aef0a68b895"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:59:30.617543 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:30.617510 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c46a2765-b445-4cd6-9b3a-1aef0a68b895-istio-token" pod "c46a2765-b445-4cd6-9b3a-1aef0a68b895" (UID: "c46a2765-b445-4cd6-9b3a-1aef0a68b895"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:59:30.716426 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:30.716395 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dzcm\" (UniqueName: \"kubernetes.io/projected/c46a2765-b445-4cd6-9b3a-1aef0a68b895-kube-api-access-6dzcm\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:59:30.716426 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:30.716423 5222 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c46a2765-b445-4cd6-9b3a-1aef0a68b895-istio-envoy\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:59:30.716426 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:30.716440 5222 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c46a2765-b445-4cd6-9b3a-1aef0a68b895-istio-podinfo\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:59:30.716915 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:30.716450 5222 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c46a2765-b445-4cd6-9b3a-1aef0a68b895-istio-token\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:59:30.976659 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:30.976626 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-4ab8e5-openshift-default-6c89ffbf45-mtckf"] Oct 31 00:59:30.984750 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:30.984717 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-4ab8e5-openshift-default-6c89ffbf45-mtckf"] Oct 31 00:59:31.655099 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:31.655065 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c46a2765-b445-4cd6-9b3a-1aef0a68b895" path="/var/lib/kubelet/pods/c46a2765-b445-4cd6-9b3a-1aef0a68b895/volumes" Oct 31 00:59:35.890242 ip-10-0-137-56 kubenswrapper[5222]: E1031 00:59:35.890204 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 00:59:44.060755 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:44.060716 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9a35539-tvbkt"] Oct 31 00:59:44.111855 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:44.111810 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-432acaff-l87vl"] Oct 31 00:59:44.113843 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:44.113824 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9a35539-tvbkt"] Oct 31 00:59:44.115729 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:44.115702 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-432acaff-l87vl"] Oct 31 00:59:45.223170 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:45.223132 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bc2453-2zdjs"] Oct 31 00:59:45.274203 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:45.274166 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bc2453-2zdjs"] Oct 31 00:59:45.655133 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:45.655097 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ce45316-6e99-4cf1-83b4-b24f3822a4f2" path="/var/lib/kubelet/pods/4ce45316-6e99-4cf1-83b4-b24f3822a4f2/volumes" Oct 31 00:59:45.655541 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:45.655527 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a63ce376-cdca-443e-8ada-6c3959ff63e4" path="/var/lib/kubelet/pods/a63ce376-cdca-443e-8ada-6c3959ff63e4/volumes" Oct 31 00:59:45.655890 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:45.655878 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef06038e-a7b5-4d47-a9a7-32129308ccc5" path="/var/lib/kubelet/pods/ef06038e-a7b5-4d47-a9a7-32129308ccc5/volumes" Oct 31 00:59:48.822269 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:48.822231 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-79tk8"] Oct 31 00:59:53.136759 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:53.136729 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-4b86e4-bfsqp"] Oct 31 00:59:53.137155 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:53.137141 5222 cpu_manager.go:414] "containerMap: removing container" podUID="c46a2765-b445-4cd6-9b3a-1aef0a68b895" containerName="istio-proxy" Oct 31 00:59:53.137155 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:53.137152 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c46a2765-b445-4cd6-9b3a-1aef0a68b895" containerName="istio-proxy" Oct 31 00:59:53.137234 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:53.137162 5222 cpu_manager.go:414] "containerMap: removing container" podUID="a63ce376-cdca-443e-8ada-6c3959ff63e4" containerName="revoke-keys" Oct 31 00:59:53.137234 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:53.137167 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a63ce376-cdca-443e-8ada-6c3959ff63e4" containerName="revoke-keys" Oct 31 00:59:53.137234 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:53.137176 5222 cpu_manager.go:414] "containerMap: removing container" podUID="c46a2765-b445-4cd6-9b3a-1aef0a68b895" containerName="istio-proxy" Oct 31 00:59:53.137234 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:53.137181 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c46a2765-b445-4cd6-9b3a-1aef0a68b895" containerName="istio-proxy" Oct 31 00:59:53.137362 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:53.137256 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="c46a2765-b445-4cd6-9b3a-1aef0a68b895" containerName="istio-proxy" Oct 31 00:59:53.137362 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:53.137262 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="a63ce376-cdca-443e-8ada-6c3959ff63e4" containerName="revoke-keys" Oct 31 00:59:53.137362 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:53.137270 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="c46a2765-b445-4cd6-9b3a-1aef0a68b895" containerName="istio-proxy" Oct 31 00:59:53.141940 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:53.141923 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-4b86e4-bfsqp" Oct 31 00:59:53.162633 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:53.162600 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d38e33c1-eccd-41cd-9cc6-e5eb7a5eb48b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-4b86e4-bfsqp\" (UID: \"d38e33c1-eccd-41cd-9cc6-e5eb7a5eb48b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-4b86e4-bfsqp" Oct 31 00:59:53.162791 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:53.162741 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbp2c\" (UniqueName: \"kubernetes.io/projected/d38e33c1-eccd-41cd-9cc6-e5eb7a5eb48b-kube-api-access-jbp2c\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-4b86e4-bfsqp\" (UID: \"d38e33c1-eccd-41cd-9cc6-e5eb7a5eb48b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-4b86e4-bfsqp" Oct 31 00:59:53.162854 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:53.162839 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d38e33c1-eccd-41cd-9cc6-e5eb7a5eb48b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-4b86e4-bfsqp\" (UID: \"d38e33c1-eccd-41cd-9cc6-e5eb7a5eb48b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-4b86e4-bfsqp" Oct 31 00:59:53.195053 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:53.195022 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-4b86e4-bfsqp"] Oct 31 00:59:53.243146 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:53.243112 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:59:53.243334 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:53.243151 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfz9z\"" Oct 31 00:59:53.250942 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:53.250920 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:59:53.250942 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:53.250925 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:59:53.264113 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:53.264080 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbp2c\" (UniqueName: \"kubernetes.io/projected/d38e33c1-eccd-41cd-9cc6-e5eb7a5eb48b-kube-api-access-jbp2c\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-4b86e4-bfsqp\" (UID: \"d38e33c1-eccd-41cd-9cc6-e5eb7a5eb48b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-4b86e4-bfsqp" Oct 31 00:59:53.264305 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:53.264143 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d38e33c1-eccd-41cd-9cc6-e5eb7a5eb48b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-4b86e4-bfsqp\" (UID: \"d38e33c1-eccd-41cd-9cc6-e5eb7a5eb48b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-4b86e4-bfsqp" Oct 31 00:59:53.264305 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:53.264184 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d38e33c1-eccd-41cd-9cc6-e5eb7a5eb48b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-4b86e4-bfsqp\" (UID: \"d38e33c1-eccd-41cd-9cc6-e5eb7a5eb48b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-4b86e4-bfsqp" Oct 31 00:59:53.264859 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:53.264822 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d38e33c1-eccd-41cd-9cc6-e5eb7a5eb48b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-4b86e4-bfsqp\" (UID: \"d38e33c1-eccd-41cd-9cc6-e5eb7a5eb48b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-4b86e4-bfsqp" Oct 31 00:59:53.266800 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:53.266777 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d38e33c1-eccd-41cd-9cc6-e5eb7a5eb48b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-4b86e4-bfsqp\" (UID: \"d38e33c1-eccd-41cd-9cc6-e5eb7a5eb48b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-4b86e4-bfsqp" Oct 31 00:59:53.326815 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:53.326785 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbp2c\" (UniqueName: \"kubernetes.io/projected/d38e33c1-eccd-41cd-9cc6-e5eb7a5eb48b-kube-api-access-jbp2c\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-4b86e4-bfsqp\" (UID: \"d38e33c1-eccd-41cd-9cc6-e5eb7a5eb48b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-4b86e4-bfsqp" Oct 31 00:59:53.454898 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:53.454811 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-4b86e4-bfsqp" Oct 31 00:59:53.825893 ip-10-0-137-56 kubenswrapper[5222]: W1031 00:59:53.825864 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd38e33c1_eccd_41cd_9cc6_e5eb7a5eb48b.slice/crio-5ad735ae56e372c700110f3c7b7eaebbadb2632d37314ec6154220613fe8df61 WatchSource:0}: Error finding container 5ad735ae56e372c700110f3c7b7eaebbadb2632d37314ec6154220613fe8df61: Status 404 returned error can't find the container with id 5ad735ae56e372c700110f3c7b7eaebbadb2632d37314ec6154220613fe8df61 Oct 31 00:59:53.827655 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:53.827628 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-4b86e4-bfsqp"] Oct 31 00:59:54.696263 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:54.696216 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-4b86e4-bfsqp" event={"ID":"d38e33c1-eccd-41cd-9cc6-e5eb7a5eb48b","Type":"ContainerStarted","Data":"114183d4066cf62037a2f026023764665f461ece8a5487d5c819b65bf0f8c134"} Oct 31 00:59:54.696796 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:54.696282 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-4b86e4-bfsqp" event={"ID":"d38e33c1-eccd-41cd-9cc6-e5eb7a5eb48b","Type":"ContainerStarted","Data":"5ad735ae56e372c700110f3c7b7eaebbadb2632d37314ec6154220613fe8df61"} Oct 31 00:59:55.703338 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:55.703300 5222 generic.go:356] "Generic (PLEG): container finished" podID="d38e33c1-eccd-41cd-9cc6-e5eb7a5eb48b" containerID="114183d4066cf62037a2f026023764665f461ece8a5487d5c819b65bf0f8c134" exitCode=0 Oct 31 00:59:55.703769 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:55.703399 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-4b86e4-bfsqp" event={"ID":"d38e33c1-eccd-41cd-9cc6-e5eb7a5eb48b","Type":"ContainerDied","Data":"114183d4066cf62037a2f026023764665f461ece8a5487d5c819b65bf0f8c134"} Oct 31 00:59:56.849862 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:56.849835 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-4b86e4-bfsqp" Oct 31 00:59:56.896641 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:56.896603 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d38e33c1-eccd-41cd-9cc6-e5eb7a5eb48b-kube-api-access-jbp2c\" (UniqueName: \"kubernetes.io/projected/d38e33c1-eccd-41cd-9cc6-e5eb7a5eb48b-kube-api-access-jbp2c\") pod \"d38e33c1-eccd-41cd-9cc6-e5eb7a5eb48b\" (UID: \"d38e33c1-eccd-41cd-9cc6-e5eb7a5eb48b\") " Oct 31 00:59:56.896855 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:56.896725 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d38e33c1-eccd-41cd-9cc6-e5eb7a5eb48b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d38e33c1-eccd-41cd-9cc6-e5eb7a5eb48b-maas-api-openshift-ca\") pod \"d38e33c1-eccd-41cd-9cc6-e5eb7a5eb48b\" (UID: \"d38e33c1-eccd-41cd-9cc6-e5eb7a5eb48b\") " Oct 31 00:59:56.896855 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:56.896754 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d38e33c1-eccd-41cd-9cc6-e5eb7a5eb48b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d38e33c1-eccd-41cd-9cc6-e5eb7a5eb48b-maas-api-xks-ca\") pod \"d38e33c1-eccd-41cd-9cc6-e5eb7a5eb48b\" (UID: \"d38e33c1-eccd-41cd-9cc6-e5eb7a5eb48b\") " Oct 31 00:59:56.897135 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:56.897110 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d38e33c1-eccd-41cd-9cc6-e5eb7a5eb48b-maas-api-openshift-ca" pod "d38e33c1-eccd-41cd-9cc6-e5eb7a5eb48b" (UID: "d38e33c1-eccd-41cd-9cc6-e5eb7a5eb48b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:59:56.898942 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:56.898917 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d38e33c1-eccd-41cd-9cc6-e5eb7a5eb48b-maas-api-xks-ca" pod "d38e33c1-eccd-41cd-9cc6-e5eb7a5eb48b" (UID: "d38e33c1-eccd-41cd-9cc6-e5eb7a5eb48b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:59:56.899167 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:56.899140 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d38e33c1-eccd-41cd-9cc6-e5eb7a5eb48b-kube-api-access-jbp2c" pod "d38e33c1-eccd-41cd-9cc6-e5eb7a5eb48b" (UID: "d38e33c1-eccd-41cd-9cc6-e5eb7a5eb48b"). InnerVolumeSpecName "kube-api-access-jbp2c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:59:56.998006 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:56.997909 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d38e33c1-eccd-41cd-9cc6-e5eb7a5eb48b-maas-api-openshift-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:59:56.998006 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:56.997939 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d38e33c1-eccd-41cd-9cc6-e5eb7a5eb48b-maas-api-xks-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:59:56.998006 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:56.997949 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbp2c\" (UniqueName: \"kubernetes.io/projected/d38e33c1-eccd-41cd-9cc6-e5eb7a5eb48b-kube-api-access-jbp2c\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:59:57.716545 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:57.716510 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-4b86e4-bfsqp" event={"ID":"d38e33c1-eccd-41cd-9cc6-e5eb7a5eb48b","Type":"ContainerDied","Data":"5ad735ae56e372c700110f3c7b7eaebbadb2632d37314ec6154220613fe8df61"} Oct 31 00:59:57.716545 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:57.716536 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-4b86e4-bfsqp" Oct 31 00:59:57.716758 ip-10-0-137-56 kubenswrapper[5222]: I1031 00:59:57.716564 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5ad735ae56e372c700110f3c7b7eaebbadb2632d37314ec6154220613fe8df61" Oct 31 01:00:00.186642 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:00.186598 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-0da668-29890140-tkrfp"] Oct 31 01:00:00.187261 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:00.187238 5222 cpu_manager.go:414] "containerMap: removing container" podUID="d38e33c1-eccd-41cd-9cc6-e5eb7a5eb48b" containerName="revoke-keys" Oct 31 01:00:00.187391 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:00.187264 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d38e33c1-eccd-41cd-9cc6-e5eb7a5eb48b" containerName="revoke-keys" Oct 31 01:00:00.187391 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:00.187362 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="d38e33c1-eccd-41cd-9cc6-e5eb7a5eb48b" containerName="revoke-keys" Oct 31 01:00:00.190856 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:00.190831 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4b86e4-29890140-lvlwb"] Oct 31 01:00:00.191026 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:00.190958 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-0da668-29890140-tkrfp" Oct 31 01:00:00.194473 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:00.194448 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890140-978j5"] Oct 31 01:00:00.194596 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:00.194549 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4b86e4-29890140-lvlwb" Oct 31 01:00:00.197825 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:00.197805 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890140-978j5" Oct 31 01:00:00.229819 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:00.229785 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpw5f\" (UniqueName: \"kubernetes.io/projected/1e0d5e52-a885-4074-b8d5-e146eb6061a6-kube-api-access-vpw5f\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-0da668-29890140-tkrfp\" (UID: \"1e0d5e52-a885-4074-b8d5-e146eb6061a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-0da668-29890140-tkrfp" Oct 31 01:00:00.230014 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:00.229827 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsr8m\" (UniqueName: \"kubernetes.io/projected/0399f8bf-008c-45eb-8550-d5c86ed8f7bf-kube-api-access-rsr8m\") pod \"maas-api-key-cleanup-29890140-978j5\" (UID: \"0399f8bf-008c-45eb-8550-d5c86ed8f7bf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890140-978j5" Oct 31 01:00:00.230014 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:00.229930 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a6b45a6-056a-4396-a140-4336f6a6179b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-4b86e4-29890140-lvlwb\" (UID: \"9a6b45a6-056a-4396-a140-4336f6a6179b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4b86e4-29890140-lvlwb" Oct 31 01:00:00.230014 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:00.229991 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e0d5e52-a885-4074-b8d5-e146eb6061a6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-0da668-29890140-tkrfp\" (UID: \"1e0d5e52-a885-4074-b8d5-e146eb6061a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-0da668-29890140-tkrfp" Oct 31 01:00:00.230205 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:00.230041 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0399f8bf-008c-45eb-8550-d5c86ed8f7bf-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890140-978j5\" (UID: \"0399f8bf-008c-45eb-8550-d5c86ed8f7bf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890140-978j5" Oct 31 01:00:00.230205 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:00.230059 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffqx9\" (UniqueName: \"kubernetes.io/projected/9a6b45a6-056a-4396-a140-4336f6a6179b-kube-api-access-ffqx9\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-4b86e4-29890140-lvlwb\" (UID: \"9a6b45a6-056a-4396-a140-4336f6a6179b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4b86e4-29890140-lvlwb" Oct 31 01:00:00.254363 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:00.254327 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfz9z\"" Oct 31 01:00:00.255521 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:00.255481 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:00:00.255666 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:00.255552 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:00:00.308049 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:00.308006 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890140-978j5"] Oct 31 01:00:00.310265 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:00.310234 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-0da668-29890140-tkrfp"] Oct 31 01:00:00.312018 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:00.311995 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4b86e4-29890140-lvlwb"] Oct 31 01:00:00.330980 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:00.330944 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpw5f\" (UniqueName: \"kubernetes.io/projected/1e0d5e52-a885-4074-b8d5-e146eb6061a6-kube-api-access-vpw5f\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-0da668-29890140-tkrfp\" (UID: \"1e0d5e52-a885-4074-b8d5-e146eb6061a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-0da668-29890140-tkrfp" Oct 31 01:00:00.331171 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:00.330989 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsr8m\" (UniqueName: \"kubernetes.io/projected/0399f8bf-008c-45eb-8550-d5c86ed8f7bf-kube-api-access-rsr8m\") pod \"maas-api-key-cleanup-29890140-978j5\" (UID: \"0399f8bf-008c-45eb-8550-d5c86ed8f7bf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890140-978j5" Oct 31 01:00:00.331171 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:00.331031 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a6b45a6-056a-4396-a140-4336f6a6179b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-4b86e4-29890140-lvlwb\" (UID: \"9a6b45a6-056a-4396-a140-4336f6a6179b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4b86e4-29890140-lvlwb" Oct 31 01:00:00.331171 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:00.331051 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e0d5e52-a885-4074-b8d5-e146eb6061a6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-0da668-29890140-tkrfp\" (UID: \"1e0d5e52-a885-4074-b8d5-e146eb6061a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-0da668-29890140-tkrfp" Oct 31 01:00:00.331171 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:00.331096 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0399f8bf-008c-45eb-8550-d5c86ed8f7bf-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890140-978j5\" (UID: \"0399f8bf-008c-45eb-8550-d5c86ed8f7bf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890140-978j5" Oct 31 01:00:00.331171 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:00.331149 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffqx9\" (UniqueName: \"kubernetes.io/projected/9a6b45a6-056a-4396-a140-4336f6a6179b-kube-api-access-ffqx9\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-4b86e4-29890140-lvlwb\" (UID: \"9a6b45a6-056a-4396-a140-4336f6a6179b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4b86e4-29890140-lvlwb" Oct 31 01:00:00.331830 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:00.331810 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a6b45a6-056a-4396-a140-4336f6a6179b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-4b86e4-29890140-lvlwb\" (UID: \"9a6b45a6-056a-4396-a140-4336f6a6179b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4b86e4-29890140-lvlwb" Oct 31 01:00:00.331933 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:00.331911 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e0d5e52-a885-4074-b8d5-e146eb6061a6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-0da668-29890140-tkrfp\" (UID: \"1e0d5e52-a885-4074-b8d5-e146eb6061a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-0da668-29890140-tkrfp" Oct 31 01:00:00.331972 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:00.331911 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0399f8bf-008c-45eb-8550-d5c86ed8f7bf-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890140-978j5\" (UID: \"0399f8bf-008c-45eb-8550-d5c86ed8f7bf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890140-978j5" Oct 31 01:00:00.393942 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:00.393905 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffqx9\" (UniqueName: \"kubernetes.io/projected/9a6b45a6-056a-4396-a140-4336f6a6179b-kube-api-access-ffqx9\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-4b86e4-29890140-lvlwb\" (UID: \"9a6b45a6-056a-4396-a140-4336f6a6179b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4b86e4-29890140-lvlwb" Oct 31 01:00:00.394096 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:00.393972 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpw5f\" (UniqueName: \"kubernetes.io/projected/1e0d5e52-a885-4074-b8d5-e146eb6061a6-kube-api-access-vpw5f\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-0da668-29890140-tkrfp\" (UID: \"1e0d5e52-a885-4074-b8d5-e146eb6061a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-0da668-29890140-tkrfp" Oct 31 01:00:00.399265 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:00.399235 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsr8m\" (UniqueName: \"kubernetes.io/projected/0399f8bf-008c-45eb-8550-d5c86ed8f7bf-kube-api-access-rsr8m\") pod \"maas-api-key-cleanup-29890140-978j5\" (UID: \"0399f8bf-008c-45eb-8550-d5c86ed8f7bf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890140-978j5" Oct 31 01:00:00.506452 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:00.506339 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-0da668-29890140-tkrfp" Oct 31 01:00:00.515798 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:00.515761 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4b86e4-29890140-lvlwb" Oct 31 01:00:00.523050 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:00.523022 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890140-978j5" Oct 31 01:00:00.904248 ip-10-0-137-56 kubenswrapper[5222]: W1031 01:00:00.904216 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e0d5e52_a885_4074_b8d5_e146eb6061a6.slice/crio-aed826cf672d110e144a99d831f69aafba06755190f71c7f9a2e5093a619e4f2 WatchSource:0}: Error finding container aed826cf672d110e144a99d831f69aafba06755190f71c7f9a2e5093a619e4f2: Status 404 returned error can't find the container with id aed826cf672d110e144a99d831f69aafba06755190f71c7f9a2e5093a619e4f2 Oct 31 01:00:00.906519 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:00.906495 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-0da668-29890140-tkrfp"] Oct 31 01:00:00.941192 ip-10-0-137-56 kubenswrapper[5222]: W1031 01:00:00.941156 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a6b45a6_056a_4396_a140_4336f6a6179b.slice/crio-12596be7149da9a20dec7db6dcf70f033dcd86fbe9cb2ad410b4152cf0dd5c6f WatchSource:0}: Error finding container 12596be7149da9a20dec7db6dcf70f033dcd86fbe9cb2ad410b4152cf0dd5c6f: Status 404 returned error can't find the container with id 12596be7149da9a20dec7db6dcf70f033dcd86fbe9cb2ad410b4152cf0dd5c6f Oct 31 01:00:00.943835 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:00.943814 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4b86e4-29890140-lvlwb"] Oct 31 01:00:00.990908 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:00.990880 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890140-978j5"] Oct 31 01:00:01.040317 ip-10-0-137-56 kubenswrapper[5222]: W1031 01:00:01.040275 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0399f8bf_008c_45eb_8550_d5c86ed8f7bf.slice/crio-7794933a72bfda68008cf71d98067a428cfbb2df032f8db73add72993c8dddc6 WatchSource:0}: Error finding container 7794933a72bfda68008cf71d98067a428cfbb2df032f8db73add72993c8dddc6: Status 404 returned error can't find the container with id 7794933a72bfda68008cf71d98067a428cfbb2df032f8db73add72993c8dddc6 Oct 31 01:00:01.743328 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:01.743293 5222 generic.go:356] "Generic (PLEG): container finished" podID="1e0d5e52-a885-4074-b8d5-e146eb6061a6" containerID="a8a30b30b438a3be67c844d0567ab67e0571b6681a143cc678f9bfc4a56fe423" exitCode=60 Oct 31 01:00:01.743800 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:01.743404 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-0da668-29890140-tkrfp" event={"ID":"1e0d5e52-a885-4074-b8d5-e146eb6061a6","Type":"ContainerDied","Data":"a8a30b30b438a3be67c844d0567ab67e0571b6681a143cc678f9bfc4a56fe423"} Oct 31 01:00:01.743800 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:01.743460 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-0da668-29890140-tkrfp" event={"ID":"1e0d5e52-a885-4074-b8d5-e146eb6061a6","Type":"ContainerStarted","Data":"aed826cf672d110e144a99d831f69aafba06755190f71c7f9a2e5093a619e4f2"} Oct 31 01:00:01.743800 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:01.743644 5222 scope.go:122] "RemoveContainer" containerID="a8a30b30b438a3be67c844d0567ab67e0571b6681a143cc678f9bfc4a56fe423" Oct 31 01:00:01.745444 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:01.745282 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890140-978j5" event={"ID":"0399f8bf-008c-45eb-8550-d5c86ed8f7bf","Type":"ContainerStarted","Data":"951128cbba555be5c651405907a96123b1826cc1cc35115a4c150e2c2de8ae01"} Oct 31 01:00:01.745444 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:01.745334 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890140-978j5" event={"ID":"0399f8bf-008c-45eb-8550-d5c86ed8f7bf","Type":"ContainerStarted","Data":"7794933a72bfda68008cf71d98067a428cfbb2df032f8db73add72993c8dddc6"} Oct 31 01:00:01.747046 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:01.747024 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4b86e4-29890140-lvlwb" event={"ID":"9a6b45a6-056a-4396-a140-4336f6a6179b","Type":"ContainerStarted","Data":"0585721e5cf6ccaedc2ed22e7d13f41e8da7872bb521e91aa15b47bc566e6507"} Oct 31 01:00:01.747115 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:01.747077 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4b86e4-29890140-lvlwb" event={"ID":"9a6b45a6-056a-4396-a140-4336f6a6179b","Type":"ContainerStarted","Data":"12596be7149da9a20dec7db6dcf70f033dcd86fbe9cb2ad410b4152cf0dd5c6f"} Oct 31 01:00:01.984510 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:01.984447 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4b86e4-29890140-lvlwb" podStartSLOduration=1.984421996 podStartE2EDuration="1.984421996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:00:01.982211094 +0000 UTC m=+2694.453380869" watchObservedRunningTime="2026-10-31 01:00:01.984421996 +0000 UTC m=+2694.455591744" Oct 31 01:00:02.100964 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:02.100906 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890140-978j5" podStartSLOduration=2.100896471 podStartE2EDuration="2.100896471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:00:02.100654539 +0000 UTC m=+2694.571824287" watchObservedRunningTime="2026-10-31 01:00:02.100896471 +0000 UTC m=+2694.572066220" Oct 31 01:00:02.756341 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:02.756307 5222 generic.go:356] "Generic (PLEG): container finished" podID="0399f8bf-008c-45eb-8550-d5c86ed8f7bf" containerID="951128cbba555be5c651405907a96123b1826cc1cc35115a4c150e2c2de8ae01" exitCode=60 Oct 31 01:00:02.756875 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:02.756414 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890140-978j5" event={"ID":"0399f8bf-008c-45eb-8550-d5c86ed8f7bf","Type":"ContainerDied","Data":"951128cbba555be5c651405907a96123b1826cc1cc35115a4c150e2c2de8ae01"} Oct 31 01:00:02.756875 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:02.756701 5222 scope.go:122] "RemoveContainer" containerID="951128cbba555be5c651405907a96123b1826cc1cc35115a4c150e2c2de8ae01" Oct 31 01:00:02.758401 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:02.758384 5222 generic.go:356] "Generic (PLEG): container finished" podID="9a6b45a6-056a-4396-a140-4336f6a6179b" containerID="0585721e5cf6ccaedc2ed22e7d13f41e8da7872bb521e91aa15b47bc566e6507" exitCode=60 Oct 31 01:00:02.758506 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:02.758415 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4b86e4-29890140-lvlwb" event={"ID":"9a6b45a6-056a-4396-a140-4336f6a6179b","Type":"ContainerDied","Data":"0585721e5cf6ccaedc2ed22e7d13f41e8da7872bb521e91aa15b47bc566e6507"} Oct 31 01:00:02.758720 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:02.758695 5222 scope.go:122] "RemoveContainer" containerID="0585721e5cf6ccaedc2ed22e7d13f41e8da7872bb521e91aa15b47bc566e6507" Oct 31 01:00:02.760733 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:02.760709 5222 generic.go:356] "Generic (PLEG): container finished" podID="1e0d5e52-a885-4074-b8d5-e146eb6061a6" containerID="9ad48672cfe4fb3ef10953c4f426fb11b021c88601c2b24ed510f7e7a7286e42" exitCode=60 Oct 31 01:00:02.760824 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:02.760755 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-0da668-29890140-tkrfp" event={"ID":"1e0d5e52-a885-4074-b8d5-e146eb6061a6","Type":"ContainerDied","Data":"9ad48672cfe4fb3ef10953c4f426fb11b021c88601c2b24ed510f7e7a7286e42"} Oct 31 01:00:02.760824 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:02.760803 5222 scope.go:122] "RemoveContainer" containerID="a8a30b30b438a3be67c844d0567ab67e0571b6681a143cc678f9bfc4a56fe423" Oct 31 01:00:02.761103 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:02.761083 5222 scope.go:122] "RemoveContainer" containerID="9ad48672cfe4fb3ef10953c4f426fb11b021c88601c2b24ed510f7e7a7286e42" Oct 31 01:00:02.761320 ip-10-0-137-56 kubenswrapper[5222]: E1031 01:00:02.761301 5222 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-0da668-29890140-tkrfp_odh-ai-gateway-infra(1e0d5e52-a885-4074-b8d5-e146eb6061a6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-0da668-29890140-tkrfp" podUID="1e0d5e52-a885-4074-b8d5-e146eb6061a6" Oct 31 01:00:03.768443 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:03.768407 5222 scope.go:122] "RemoveContainer" containerID="9ad48672cfe4fb3ef10953c4f426fb11b021c88601c2b24ed510f7e7a7286e42" Oct 31 01:00:03.768906 ip-10-0-137-56 kubenswrapper[5222]: E1031 01:00:03.768637 5222 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-0da668-29890140-tkrfp_odh-ai-gateway-infra(1e0d5e52-a885-4074-b8d5-e146eb6061a6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-0da668-29890140-tkrfp" podUID="1e0d5e52-a885-4074-b8d5-e146eb6061a6" Oct 31 01:00:03.770215 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:03.770188 5222 generic.go:356] "Generic (PLEG): container finished" podID="0399f8bf-008c-45eb-8550-d5c86ed8f7bf" containerID="830604c3ca4f6c390a7b56f0f0e67a82737e2b75f080ede2d4c025c54f0e2ed0" exitCode=60 Oct 31 01:00:03.770390 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:03.770280 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890140-978j5" event={"ID":"0399f8bf-008c-45eb-8550-d5c86ed8f7bf","Type":"ContainerDied","Data":"830604c3ca4f6c390a7b56f0f0e67a82737e2b75f080ede2d4c025c54f0e2ed0"} Oct 31 01:00:03.770390 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:03.770352 5222 scope.go:122] "RemoveContainer" containerID="951128cbba555be5c651405907a96123b1826cc1cc35115a4c150e2c2de8ae01" Oct 31 01:00:03.770513 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:03.770486 5222 scope.go:122] "RemoveContainer" containerID="830604c3ca4f6c390a7b56f0f0e67a82737e2b75f080ede2d4c025c54f0e2ed0" Oct 31 01:00:03.770748 ip-10-0-137-56 kubenswrapper[5222]: E1031 01:00:03.770711 5222 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890140-978j5_odh-ai-gateway-infra(0399f8bf-008c-45eb-8550-d5c86ed8f7bf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890140-978j5" podUID="0399f8bf-008c-45eb-8550-d5c86ed8f7bf" Oct 31 01:00:03.772198 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:03.772174 5222 generic.go:356] "Generic (PLEG): container finished" podID="9a6b45a6-056a-4396-a140-4336f6a6179b" containerID="2cfd469d870d1e9d25ce3dcaf06074ecb0f2ce3527874d5a7501729df1ea2ce2" exitCode=60 Oct 31 01:00:03.772292 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:03.772244 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4b86e4-29890140-lvlwb" event={"ID":"9a6b45a6-056a-4396-a140-4336f6a6179b","Type":"ContainerDied","Data":"2cfd469d870d1e9d25ce3dcaf06074ecb0f2ce3527874d5a7501729df1ea2ce2"} Oct 31 01:00:03.772493 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:03.772479 5222 scope.go:122] "RemoveContainer" containerID="2cfd469d870d1e9d25ce3dcaf06074ecb0f2ce3527874d5a7501729df1ea2ce2" Oct 31 01:00:03.772708 ip-10-0-137-56 kubenswrapper[5222]: E1031 01:00:03.772688 5222 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-4b86e4-29890140-lvlwb_odh-ai-gateway-infra(9a6b45a6-056a-4396-a140-4336f6a6179b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4b86e4-29890140-lvlwb" podUID="9a6b45a6-056a-4396-a140-4336f6a6179b" Oct 31 01:00:03.782838 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:03.782817 5222 scope.go:122] "RemoveContainer" containerID="0585721e5cf6ccaedc2ed22e7d13f41e8da7872bb521e91aa15b47bc566e6507" Oct 31 01:00:04.074014 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:04.073929 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54259ec6-rsllx"] Oct 31 01:00:04.077540 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:04.077514 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e83b6db1-gmc99"] Oct 31 01:00:04.082974 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:04.082950 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e83b6db1-gmc99"] Oct 31 01:00:04.085097 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:04.085076 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54259ec6-rsllx"] Oct 31 01:00:04.579343 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:04.579304 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-4b86e4-5f8755b487-kp5qc"] Oct 31 01:00:04.676772 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:04.676733 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-4b86e4-6c67bcf5f-8tb44"] Oct 31 01:00:04.783685 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:04.783659 5222 scope.go:122] "RemoveContainer" containerID="830604c3ca4f6c390a7b56f0f0e67a82737e2b75f080ede2d4c025c54f0e2ed0" Oct 31 01:00:04.784177 ip-10-0-137-56 kubenswrapper[5222]: E1031 01:00:04.783975 5222 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890140-978j5_odh-ai-gateway-infra(0399f8bf-008c-45eb-8550-d5c86ed8f7bf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890140-978j5" podUID="0399f8bf-008c-45eb-8550-d5c86ed8f7bf" Oct 31 01:00:04.785393 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:04.785358 5222 scope.go:122] "RemoveContainer" containerID="2cfd469d870d1e9d25ce3dcaf06074ecb0f2ce3527874d5a7501729df1ea2ce2" Oct 31 01:00:04.785625 ip-10-0-137-56 kubenswrapper[5222]: E1031 01:00:04.785605 5222 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-4b86e4-29890140-lvlwb_odh-ai-gateway-infra(9a6b45a6-056a-4396-a140-4336f6a6179b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4b86e4-29890140-lvlwb" podUID="9a6b45a6-056a-4396-a140-4336f6a6179b" Oct 31 01:00:05.655363 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:05.655330 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10bf9d7a-5f72-4d13-a167-38cc2b204282" path="/var/lib/kubelet/pods/10bf9d7a-5f72-4d13-a167-38cc2b204282/volumes" Oct 31 01:00:05.655772 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:05.655758 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="328b5ab8-81eb-4f94-8210-a0f390052a86" path="/var/lib/kubelet/pods/328b5ab8-81eb-4f94-8210-a0f390052a86/volumes" Oct 31 01:00:05.890553 ip-10-0-137-56 kubenswrapper[5222]: E1031 01:00:05.890514 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 01:00:08.091930 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:08.091901 5222 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h9hsf_0f8c3c03-a1fc-4376-98c6-feeeefa08068/ovn-acl-logging/0.log" Oct 31 01:00:08.102341 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:08.102314 5222 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h9hsf_0f8c3c03-a1fc-4376-98c6-feeeefa08068/ovn-acl-logging/0.log" Oct 31 01:00:09.580603 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:09.580553 5222 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-4b86e4-5f8755b487-kp5qc" podUID="b269e17a-5cf4-4d8a-9668-2587522d25da" containerName="payload-processing" containerID="cri-o://e03dbf0baafa9097030f3ac9822f5878a21be45385643db121b15dbc138f3882" gracePeriod=25 Oct 31 01:00:09.677303 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:09.677226 5222 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-4b86e4-6c67bcf5f-8tb44" podUID="c94d94ac-1fb6-45d2-a286-5a0d8542ad55" containerName="payload-pre-processing" containerID="cri-o://bc5e088a3c680693be801212e9d5c2b3fdcc00beafdef804bb044789d839ac20" gracePeriod=25 Oct 31 01:00:09.817134 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:09.817092 5222 generic.go:356] "Generic (PLEG): container finished" podID="b269e17a-5cf4-4d8a-9668-2587522d25da" containerID="e03dbf0baafa9097030f3ac9822f5878a21be45385643db121b15dbc138f3882" exitCode=0 Oct 31 01:00:09.817312 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:09.817178 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-4b86e4-5f8755b487-kp5qc" event={"ID":"b269e17a-5cf4-4d8a-9668-2587522d25da","Type":"ContainerDied","Data":"e03dbf0baafa9097030f3ac9822f5878a21be45385643db121b15dbc138f3882"} Oct 31 01:00:09.819066 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:09.818886 5222 generic.go:356] "Generic (PLEG): container finished" podID="c94d94ac-1fb6-45d2-a286-5a0d8542ad55" containerID="bc5e088a3c680693be801212e9d5c2b3fdcc00beafdef804bb044789d839ac20" exitCode=0 Oct 31 01:00:09.819066 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:09.819014 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-4b86e4-6c67bcf5f-8tb44" event={"ID":"c94d94ac-1fb6-45d2-a286-5a0d8542ad55","Type":"ContainerDied","Data":"bc5e088a3c680693be801212e9d5c2b3fdcc00beafdef804bb044789d839ac20"} Oct 31 01:00:09.933075 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:09.933045 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-4b86e4-5f8755b487-kp5qc" Oct 31 01:00:09.966634 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:09.966609 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-4b86e4-6c67bcf5f-8tb44" Oct 31 01:00:10.072386 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:10.072335 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b269e17a-5cf4-4d8a-9668-2587522d25da-kube-api-access-bdj8p\" (UniqueName: \"kubernetes.io/projected/b269e17a-5cf4-4d8a-9668-2587522d25da-kube-api-access-bdj8p\") pod \"b269e17a-5cf4-4d8a-9668-2587522d25da\" (UID: \"b269e17a-5cf4-4d8a-9668-2587522d25da\") " Oct 31 01:00:10.072607 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:10.072422 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b269e17a-5cf4-4d8a-9668-2587522d25da-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b269e17a-5cf4-4d8a-9668-2587522d25da-plugins-config-volume\") pod \"b269e17a-5cf4-4d8a-9668-2587522d25da\" (UID: \"b269e17a-5cf4-4d8a-9668-2587522d25da\") " Oct 31 01:00:10.072607 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:10.072527 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c94d94ac-1fb6-45d2-a286-5a0d8542ad55-kube-api-access-bb6mk\" (UniqueName: \"kubernetes.io/projected/c94d94ac-1fb6-45d2-a286-5a0d8542ad55-kube-api-access-bb6mk\") pod \"c94d94ac-1fb6-45d2-a286-5a0d8542ad55\" (UID: \"c94d94ac-1fb6-45d2-a286-5a0d8542ad55\") " Oct 31 01:00:10.072607 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:10.072592 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c94d94ac-1fb6-45d2-a286-5a0d8542ad55-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c94d94ac-1fb6-45d2-a286-5a0d8542ad55-plugins-config-volume\") pod \"c94d94ac-1fb6-45d2-a286-5a0d8542ad55\" (UID: \"c94d94ac-1fb6-45d2-a286-5a0d8542ad55\") " Oct 31 01:00:10.072909 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:10.072866 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b269e17a-5cf4-4d8a-9668-2587522d25da-plugins-config-volume" pod "b269e17a-5cf4-4d8a-9668-2587522d25da" (UID: "b269e17a-5cf4-4d8a-9668-2587522d25da"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:00:10.073218 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:10.073195 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c94d94ac-1fb6-45d2-a286-5a0d8542ad55-plugins-config-volume" pod "c94d94ac-1fb6-45d2-a286-5a0d8542ad55" (UID: "c94d94ac-1fb6-45d2-a286-5a0d8542ad55"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:00:10.074737 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:10.074716 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b269e17a-5cf4-4d8a-9668-2587522d25da-kube-api-access-bdj8p" pod "b269e17a-5cf4-4d8a-9668-2587522d25da" (UID: "b269e17a-5cf4-4d8a-9668-2587522d25da"). InnerVolumeSpecName "kube-api-access-bdj8p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:00:10.075178 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:10.075159 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c94d94ac-1fb6-45d2-a286-5a0d8542ad55-kube-api-access-bb6mk" pod "c94d94ac-1fb6-45d2-a286-5a0d8542ad55" (UID: "c94d94ac-1fb6-45d2-a286-5a0d8542ad55"). InnerVolumeSpecName "kube-api-access-bb6mk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:00:10.174112 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:10.174034 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bb6mk\" (UniqueName: \"kubernetes.io/projected/c94d94ac-1fb6-45d2-a286-5a0d8542ad55-kube-api-access-bb6mk\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:00:10.174112 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:10.174072 5222 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c94d94ac-1fb6-45d2-a286-5a0d8542ad55-plugins-config-volume\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:00:10.174112 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:10.174086 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdj8p\" (UniqueName: \"kubernetes.io/projected/b269e17a-5cf4-4d8a-9668-2587522d25da-kube-api-access-bdj8p\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:00:10.174112 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:10.174095 5222 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b269e17a-5cf4-4d8a-9668-2587522d25da-plugins-config-volume\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:00:10.183903 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:10.183872 5222 scope.go:122] "RemoveContainer" containerID="bc5e088a3c680693be801212e9d5c2b3fdcc00beafdef804bb044789d839ac20" Oct 31 01:00:10.196201 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:10.196169 5222 scope.go:122] "RemoveContainer" containerID="d1c36e9a8e07acc99fdecc76b40d3b3e758987c5e3aff0c47ec5138158610128" Oct 31 01:00:10.206624 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:10.206598 5222 scope.go:122] "RemoveContainer" containerID="e03dbf0baafa9097030f3ac9822f5878a21be45385643db121b15dbc138f3882" Oct 31 01:00:10.216520 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:10.216497 5222 scope.go:122] "RemoveContainer" containerID="50ec86e9147149f85040de005652fff27e877a83b824e7ef338ef624eb545c14" Oct 31 01:00:10.834927 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:10.834883 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-4b86e4-6c67bcf5f-8tb44" event={"ID":"c94d94ac-1fb6-45d2-a286-5a0d8542ad55","Type":"ContainerDied","Data":"e6965452b1c0c81c8f7e808f9624f30594e9d7784dcfb8a53c86bca91dece724"} Oct 31 01:00:10.835480 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:10.834964 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-4b86e4-5f8755b487-kp5qc" event={"ID":"b269e17a-5cf4-4d8a-9668-2587522d25da","Type":"ContainerDied","Data":"5e8654d2b8ba5adebed2102ec387f6b08e247c0dde94607bbd6ba5e6d5575711"} Oct 31 01:00:10.835480 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:10.834980 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-4b86e4-6c67bcf5f-8tb44" Oct 31 01:00:10.835480 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:10.834995 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-4b86e4-5f8755b487-kp5qc" Oct 31 01:00:11.068626 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:11.068588 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-4b86e4-6c67bcf5f-8tb44"] Oct 31 01:00:11.072861 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:11.072830 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-4b86e4-6c67bcf5f-8tb44"] Oct 31 01:00:11.250073 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:11.250039 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-4b86e4-5f8755b487-kp5qc"] Oct 31 01:00:11.255836 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:11.255801 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-4b86e4-5f8755b487-kp5qc"] Oct 31 01:00:11.656736 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:11.656701 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b269e17a-5cf4-4d8a-9668-2587522d25da" path="/var/lib/kubelet/pods/b269e17a-5cf4-4d8a-9668-2587522d25da/volumes" Oct 31 01:00:11.657090 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:11.657077 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c94d94ac-1fb6-45d2-a286-5a0d8542ad55" path="/var/lib/kubelet/pods/c94d94ac-1fb6-45d2-a286-5a0d8542ad55/volumes" Oct 31 01:00:14.649103 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:14.649021 5222 scope.go:122] "RemoveContainer" containerID="9ad48672cfe4fb3ef10953c4f426fb11b021c88601c2b24ed510f7e7a7286e42" Oct 31 01:00:15.053721 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:15.053688 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32633ca9-nmg42"] Oct 31 01:00:15.064146 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:15.064116 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32633ca9-nmg42"] Oct 31 01:00:15.158852 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:15.158812 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4b86e4-29890140-lvlwb"] Oct 31 01:00:15.318988 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:15.318961 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4b86e4-29890140-lvlwb" Oct 31 01:00:15.426382 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:15.426329 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a6b45a6-056a-4396-a140-4336f6a6179b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a6b45a6-056a-4396-a140-4336f6a6179b-maas-api-service-ca\") pod \"9a6b45a6-056a-4396-a140-4336f6a6179b\" (UID: \"9a6b45a6-056a-4396-a140-4336f6a6179b\") " Oct 31 01:00:15.426607 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:15.426515 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a6b45a6-056a-4396-a140-4336f6a6179b-kube-api-access-ffqx9\" (UniqueName: \"kubernetes.io/projected/9a6b45a6-056a-4396-a140-4336f6a6179b-kube-api-access-ffqx9\") pod \"9a6b45a6-056a-4396-a140-4336f6a6179b\" (UID: \"9a6b45a6-056a-4396-a140-4336f6a6179b\") " Oct 31 01:00:15.426774 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:15.426742 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a6b45a6-056a-4396-a140-4336f6a6179b-maas-api-service-ca" pod "9a6b45a6-056a-4396-a140-4336f6a6179b" (UID: "9a6b45a6-056a-4396-a140-4336f6a6179b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:00:15.428761 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:15.428732 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a6b45a6-056a-4396-a140-4336f6a6179b-kube-api-access-ffqx9" pod "9a6b45a6-056a-4396-a140-4336f6a6179b" (UID: "9a6b45a6-056a-4396-a140-4336f6a6179b"). InnerVolumeSpecName "kube-api-access-ffqx9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:00:15.527970 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:15.527935 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffqx9\" (UniqueName: \"kubernetes.io/projected/9a6b45a6-056a-4396-a140-4336f6a6179b-kube-api-access-ffqx9\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:00:15.527970 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:15.527969 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a6b45a6-056a-4396-a140-4336f6a6179b-maas-api-service-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:00:15.655049 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:15.655014 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23151b79-282e-4b50-9e17-0dd63053fd4d" path="/var/lib/kubelet/pods/23151b79-282e-4b50-9e17-0dd63053fd4d/volumes" Oct 31 01:00:15.868802 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:15.868706 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4b86e4-29890140-lvlwb" Oct 31 01:00:15.868802 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:15.868719 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4b86e4-29890140-lvlwb" event={"ID":"9a6b45a6-056a-4396-a140-4336f6a6179b","Type":"ContainerDied","Data":"12596be7149da9a20dec7db6dcf70f033dcd86fbe9cb2ad410b4152cf0dd5c6f"} Oct 31 01:00:15.869053 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:15.868817 5222 scope.go:122] "RemoveContainer" containerID="2cfd469d870d1e9d25ce3dcaf06074ecb0f2ce3527874d5a7501729df1ea2ce2" Oct 31 01:00:15.871211 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:15.871185 5222 generic.go:356] "Generic (PLEG): container finished" podID="1e0d5e52-a885-4074-b8d5-e146eb6061a6" containerID="72f3cfd4ddbdf0175fb19570c5ac34a69fad5a66149038e0c2df57ca4a890de2" exitCode=60 Oct 31 01:00:15.871343 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:15.871326 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-0da668-29890140-tkrfp" event={"ID":"1e0d5e52-a885-4074-b8d5-e146eb6061a6","Type":"ContainerDied","Data":"72f3cfd4ddbdf0175fb19570c5ac34a69fad5a66149038e0c2df57ca4a890de2"} Oct 31 01:00:15.871762 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:15.871744 5222 scope.go:122] "RemoveContainer" containerID="72f3cfd4ddbdf0175fb19570c5ac34a69fad5a66149038e0c2df57ca4a890de2" Oct 31 01:00:15.872034 ip-10-0-137-56 kubenswrapper[5222]: E1031 01:00:15.872010 5222 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-0da668-29890140-tkrfp_odh-ai-gateway-infra(1e0d5e52-a885-4074-b8d5-e146eb6061a6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-0da668-29890140-tkrfp" podUID="1e0d5e52-a885-4074-b8d5-e146eb6061a6" Oct 31 01:00:15.878584 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:15.878564 5222 scope.go:122] "RemoveContainer" containerID="9ad48672cfe4fb3ef10953c4f426fb11b021c88601c2b24ed510f7e7a7286e42" Oct 31 01:00:16.045282 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:16.045253 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4b86e4-29890140-lvlwb"] Oct 31 01:00:16.049552 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:16.049529 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4b86e4-29890140-lvlwb"] Oct 31 01:00:16.649197 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:16.649167 5222 scope.go:122] "RemoveContainer" containerID="830604c3ca4f6c390a7b56f0f0e67a82737e2b75f080ede2d4c025c54f0e2ed0" Oct 31 01:00:17.176865 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:17.176829 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-0da668-29890140-tkrfp"] Oct 31 01:00:17.218247 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:17.218131 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-4b86e4-bfsqp"] Oct 31 01:00:17.228060 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:17.228029 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-4b86e4-bfsqp"] Oct 31 01:00:17.323429 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:17.323397 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-0da668-29890140-tkrfp" Oct 31 01:00:17.448801 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:17.448763 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e0d5e52-a885-4074-b8d5-e146eb6061a6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e0d5e52-a885-4074-b8d5-e146eb6061a6-maas-api-service-ca\") pod \"1e0d5e52-a885-4074-b8d5-e146eb6061a6\" (UID: \"1e0d5e52-a885-4074-b8d5-e146eb6061a6\") " Oct 31 01:00:17.448971 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:17.448883 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e0d5e52-a885-4074-b8d5-e146eb6061a6-kube-api-access-vpw5f\" (UniqueName: \"kubernetes.io/projected/1e0d5e52-a885-4074-b8d5-e146eb6061a6-kube-api-access-vpw5f\") pod \"1e0d5e52-a885-4074-b8d5-e146eb6061a6\" (UID: \"1e0d5e52-a885-4074-b8d5-e146eb6061a6\") " Oct 31 01:00:17.449174 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:17.449148 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e0d5e52-a885-4074-b8d5-e146eb6061a6-maas-api-service-ca" pod "1e0d5e52-a885-4074-b8d5-e146eb6061a6" (UID: "1e0d5e52-a885-4074-b8d5-e146eb6061a6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:00:17.451477 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:17.451451 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e0d5e52-a885-4074-b8d5-e146eb6061a6-kube-api-access-vpw5f" pod "1e0d5e52-a885-4074-b8d5-e146eb6061a6" (UID: "1e0d5e52-a885-4074-b8d5-e146eb6061a6"). InnerVolumeSpecName "kube-api-access-vpw5f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:00:17.549894 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:17.549856 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e0d5e52-a885-4074-b8d5-e146eb6061a6-maas-api-service-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:00:17.549894 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:17.549887 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpw5f\" (UniqueName: \"kubernetes.io/projected/1e0d5e52-a885-4074-b8d5-e146eb6061a6-kube-api-access-vpw5f\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:00:17.655280 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:17.655244 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a6b45a6-056a-4396-a140-4336f6a6179b" path="/var/lib/kubelet/pods/9a6b45a6-056a-4396-a140-4336f6a6179b/volumes" Oct 31 01:00:17.655665 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:17.655651 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d38e33c1-eccd-41cd-9cc6-e5eb7a5eb48b" path="/var/lib/kubelet/pods/d38e33c1-eccd-41cd-9cc6-e5eb7a5eb48b/volumes" Oct 31 01:00:17.888697 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:17.888615 5222 generic.go:356] "Generic (PLEG): container finished" podID="0399f8bf-008c-45eb-8550-d5c86ed8f7bf" containerID="da56dd27aa964455b209470de01ecd875006e3f13a8314b6fae68443c9514720" exitCode=60 Oct 31 01:00:17.888697 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:17.888684 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890140-978j5" event={"ID":"0399f8bf-008c-45eb-8550-d5c86ed8f7bf","Type":"ContainerDied","Data":"da56dd27aa964455b209470de01ecd875006e3f13a8314b6fae68443c9514720"} Oct 31 01:00:17.888935 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:17.888746 5222 scope.go:122] "RemoveContainer" containerID="830604c3ca4f6c390a7b56f0f0e67a82737e2b75f080ede2d4c025c54f0e2ed0" Oct 31 01:00:17.889143 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:17.889125 5222 scope.go:122] "RemoveContainer" containerID="da56dd27aa964455b209470de01ecd875006e3f13a8314b6fae68443c9514720" Oct 31 01:00:17.889455 ip-10-0-137-56 kubenswrapper[5222]: E1031 01:00:17.889434 5222 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29890140-978j5_odh-ai-gateway-infra(0399f8bf-008c-45eb-8550-d5c86ed8f7bf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890140-978j5" podUID="0399f8bf-008c-45eb-8550-d5c86ed8f7bf" Oct 31 01:00:17.890408 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:17.890391 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-0da668-29890140-tkrfp" Oct 31 01:00:17.890501 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:17.890411 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-0da668-29890140-tkrfp" event={"ID":"1e0d5e52-a885-4074-b8d5-e146eb6061a6","Type":"ContainerDied","Data":"aed826cf672d110e144a99d831f69aafba06755190f71c7f9a2e5093a619e4f2"} Oct 31 01:00:17.901750 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:17.901725 5222 scope.go:122] "RemoveContainer" containerID="72f3cfd4ddbdf0175fb19570c5ac34a69fad5a66149038e0c2df57ca4a890de2" Oct 31 01:00:18.121458 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:18.121423 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-0da668-29890140-tkrfp"] Oct 31 01:00:18.125611 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:18.125582 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-0da668-29890140-tkrfp"] Oct 31 01:00:19.020702 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:19.020670 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890140-978j5"] Oct 31 01:00:19.161931 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:19.161905 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890140-978j5" Oct 31 01:00:19.266588 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:19.266550 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0399f8bf-008c-45eb-8550-d5c86ed8f7bf-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0399f8bf-008c-45eb-8550-d5c86ed8f7bf-maas-api-service-ca\") pod \"0399f8bf-008c-45eb-8550-d5c86ed8f7bf\" (UID: \"0399f8bf-008c-45eb-8550-d5c86ed8f7bf\") " Oct 31 01:00:19.266772 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:19.266709 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0399f8bf-008c-45eb-8550-d5c86ed8f7bf-kube-api-access-rsr8m\" (UniqueName: \"kubernetes.io/projected/0399f8bf-008c-45eb-8550-d5c86ed8f7bf-kube-api-access-rsr8m\") pod \"0399f8bf-008c-45eb-8550-d5c86ed8f7bf\" (UID: \"0399f8bf-008c-45eb-8550-d5c86ed8f7bf\") " Oct 31 01:00:19.266934 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:19.266909 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0399f8bf-008c-45eb-8550-d5c86ed8f7bf-maas-api-service-ca" pod "0399f8bf-008c-45eb-8550-d5c86ed8f7bf" (UID: "0399f8bf-008c-45eb-8550-d5c86ed8f7bf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:00:19.267074 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:19.267061 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0399f8bf-008c-45eb-8550-d5c86ed8f7bf-maas-api-service-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:00:19.269072 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:19.269047 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0399f8bf-008c-45eb-8550-d5c86ed8f7bf-kube-api-access-rsr8m" pod "0399f8bf-008c-45eb-8550-d5c86ed8f7bf" (UID: "0399f8bf-008c-45eb-8550-d5c86ed8f7bf"). InnerVolumeSpecName "kube-api-access-rsr8m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:00:19.367910 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:19.367872 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsr8m\" (UniqueName: \"kubernetes.io/projected/0399f8bf-008c-45eb-8550-d5c86ed8f7bf-kube-api-access-rsr8m\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:00:19.654939 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:19.654863 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e0d5e52-a885-4074-b8d5-e146eb6061a6" path="/var/lib/kubelet/pods/1e0d5e52-a885-4074-b8d5-e146eb6061a6/volumes" Oct 31 01:00:19.906854 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:19.906774 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890140-978j5" Oct 31 01:00:19.906854 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:19.906788 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890140-978j5" event={"ID":"0399f8bf-008c-45eb-8550-d5c86ed8f7bf","Type":"ContainerDied","Data":"7794933a72bfda68008cf71d98067a428cfbb2df032f8db73add72993c8dddc6"} Oct 31 01:00:19.906854 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:19.906847 5222 scope.go:122] "RemoveContainer" containerID="da56dd27aa964455b209470de01ecd875006e3f13a8314b6fae68443c9514720" Oct 31 01:00:20.076781 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:20.076743 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890140-978j5"] Oct 31 01:00:20.128180 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:20.128144 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890140-978j5"] Oct 31 01:00:20.734560 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:20.734517 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-79tk8"] Oct 31 01:00:21.657791 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:21.657759 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0399f8bf-008c-45eb-8550-d5c86ed8f7bf" path="/var/lib/kubelet/pods/0399f8bf-008c-45eb-8550-d5c86ed8f7bf/volumes" Oct 31 01:00:25.007599 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:25.007565 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-0da668-7mkrn"] Oct 31 01:00:25.008006 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:25.007987 5222 cpu_manager.go:414] "containerMap: removing container" podUID="c94d94ac-1fb6-45d2-a286-5a0d8542ad55" containerName="payload-pre-processing" Oct 31 01:00:25.008006 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:25.007998 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c94d94ac-1fb6-45d2-a286-5a0d8542ad55" containerName="payload-pre-processing" Oct 31 01:00:25.008087 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:25.008017 5222 cpu_manager.go:414] "containerMap: removing container" podUID="1e0d5e52-a885-4074-b8d5-e146eb6061a6" containerName="cleanup" Oct 31 01:00:25.008087 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:25.008022 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e0d5e52-a885-4074-b8d5-e146eb6061a6" containerName="cleanup" Oct 31 01:00:25.008087 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:25.008030 5222 cpu_manager.go:414] "containerMap: removing container" podUID="b269e17a-5cf4-4d8a-9668-2587522d25da" containerName="payload-processing" Oct 31 01:00:25.008087 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:25.008035 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b269e17a-5cf4-4d8a-9668-2587522d25da" containerName="payload-processing" Oct 31 01:00:25.008087 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:25.008042 5222 cpu_manager.go:414] "containerMap: removing container" podUID="9a6b45a6-056a-4396-a140-4336f6a6179b" containerName="cleanup" Oct 31 01:00:25.008087 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:25.008046 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a6b45a6-056a-4396-a140-4336f6a6179b" containerName="cleanup" Oct 31 01:00:25.008087 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:25.008053 5222 cpu_manager.go:414] "containerMap: removing container" podUID="1e0d5e52-a885-4074-b8d5-e146eb6061a6" containerName="cleanup" Oct 31 01:00:25.008087 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:25.008057 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e0d5e52-a885-4074-b8d5-e146eb6061a6" containerName="cleanup" Oct 31 01:00:25.008087 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:25.008065 5222 cpu_manager.go:414] "containerMap: removing container" podUID="1e0d5e52-a885-4074-b8d5-e146eb6061a6" containerName="cleanup" Oct 31 01:00:25.008087 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:25.008070 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e0d5e52-a885-4074-b8d5-e146eb6061a6" containerName="cleanup" Oct 31 01:00:25.008087 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:25.008081 5222 cpu_manager.go:414] "containerMap: removing container" podUID="0399f8bf-008c-45eb-8550-d5c86ed8f7bf" containerName="cleanup" Oct 31 01:00:25.008087 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:25.008085 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0399f8bf-008c-45eb-8550-d5c86ed8f7bf" containerName="cleanup" Oct 31 01:00:25.008527 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:25.008095 5222 cpu_manager.go:414] "containerMap: removing container" podUID="9a6b45a6-056a-4396-a140-4336f6a6179b" containerName="cleanup" Oct 31 01:00:25.008527 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:25.008099 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a6b45a6-056a-4396-a140-4336f6a6179b" containerName="cleanup" Oct 31 01:00:25.008527 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:25.008107 5222 cpu_manager.go:414] "containerMap: removing container" podUID="0399f8bf-008c-45eb-8550-d5c86ed8f7bf" containerName="cleanup" Oct 31 01:00:25.008527 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:25.008111 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0399f8bf-008c-45eb-8550-d5c86ed8f7bf" containerName="cleanup" Oct 31 01:00:25.008527 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:25.008117 5222 cpu_manager.go:414] "containerMap: removing container" podUID="0399f8bf-008c-45eb-8550-d5c86ed8f7bf" containerName="cleanup" Oct 31 01:00:25.008527 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:25.008121 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0399f8bf-008c-45eb-8550-d5c86ed8f7bf" containerName="cleanup" Oct 31 01:00:25.008527 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:25.008175 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e0d5e52-a885-4074-b8d5-e146eb6061a6" containerName="cleanup" Oct 31 01:00:25.008527 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:25.008182 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e0d5e52-a885-4074-b8d5-e146eb6061a6" containerName="cleanup" Oct 31 01:00:25.008527 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:25.008191 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="0399f8bf-008c-45eb-8550-d5c86ed8f7bf" containerName="cleanup" Oct 31 01:00:25.008527 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:25.008197 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a6b45a6-056a-4396-a140-4336f6a6179b" containerName="cleanup" Oct 31 01:00:25.008527 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:25.008203 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="0399f8bf-008c-45eb-8550-d5c86ed8f7bf" containerName="cleanup" Oct 31 01:00:25.008527 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:25.008208 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="0399f8bf-008c-45eb-8550-d5c86ed8f7bf" containerName="cleanup" Oct 31 01:00:25.008527 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:25.008217 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="c94d94ac-1fb6-45d2-a286-5a0d8542ad55" containerName="payload-pre-processing" Oct 31 01:00:25.008527 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:25.008225 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="b269e17a-5cf4-4d8a-9668-2587522d25da" containerName="payload-processing" Oct 31 01:00:25.008527 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:25.008230 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a6b45a6-056a-4396-a140-4336f6a6179b" containerName="cleanup" Oct 31 01:00:25.012741 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:25.012723 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-0da668-7mkrn" Oct 31 01:00:25.029994 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:25.029961 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33166625-f2d8-46d9-8ee1-1e61a8fc38bb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-0da668-7mkrn\" (UID: \"33166625-f2d8-46d9-8ee1-1e61a8fc38bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-0da668-7mkrn" Oct 31 01:00:25.030164 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:25.030053 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33166625-f2d8-46d9-8ee1-1e61a8fc38bb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-0da668-7mkrn\" (UID: \"33166625-f2d8-46d9-8ee1-1e61a8fc38bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-0da668-7mkrn" Oct 31 01:00:25.030164 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:25.030071 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4lh6\" (UniqueName: \"kubernetes.io/projected/33166625-f2d8-46d9-8ee1-1e61a8fc38bb-kube-api-access-w4lh6\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-0da668-7mkrn\" (UID: \"33166625-f2d8-46d9-8ee1-1e61a8fc38bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-0da668-7mkrn" Oct 31 01:00:25.064905 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:25.064872 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-0da668-7mkrn"] Oct 31 01:00:25.114294 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:25.114251 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:00:25.114294 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:25.114271 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:00:25.114294 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:25.114270 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:00:25.114613 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:25.114360 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfz9z\"" Oct 31 01:00:25.130961 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:25.130928 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33166625-f2d8-46d9-8ee1-1e61a8fc38bb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-0da668-7mkrn\" (UID: \"33166625-f2d8-46d9-8ee1-1e61a8fc38bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-0da668-7mkrn" Oct 31 01:00:25.131098 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:25.130965 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4lh6\" (UniqueName: \"kubernetes.io/projected/33166625-f2d8-46d9-8ee1-1e61a8fc38bb-kube-api-access-w4lh6\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-0da668-7mkrn\" (UID: \"33166625-f2d8-46d9-8ee1-1e61a8fc38bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-0da668-7mkrn" Oct 31 01:00:25.131098 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:25.131047 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33166625-f2d8-46d9-8ee1-1e61a8fc38bb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-0da668-7mkrn\" (UID: \"33166625-f2d8-46d9-8ee1-1e61a8fc38bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-0da668-7mkrn" Oct 31 01:00:25.131711 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:25.131683 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33166625-f2d8-46d9-8ee1-1e61a8fc38bb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-0da668-7mkrn\" (UID: \"33166625-f2d8-46d9-8ee1-1e61a8fc38bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-0da668-7mkrn" Oct 31 01:00:25.133664 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:25.133642 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33166625-f2d8-46d9-8ee1-1e61a8fc38bb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-0da668-7mkrn\" (UID: \"33166625-f2d8-46d9-8ee1-1e61a8fc38bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-0da668-7mkrn" Oct 31 01:00:25.194018 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:25.193981 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4lh6\" (UniqueName: \"kubernetes.io/projected/33166625-f2d8-46d9-8ee1-1e61a8fc38bb-kube-api-access-w4lh6\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-0da668-7mkrn\" (UID: \"33166625-f2d8-46d9-8ee1-1e61a8fc38bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-0da668-7mkrn" Oct 31 01:00:25.326180 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:25.326142 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-0da668-7mkrn" Oct 31 01:00:25.697693 ip-10-0-137-56 kubenswrapper[5222]: W1031 01:00:25.697654 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33166625_f2d8_46d9_8ee1_1e61a8fc38bb.slice/crio-2403934f9066dd36349fb75d5fc54ad5c9a526f12ef396e7683f8667c01ed31a WatchSource:0}: Error finding container 2403934f9066dd36349fb75d5fc54ad5c9a526f12ef396e7683f8667c01ed31a: Status 404 returned error can't find the container with id 2403934f9066dd36349fb75d5fc54ad5c9a526f12ef396e7683f8667c01ed31a Oct 31 01:00:25.699855 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:25.699827 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-0da668-7mkrn"] Oct 31 01:00:25.942133 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:25.942094 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-0da668-7mkrn" event={"ID":"33166625-f2d8-46d9-8ee1-1e61a8fc38bb","Type":"ContainerStarted","Data":"2403934f9066dd36349fb75d5fc54ad5c9a526f12ef396e7683f8667c01ed31a"} Oct 31 01:00:26.949997 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:26.949897 5222 generic.go:356] "Generic (PLEG): container finished" podID="33166625-f2d8-46d9-8ee1-1e61a8fc38bb" containerID="175fd51abffffaa516b5ca835acfadf78eadb4160657d97733e36b6119d45f94" exitCode=0 Oct 31 01:00:26.949997 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:26.949982 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-0da668-7mkrn" event={"ID":"33166625-f2d8-46d9-8ee1-1e61a8fc38bb","Type":"ContainerDied","Data":"175fd51abffffaa516b5ca835acfadf78eadb4160657d97733e36b6119d45f94"} Oct 31 01:00:28.093270 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:28.093241 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-0da668-7mkrn" Oct 31 01:00:28.162721 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:28.162684 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33166625-f2d8-46d9-8ee1-1e61a8fc38bb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33166625-f2d8-46d9-8ee1-1e61a8fc38bb-maas-api-openshift-ca\") pod \"33166625-f2d8-46d9-8ee1-1e61a8fc38bb\" (UID: \"33166625-f2d8-46d9-8ee1-1e61a8fc38bb\") " Oct 31 01:00:28.162893 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:28.162782 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33166625-f2d8-46d9-8ee1-1e61a8fc38bb-kube-api-access-w4lh6\" (UniqueName: \"kubernetes.io/projected/33166625-f2d8-46d9-8ee1-1e61a8fc38bb-kube-api-access-w4lh6\") pod \"33166625-f2d8-46d9-8ee1-1e61a8fc38bb\" (UID: \"33166625-f2d8-46d9-8ee1-1e61a8fc38bb\") " Oct 31 01:00:28.162893 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:28.162854 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33166625-f2d8-46d9-8ee1-1e61a8fc38bb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33166625-f2d8-46d9-8ee1-1e61a8fc38bb-maas-api-xks-ca\") pod \"33166625-f2d8-46d9-8ee1-1e61a8fc38bb\" (UID: \"33166625-f2d8-46d9-8ee1-1e61a8fc38bb\") " Oct 31 01:00:28.163122 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:28.163095 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33166625-f2d8-46d9-8ee1-1e61a8fc38bb-maas-api-openshift-ca" pod "33166625-f2d8-46d9-8ee1-1e61a8fc38bb" (UID: "33166625-f2d8-46d9-8ee1-1e61a8fc38bb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:00:28.163231 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:28.163215 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33166625-f2d8-46d9-8ee1-1e61a8fc38bb-maas-api-openshift-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:00:28.165095 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:28.165071 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33166625-f2d8-46d9-8ee1-1e61a8fc38bb-maas-api-xks-ca" pod "33166625-f2d8-46d9-8ee1-1e61a8fc38bb" (UID: "33166625-f2d8-46d9-8ee1-1e61a8fc38bb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:00:28.165433 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:28.165417 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33166625-f2d8-46d9-8ee1-1e61a8fc38bb-kube-api-access-w4lh6" pod "33166625-f2d8-46d9-8ee1-1e61a8fc38bb" (UID: "33166625-f2d8-46d9-8ee1-1e61a8fc38bb"). InnerVolumeSpecName "kube-api-access-w4lh6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:00:28.263861 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:28.263752 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4lh6\" (UniqueName: \"kubernetes.io/projected/33166625-f2d8-46d9-8ee1-1e61a8fc38bb-kube-api-access-w4lh6\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:00:28.263861 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:28.263805 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33166625-f2d8-46d9-8ee1-1e61a8fc38bb-maas-api-xks-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:00:28.962459 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:28.962424 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-0da668-7mkrn" event={"ID":"33166625-f2d8-46d9-8ee1-1e61a8fc38bb","Type":"ContainerDied","Data":"2403934f9066dd36349fb75d5fc54ad5c9a526f12ef396e7683f8667c01ed31a"} Oct 31 01:00:28.962459 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:28.962454 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-0da668-7mkrn" Oct 31 01:00:28.962672 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:28.962477 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2403934f9066dd36349fb75d5fc54ad5c9a526f12ef396e7683f8667c01ed31a" Oct 31 01:00:35.889613 ip-10-0-137-56 kubenswrapper[5222]: E1031 01:00:35.889571 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 01:00:36.456959 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:36.456917 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-0da668-78b845796f-5srmm"] Oct 31 01:00:36.567773 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:36.567733 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-0da668-dff997c59-8nrck"] Oct 31 01:00:41.458225 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:41.458167 5222 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-0da668-78b845796f-5srmm" podUID="958dc2bf-130f-4861-981b-c5fb31d39f67" containerName="payload-processing" containerID="cri-o://a05915bea71c90a673f33bdcac902414d31334ce5369c6a55ad4ab7dbf783340" gracePeriod=25 Oct 31 01:00:41.568426 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:41.568380 5222 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-0da668-dff997c59-8nrck" podUID="8532f94b-ef23-4b4d-967f-dc2aa9e3c3dd" containerName="payload-pre-processing" containerID="cri-o://4b15be256121d29b82d6b5480b34faa7485c400e99e849b86eeb73ad610a1ffa" gracePeriod=25 Oct 31 01:00:41.733651 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:41.733620 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-0da668-78b845796f-5srmm" Oct 31 01:00:41.810869 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:41.810842 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/958dc2bf-130f-4861-981b-c5fb31d39f67-kube-api-access-ljk86\" (UniqueName: \"kubernetes.io/projected/958dc2bf-130f-4861-981b-c5fb31d39f67-kube-api-access-ljk86\") pod \"958dc2bf-130f-4861-981b-c5fb31d39f67\" (UID: \"958dc2bf-130f-4861-981b-c5fb31d39f67\") " Oct 31 01:00:41.811008 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:41.810913 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/958dc2bf-130f-4861-981b-c5fb31d39f67-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/958dc2bf-130f-4861-981b-c5fb31d39f67-plugins-config-volume\") pod \"958dc2bf-130f-4861-981b-c5fb31d39f67\" (UID: \"958dc2bf-130f-4861-981b-c5fb31d39f67\") " Oct 31 01:00:41.811316 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:41.811288 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/958dc2bf-130f-4861-981b-c5fb31d39f67-plugins-config-volume" pod "958dc2bf-130f-4861-981b-c5fb31d39f67" (UID: "958dc2bf-130f-4861-981b-c5fb31d39f67"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:00:41.813213 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:41.813190 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/958dc2bf-130f-4861-981b-c5fb31d39f67-kube-api-access-ljk86" pod "958dc2bf-130f-4861-981b-c5fb31d39f67" (UID: "958dc2bf-130f-4861-981b-c5fb31d39f67"). InnerVolumeSpecName "kube-api-access-ljk86". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:00:41.818572 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:41.818549 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-0da668-dff997c59-8nrck" Oct 31 01:00:41.912000 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:41.911963 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8532f94b-ef23-4b4d-967f-dc2aa9e3c3dd-kube-api-access-sdjwd\" (UniqueName: \"kubernetes.io/projected/8532f94b-ef23-4b4d-967f-dc2aa9e3c3dd-kube-api-access-sdjwd\") pod \"8532f94b-ef23-4b4d-967f-dc2aa9e3c3dd\" (UID: \"8532f94b-ef23-4b4d-967f-dc2aa9e3c3dd\") " Oct 31 01:00:41.912223 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:41.912020 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8532f94b-ef23-4b4d-967f-dc2aa9e3c3dd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8532f94b-ef23-4b4d-967f-dc2aa9e3c3dd-plugins-config-volume\") pod \"8532f94b-ef23-4b4d-967f-dc2aa9e3c3dd\" (UID: \"8532f94b-ef23-4b4d-967f-dc2aa9e3c3dd\") " Oct 31 01:00:41.912223 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:41.912211 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljk86\" (UniqueName: \"kubernetes.io/projected/958dc2bf-130f-4861-981b-c5fb31d39f67-kube-api-access-ljk86\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:00:41.912223 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:41.912223 5222 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/958dc2bf-130f-4861-981b-c5fb31d39f67-plugins-config-volume\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:00:41.912528 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:41.912504 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8532f94b-ef23-4b4d-967f-dc2aa9e3c3dd-plugins-config-volume" pod "8532f94b-ef23-4b4d-967f-dc2aa9e3c3dd" (UID: "8532f94b-ef23-4b4d-967f-dc2aa9e3c3dd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:00:41.914282 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:41.914260 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8532f94b-ef23-4b4d-967f-dc2aa9e3c3dd-kube-api-access-sdjwd" pod "8532f94b-ef23-4b4d-967f-dc2aa9e3c3dd" (UID: "8532f94b-ef23-4b4d-967f-dc2aa9e3c3dd"). InnerVolumeSpecName "kube-api-access-sdjwd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:00:42.013299 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:42.013206 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdjwd\" (UniqueName: \"kubernetes.io/projected/8532f94b-ef23-4b4d-967f-dc2aa9e3c3dd-kube-api-access-sdjwd\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:00:42.013299 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:42.013247 5222 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8532f94b-ef23-4b4d-967f-dc2aa9e3c3dd-plugins-config-volume\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:00:42.040437 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:42.040399 5222 generic.go:356] "Generic (PLEG): container finished" podID="958dc2bf-130f-4861-981b-c5fb31d39f67" containerID="a05915bea71c90a673f33bdcac902414d31334ce5369c6a55ad4ab7dbf783340" exitCode=0 Oct 31 01:00:42.040635 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:42.040505 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-0da668-78b845796f-5srmm" Oct 31 01:00:42.040635 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:42.040529 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-0da668-78b845796f-5srmm" event={"ID":"958dc2bf-130f-4861-981b-c5fb31d39f67","Type":"ContainerDied","Data":"a05915bea71c90a673f33bdcac902414d31334ce5369c6a55ad4ab7dbf783340"} Oct 31 01:00:42.040635 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:42.040578 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-0da668-78b845796f-5srmm" event={"ID":"958dc2bf-130f-4861-981b-c5fb31d39f67","Type":"ContainerDied","Data":"a581d3a324ac3b302c300889fd23f2c182236965363ba7f751eb3d54f9698dab"} Oct 31 01:00:42.040635 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:42.040601 5222 scope.go:122] "RemoveContainer" containerID="a05915bea71c90a673f33bdcac902414d31334ce5369c6a55ad4ab7dbf783340" Oct 31 01:00:42.042051 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:42.042024 5222 generic.go:356] "Generic (PLEG): container finished" podID="8532f94b-ef23-4b4d-967f-dc2aa9e3c3dd" containerID="4b15be256121d29b82d6b5480b34faa7485c400e99e849b86eeb73ad610a1ffa" exitCode=0 Oct 31 01:00:42.042178 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:42.042066 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-0da668-dff997c59-8nrck" Oct 31 01:00:42.042178 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:42.042079 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-0da668-dff997c59-8nrck" event={"ID":"8532f94b-ef23-4b4d-967f-dc2aa9e3c3dd","Type":"ContainerDied","Data":"4b15be256121d29b82d6b5480b34faa7485c400e99e849b86eeb73ad610a1ffa"} Oct 31 01:00:42.042178 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:42.042136 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-0da668-dff997c59-8nrck" event={"ID":"8532f94b-ef23-4b4d-967f-dc2aa9e3c3dd","Type":"ContainerDied","Data":"fbb4834e664e93ee81f780fa8b1539d8dd0adfaa8eddd5609f535649ac9eac51"} Oct 31 01:00:42.051496 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:42.051473 5222 scope.go:122] "RemoveContainer" containerID="a05915bea71c90a673f33bdcac902414d31334ce5369c6a55ad4ab7dbf783340" Oct 31 01:00:42.051907 ip-10-0-137-56 kubenswrapper[5222]: E1031 01:00:42.051879 5222 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a05915bea71c90a673f33bdcac902414d31334ce5369c6a55ad4ab7dbf783340\": container with ID starting with a05915bea71c90a673f33bdcac902414d31334ce5369c6a55ad4ab7dbf783340 not found: ID does not exist" containerID="a05915bea71c90a673f33bdcac902414d31334ce5369c6a55ad4ab7dbf783340" Oct 31 01:00:42.052015 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:42.051913 5222 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a05915bea71c90a673f33bdcac902414d31334ce5369c6a55ad4ab7dbf783340"} err="failed to get container status \"a05915bea71c90a673f33bdcac902414d31334ce5369c6a55ad4ab7dbf783340\": rpc error: code = NotFound desc = could not find container \"a05915bea71c90a673f33bdcac902414d31334ce5369c6a55ad4ab7dbf783340\": container with ID starting with a05915bea71c90a673f33bdcac902414d31334ce5369c6a55ad4ab7dbf783340 not found: ID does not exist" Oct 31 01:00:42.052015 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:42.051940 5222 scope.go:122] "RemoveContainer" containerID="4b15be256121d29b82d6b5480b34faa7485c400e99e849b86eeb73ad610a1ffa" Oct 31 01:00:42.060881 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:42.060860 5222 scope.go:122] "RemoveContainer" containerID="4b15be256121d29b82d6b5480b34faa7485c400e99e849b86eeb73ad610a1ffa" Oct 31 01:00:42.061172 ip-10-0-137-56 kubenswrapper[5222]: E1031 01:00:42.061153 5222 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b15be256121d29b82d6b5480b34faa7485c400e99e849b86eeb73ad610a1ffa\": container with ID starting with 4b15be256121d29b82d6b5480b34faa7485c400e99e849b86eeb73ad610a1ffa not found: ID does not exist" containerID="4b15be256121d29b82d6b5480b34faa7485c400e99e849b86eeb73ad610a1ffa" Oct 31 01:00:42.061219 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:42.061182 5222 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b15be256121d29b82d6b5480b34faa7485c400e99e849b86eeb73ad610a1ffa"} err="failed to get container status \"4b15be256121d29b82d6b5480b34faa7485c400e99e849b86eeb73ad610a1ffa\": rpc error: code = NotFound desc = could not find container \"4b15be256121d29b82d6b5480b34faa7485c400e99e849b86eeb73ad610a1ffa\": container with ID starting with 4b15be256121d29b82d6b5480b34faa7485c400e99e849b86eeb73ad610a1ffa not found: ID does not exist" Oct 31 01:00:42.209997 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:42.209957 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-0da668-78b845796f-5srmm"] Oct 31 01:00:42.214242 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:42.214211 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-0da668-78b845796f-5srmm"] Oct 31 01:00:42.327576 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:42.327540 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-0da668-dff997c59-8nrck"] Oct 31 01:00:42.331223 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:42.331197 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-0da668-dff997c59-8nrck"] Oct 31 01:00:43.655225 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:43.655189 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8532f94b-ef23-4b4d-967f-dc2aa9e3c3dd" path="/var/lib/kubelet/pods/8532f94b-ef23-4b4d-967f-dc2aa9e3c3dd/volumes" Oct 31 01:00:43.655676 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:43.655606 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="958dc2bf-130f-4861-981b-c5fb31d39f67" path="/var/lib/kubelet/pods/958dc2bf-130f-4861-981b-c5fb31d39f67/volumes" Oct 31 01:00:48.946719 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:48.946676 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-0da668-7mkrn"] Oct 31 01:00:48.997944 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:48.997901 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-0da668-7mkrn"] Oct 31 01:00:49.655316 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:49.655277 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33166625-f2d8-46d9-8ee1-1e61a8fc38bb" path="/var/lib/kubelet/pods/33166625-f2d8-46d9-8ee1-1e61a8fc38bb/volumes" Oct 31 01:00:52.330186 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:52.330151 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-79tk8"] Oct 31 01:00:55.883425 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:55.883365 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-0da668-openshift-default-5467f58b7-tzrb7"] Oct 31 01:00:55.883969 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:55.883661 5222 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-0da668-openshift-default-5467f58b7-tzrb7" podUID="d5eb6fe4-1be0-468b-91e0-d398870307b6" containerName="istio-proxy" containerID="cri-o://8e6e7087ecc3616c96560b1b7b372159fb4eb1f877a9ab9e45a03c54a7cb4b73" gracePeriod=30 Oct 31 01:00:56.907262 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:56.907220 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-da9d07-8g95z"] Oct 31 01:00:56.907860 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:56.907659 5222 cpu_manager.go:414] "containerMap: removing container" podUID="958dc2bf-130f-4861-981b-c5fb31d39f67" containerName="payload-processing" Oct 31 01:00:56.907860 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:56.907670 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="958dc2bf-130f-4861-981b-c5fb31d39f67" containerName="payload-processing" Oct 31 01:00:56.907860 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:56.907704 5222 cpu_manager.go:414] "containerMap: removing container" podUID="8532f94b-ef23-4b4d-967f-dc2aa9e3c3dd" containerName="payload-pre-processing" Oct 31 01:00:56.907860 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:56.907710 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8532f94b-ef23-4b4d-967f-dc2aa9e3c3dd" containerName="payload-pre-processing" Oct 31 01:00:56.907860 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:56.907722 5222 cpu_manager.go:414] "containerMap: removing container" podUID="33166625-f2d8-46d9-8ee1-1e61a8fc38bb" containerName="revoke-keys" Oct 31 01:00:56.907860 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:56.907726 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33166625-f2d8-46d9-8ee1-1e61a8fc38bb" containerName="revoke-keys" Oct 31 01:00:56.907860 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:56.907793 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="8532f94b-ef23-4b4d-967f-dc2aa9e3c3dd" containerName="payload-pre-processing" Oct 31 01:00:56.907860 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:56.907804 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="33166625-f2d8-46d9-8ee1-1e61a8fc38bb" containerName="revoke-keys" Oct 31 01:00:56.907860 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:56.907809 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="958dc2bf-130f-4861-981b-c5fb31d39f67" containerName="payload-processing" Oct 31 01:00:56.907860 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:56.907816 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e0d5e52-a885-4074-b8d5-e146eb6061a6" containerName="cleanup" Oct 31 01:00:56.912585 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:56.912562 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-da9d07-8g95z" Oct 31 01:00:56.964416 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:56.964329 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-da9d07-8g95z"] Oct 31 01:00:57.013600 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:57.013553 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:00:57.013600 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:57.013553 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:00:57.013600 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:57.013593 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:00:57.013937 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:57.013696 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfz9z\"" Oct 31 01:00:57.077358 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:57.077319 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98201195-ac44-4539-92b7-fee5bacb2703-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-da9d07-8g95z\" (UID: \"98201195-ac44-4539-92b7-fee5bacb2703\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-da9d07-8g95z" Oct 31 01:00:57.077568 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:57.077520 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98201195-ac44-4539-92b7-fee5bacb2703-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-da9d07-8g95z\" (UID: \"98201195-ac44-4539-92b7-fee5bacb2703\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-da9d07-8g95z" Oct 31 01:00:57.077568 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:57.077559 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgfdn\" (UniqueName: \"kubernetes.io/projected/98201195-ac44-4539-92b7-fee5bacb2703-kube-api-access-kgfdn\") pod \"maas-api-revoke-keys-e2e-worker-w4-da9d07-8g95z\" (UID: \"98201195-ac44-4539-92b7-fee5bacb2703\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-da9d07-8g95z" Oct 31 01:00:57.179026 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:57.178943 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98201195-ac44-4539-92b7-fee5bacb2703-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-da9d07-8g95z\" (UID: \"98201195-ac44-4539-92b7-fee5bacb2703\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-da9d07-8g95z" Oct 31 01:00:57.179026 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:57.178989 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgfdn\" (UniqueName: \"kubernetes.io/projected/98201195-ac44-4539-92b7-fee5bacb2703-kube-api-access-kgfdn\") pod \"maas-api-revoke-keys-e2e-worker-w4-da9d07-8g95z\" (UID: \"98201195-ac44-4539-92b7-fee5bacb2703\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-da9d07-8g95z" Oct 31 01:00:57.179223 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:57.179053 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98201195-ac44-4539-92b7-fee5bacb2703-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-da9d07-8g95z\" (UID: \"98201195-ac44-4539-92b7-fee5bacb2703\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-da9d07-8g95z" Oct 31 01:00:57.179743 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:57.179725 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98201195-ac44-4539-92b7-fee5bacb2703-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-da9d07-8g95z\" (UID: \"98201195-ac44-4539-92b7-fee5bacb2703\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-da9d07-8g95z" Oct 31 01:00:57.181506 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:57.181486 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98201195-ac44-4539-92b7-fee5bacb2703-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-da9d07-8g95z\" (UID: \"98201195-ac44-4539-92b7-fee5bacb2703\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-da9d07-8g95z" Oct 31 01:00:57.242038 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:57.241998 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgfdn\" (UniqueName: \"kubernetes.io/projected/98201195-ac44-4539-92b7-fee5bacb2703-kube-api-access-kgfdn\") pod \"maas-api-revoke-keys-e2e-worker-w4-da9d07-8g95z\" (UID: \"98201195-ac44-4539-92b7-fee5bacb2703\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-da9d07-8g95z" Oct 31 01:00:57.526473 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:57.526435 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-da9d07-8g95z" Oct 31 01:00:57.898489 ip-10-0-137-56 kubenswrapper[5222]: W1031 01:00:57.898455 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98201195_ac44_4539_92b7_fee5bacb2703.slice/crio-d4bedc7557c63b7d289083a950ec67cdf431f25eb39f530bc6d9b205849b991a WatchSource:0}: Error finding container d4bedc7557c63b7d289083a950ec67cdf431f25eb39f530bc6d9b205849b991a: Status 404 returned error can't find the container with id d4bedc7557c63b7d289083a950ec67cdf431f25eb39f530bc6d9b205849b991a Oct 31 01:00:57.900166 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:57.900145 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-da9d07-8g95z"] Oct 31 01:00:58.152910 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:58.152877 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-da9d07-8g95z" event={"ID":"98201195-ac44-4539-92b7-fee5bacb2703","Type":"ContainerStarted","Data":"d4bedc7557c63b7d289083a950ec67cdf431f25eb39f530bc6d9b205849b991a"} Oct 31 01:00:59.159928 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:59.159892 5222 generic.go:356] "Generic (PLEG): container finished" podID="98201195-ac44-4539-92b7-fee5bacb2703" containerID="25552adba68fe453444a71d04742ecf327b4504d7743c4ef0a2a403f58e5ade7" exitCode=0 Oct 31 01:00:59.160383 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:00:59.159933 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-da9d07-8g95z" event={"ID":"98201195-ac44-4539-92b7-fee5bacb2703","Type":"ContainerDied","Data":"25552adba68fe453444a71d04742ecf327b4504d7743c4ef0a2a403f58e5ade7"} Oct 31 01:01:00.305346 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:00.305318 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-da9d07-8g95z" Oct 31 01:01:00.412942 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:00.412912 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98201195-ac44-4539-92b7-fee5bacb2703-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98201195-ac44-4539-92b7-fee5bacb2703-maas-api-openshift-ca\") pod \"98201195-ac44-4539-92b7-fee5bacb2703\" (UID: \"98201195-ac44-4539-92b7-fee5bacb2703\") " Oct 31 01:01:00.413128 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:00.413032 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/98201195-ac44-4539-92b7-fee5bacb2703-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98201195-ac44-4539-92b7-fee5bacb2703-maas-api-xks-ca\") pod \"98201195-ac44-4539-92b7-fee5bacb2703\" (UID: \"98201195-ac44-4539-92b7-fee5bacb2703\") " Oct 31 01:01:00.413128 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:00.413055 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98201195-ac44-4539-92b7-fee5bacb2703-kube-api-access-kgfdn\" (UniqueName: \"kubernetes.io/projected/98201195-ac44-4539-92b7-fee5bacb2703-kube-api-access-kgfdn\") pod \"98201195-ac44-4539-92b7-fee5bacb2703\" (UID: \"98201195-ac44-4539-92b7-fee5bacb2703\") " Oct 31 01:01:00.413303 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:00.413280 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98201195-ac44-4539-92b7-fee5bacb2703-maas-api-openshift-ca" pod "98201195-ac44-4539-92b7-fee5bacb2703" (UID: "98201195-ac44-4539-92b7-fee5bacb2703"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:01:00.413429 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:00.413409 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98201195-ac44-4539-92b7-fee5bacb2703-maas-api-openshift-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:01:00.415282 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:00.415253 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98201195-ac44-4539-92b7-fee5bacb2703-maas-api-xks-ca" pod "98201195-ac44-4539-92b7-fee5bacb2703" (UID: "98201195-ac44-4539-92b7-fee5bacb2703"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:01:00.415391 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:00.415256 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98201195-ac44-4539-92b7-fee5bacb2703-kube-api-access-kgfdn" pod "98201195-ac44-4539-92b7-fee5bacb2703" (UID: "98201195-ac44-4539-92b7-fee5bacb2703"). InnerVolumeSpecName "kube-api-access-kgfdn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:01:00.514356 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:00.514313 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98201195-ac44-4539-92b7-fee5bacb2703-maas-api-xks-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:01:00.514356 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:00.514347 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgfdn\" (UniqueName: \"kubernetes.io/projected/98201195-ac44-4539-92b7-fee5bacb2703-kube-api-access-kgfdn\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:01:01.124405 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:01.124021 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-0da668-openshift-default-5467f58b7-tzrb7" Oct 31 01:01:01.174041 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:01.173956 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-da9d07-8g95z" Oct 31 01:01:01.174041 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:01.173994 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-da9d07-8g95z" event={"ID":"98201195-ac44-4539-92b7-fee5bacb2703","Type":"ContainerDied","Data":"d4bedc7557c63b7d289083a950ec67cdf431f25eb39f530bc6d9b205849b991a"} Oct 31 01:01:01.174296 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:01.174074 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d4bedc7557c63b7d289083a950ec67cdf431f25eb39f530bc6d9b205849b991a" Oct 31 01:01:01.176087 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:01.176057 5222 generic.go:356] "Generic (PLEG): container finished" podID="d5eb6fe4-1be0-468b-91e0-d398870307b6" containerID="8e6e7087ecc3616c96560b1b7b372159fb4eb1f877a9ab9e45a03c54a7cb4b73" exitCode=0 Oct 31 01:01:01.176221 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:01.176100 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-0da668-openshift-default-5467f58b7-tzrb7" Oct 31 01:01:01.176221 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:01.176140 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-0da668-openshift-default-5467f58b7-tzrb7" event={"ID":"d5eb6fe4-1be0-468b-91e0-d398870307b6","Type":"ContainerDied","Data":"8e6e7087ecc3616c96560b1b7b372159fb4eb1f877a9ab9e45a03c54a7cb4b73"} Oct 31 01:01:01.176221 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:01.176194 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-0da668-openshift-default-5467f58b7-tzrb7" event={"ID":"d5eb6fe4-1be0-468b-91e0-d398870307b6","Type":"ContainerDied","Data":"11248aeafc83293bda7e0f97d82528c46414f324e5eac55ddd3faaacc252bf25"} Oct 31 01:01:01.176330 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:01.176221 5222 scope.go:122] "RemoveContainer" containerID="8e6e7087ecc3616c96560b1b7b372159fb4eb1f877a9ab9e45a03c54a7cb4b73" Oct 31 01:01:01.187275 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:01.187257 5222 scope.go:122] "RemoveContainer" containerID="7c7c18c30eb9362a22bd2e11df1913e7e2d5bacdb670e046c93ef47dc33eac3f" Oct 31 01:01:01.196234 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:01.196214 5222 scope.go:122] "RemoveContainer" containerID="8e6e7087ecc3616c96560b1b7b372159fb4eb1f877a9ab9e45a03c54a7cb4b73" Oct 31 01:01:01.196553 ip-10-0-137-56 kubenswrapper[5222]: E1031 01:01:01.196531 5222 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e6e7087ecc3616c96560b1b7b372159fb4eb1f877a9ab9e45a03c54a7cb4b73\": container with ID starting with 8e6e7087ecc3616c96560b1b7b372159fb4eb1f877a9ab9e45a03c54a7cb4b73 not found: ID does not exist" containerID="8e6e7087ecc3616c96560b1b7b372159fb4eb1f877a9ab9e45a03c54a7cb4b73" Oct 31 01:01:01.196627 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:01.196565 5222 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e6e7087ecc3616c96560b1b7b372159fb4eb1f877a9ab9e45a03c54a7cb4b73"} err="failed to get container status \"8e6e7087ecc3616c96560b1b7b372159fb4eb1f877a9ab9e45a03c54a7cb4b73\": rpc error: code = NotFound desc = could not find container \"8e6e7087ecc3616c96560b1b7b372159fb4eb1f877a9ab9e45a03c54a7cb4b73\": container with ID starting with 8e6e7087ecc3616c96560b1b7b372159fb4eb1f877a9ab9e45a03c54a7cb4b73 not found: ID does not exist" Oct 31 01:01:01.196627 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:01.196584 5222 scope.go:122] "RemoveContainer" containerID="7c7c18c30eb9362a22bd2e11df1913e7e2d5bacdb670e046c93ef47dc33eac3f" Oct 31 01:01:01.196854 ip-10-0-137-56 kubenswrapper[5222]: E1031 01:01:01.196836 5222 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c7c18c30eb9362a22bd2e11df1913e7e2d5bacdb670e046c93ef47dc33eac3f\": container with ID starting with 7c7c18c30eb9362a22bd2e11df1913e7e2d5bacdb670e046c93ef47dc33eac3f not found: ID does not exist" containerID="7c7c18c30eb9362a22bd2e11df1913e7e2d5bacdb670e046c93ef47dc33eac3f" Oct 31 01:01:01.196900 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:01.196860 5222 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c7c18c30eb9362a22bd2e11df1913e7e2d5bacdb670e046c93ef47dc33eac3f"} err="failed to get container status \"7c7c18c30eb9362a22bd2e11df1913e7e2d5bacdb670e046c93ef47dc33eac3f\": rpc error: code = NotFound desc = could not find container \"7c7c18c30eb9362a22bd2e11df1913e7e2d5bacdb670e046c93ef47dc33eac3f\": container with ID starting with 7c7c18c30eb9362a22bd2e11df1913e7e2d5bacdb670e046c93ef47dc33eac3f not found: ID does not exist" Oct 31 01:01:01.223251 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:01.223211 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5eb6fe4-1be0-468b-91e0-d398870307b6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5eb6fe4-1be0-468b-91e0-d398870307b6-workload-certs\") pod \"d5eb6fe4-1be0-468b-91e0-d398870307b6\" (UID: \"d5eb6fe4-1be0-468b-91e0-d398870307b6\") " Oct 31 01:01:01.223484 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:01.223299 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5eb6fe4-1be0-468b-91e0-d398870307b6-kube-api-access-22ms5\" (UniqueName: \"kubernetes.io/projected/d5eb6fe4-1be0-468b-91e0-d398870307b6-kube-api-access-22ms5\") pod \"d5eb6fe4-1be0-468b-91e0-d398870307b6\" (UID: \"d5eb6fe4-1be0-468b-91e0-d398870307b6\") " Oct 31 01:01:01.223484 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:01.223406 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5eb6fe4-1be0-468b-91e0-d398870307b6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5eb6fe4-1be0-468b-91e0-d398870307b6-workload-socket\") pod \"d5eb6fe4-1be0-468b-91e0-d398870307b6\" (UID: \"d5eb6fe4-1be0-468b-91e0-d398870307b6\") " Oct 31 01:01:01.223484 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:01.223457 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5eb6fe4-1be0-468b-91e0-d398870307b6-istio-token\" (UniqueName: \"kubernetes.io/projected/d5eb6fe4-1be0-468b-91e0-d398870307b6-istio-token\") pod \"d5eb6fe4-1be0-468b-91e0-d398870307b6\" (UID: \"d5eb6fe4-1be0-468b-91e0-d398870307b6\") " Oct 31 01:01:01.223656 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:01.223495 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5eb6fe4-1be0-468b-91e0-d398870307b6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5eb6fe4-1be0-468b-91e0-d398870307b6-istio-data\") pod \"d5eb6fe4-1be0-468b-91e0-d398870307b6\" (UID: \"d5eb6fe4-1be0-468b-91e0-d398870307b6\") " Oct 31 01:01:01.223656 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:01.223529 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5eb6fe4-1be0-468b-91e0-d398870307b6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5eb6fe4-1be0-468b-91e0-d398870307b6-credential-socket\") pod \"d5eb6fe4-1be0-468b-91e0-d398870307b6\" (UID: \"d5eb6fe4-1be0-468b-91e0-d398870307b6\") " Oct 31 01:01:01.223656 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:01.223526 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5eb6fe4-1be0-468b-91e0-d398870307b6-workload-certs" pod "d5eb6fe4-1be0-468b-91e0-d398870307b6" (UID: "d5eb6fe4-1be0-468b-91e0-d398870307b6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:01:01.223656 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:01.223589 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5eb6fe4-1be0-468b-91e0-d398870307b6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5eb6fe4-1be0-468b-91e0-d398870307b6-istiod-ca-cert\") pod \"d5eb6fe4-1be0-468b-91e0-d398870307b6\" (UID: \"d5eb6fe4-1be0-468b-91e0-d398870307b6\") " Oct 31 01:01:01.223656 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:01.223624 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d5eb6fe4-1be0-468b-91e0-d398870307b6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5eb6fe4-1be0-468b-91e0-d398870307b6-istio-podinfo\") pod \"d5eb6fe4-1be0-468b-91e0-d398870307b6\" (UID: \"d5eb6fe4-1be0-468b-91e0-d398870307b6\") " Oct 31 01:01:01.223900 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:01.223674 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5eb6fe4-1be0-468b-91e0-d398870307b6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5eb6fe4-1be0-468b-91e0-d398870307b6-istio-envoy\") pod \"d5eb6fe4-1be0-468b-91e0-d398870307b6\" (UID: \"d5eb6fe4-1be0-468b-91e0-d398870307b6\") " Oct 31 01:01:01.223900 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:01.223704 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5eb6fe4-1be0-468b-91e0-d398870307b6-workload-socket" pod "d5eb6fe4-1be0-468b-91e0-d398870307b6" (UID: "d5eb6fe4-1be0-468b-91e0-d398870307b6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:01:01.223900 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:01.223772 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5eb6fe4-1be0-468b-91e0-d398870307b6-istio-data" pod "d5eb6fe4-1be0-468b-91e0-d398870307b6" (UID: "d5eb6fe4-1be0-468b-91e0-d398870307b6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:01:01.223900 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:01.223879 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5eb6fe4-1be0-468b-91e0-d398870307b6-credential-socket" pod "d5eb6fe4-1be0-468b-91e0-d398870307b6" (UID: "d5eb6fe4-1be0-468b-91e0-d398870307b6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:01:01.224104 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:01.224082 5222 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5eb6fe4-1be0-468b-91e0-d398870307b6-workload-socket\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:01:01.224169 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:01.224103 5222 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5eb6fe4-1be0-468b-91e0-d398870307b6-istio-data\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:01:01.224169 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:01.224116 5222 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5eb6fe4-1be0-468b-91e0-d398870307b6-credential-socket\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:01:01.224169 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:01.224156 5222 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5eb6fe4-1be0-468b-91e0-d398870307b6-workload-certs\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:01:01.224333 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:01.224305 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5eb6fe4-1be0-468b-91e0-d398870307b6-istiod-ca-cert" pod "d5eb6fe4-1be0-468b-91e0-d398870307b6" (UID: "d5eb6fe4-1be0-468b-91e0-d398870307b6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:01:01.225997 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:01.225974 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5eb6fe4-1be0-468b-91e0-d398870307b6-istio-envoy" pod "d5eb6fe4-1be0-468b-91e0-d398870307b6" (UID: "d5eb6fe4-1be0-468b-91e0-d398870307b6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:01:01.225997 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:01.225986 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d5eb6fe4-1be0-468b-91e0-d398870307b6-istio-podinfo" pod "d5eb6fe4-1be0-468b-91e0-d398870307b6" (UID: "d5eb6fe4-1be0-468b-91e0-d398870307b6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:01:01.226276 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:01.226259 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5eb6fe4-1be0-468b-91e0-d398870307b6-kube-api-access-22ms5" pod "d5eb6fe4-1be0-468b-91e0-d398870307b6" (UID: "d5eb6fe4-1be0-468b-91e0-d398870307b6"). InnerVolumeSpecName "kube-api-access-22ms5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:01:01.226331 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:01.226278 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5eb6fe4-1be0-468b-91e0-d398870307b6-istio-token" pod "d5eb6fe4-1be0-468b-91e0-d398870307b6" (UID: "d5eb6fe4-1be0-468b-91e0-d398870307b6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:01:01.324727 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:01.324688 5222 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5eb6fe4-1be0-468b-91e0-d398870307b6-istio-token\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:01:01.324727 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:01.324717 5222 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5eb6fe4-1be0-468b-91e0-d398870307b6-istiod-ca-cert\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:01:01.324727 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:01.324727 5222 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5eb6fe4-1be0-468b-91e0-d398870307b6-istio-podinfo\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:01:01.324727 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:01.324736 5222 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5eb6fe4-1be0-468b-91e0-d398870307b6-istio-envoy\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:01:01.325389 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:01.324746 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22ms5\" (UniqueName: \"kubernetes.io/projected/d5eb6fe4-1be0-468b-91e0-d398870307b6-kube-api-access-22ms5\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:01:01.379206 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:01.379170 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-da9d07/simulator-w4-da9d07-kserve-7c7588b894-snqq4"] Oct 31 01:01:01.656046 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:01.656014 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-0da668-openshift-default-5467f58b7-tzrb7"] Oct 31 01:01:01.658305 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:01.658288 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-0da668-openshift-default-5467f58b7-tzrb7"] Oct 31 01:01:03.654071 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:03.654036 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5eb6fe4-1be0-468b-91e0-d398870307b6" path="/var/lib/kubelet/pods/d5eb6fe4-1be0-468b-91e0-d398870307b6/volumes" Oct 31 01:01:04.050201 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:04.050163 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ef266b8-vkmzd"] Oct 31 01:01:04.101701 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:04.101652 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ef266b8-vkmzd"] Oct 31 01:01:04.103302 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:04.103277 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-162fe38d-xcsh4"] Oct 31 01:01:04.104692 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:04.104672 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-162fe38d-xcsh4"] Oct 31 01:01:05.654532 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:05.654497 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="343c42b7-22b0-4f9d-8be7-0ffbd7ce64e1" path="/var/lib/kubelet/pods/343c42b7-22b0-4f9d-8be7-0ffbd7ce64e1/volumes" Oct 31 01:01:05.654944 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:05.654930 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="684d2727-5c85-4374-bc7f-69634fd00e17" path="/var/lib/kubelet/pods/684d2727-5c85-4374-bc7f-69634fd00e17/volumes" Oct 31 01:01:05.890695 ip-10-0-137-56 kubenswrapper[5222]: E1031 01:01:05.890658 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 01:01:08.337885 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:08.337851 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-da9d07-5848fd449-s4wwm"] Oct 31 01:01:08.444921 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:08.444887 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-da9d07-69bb55d779-hg9l2"] Oct 31 01:01:10.265156 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:10.265125 5222 scope.go:122] "RemoveContainer" containerID="d3315a0346a141c03cca81210f6cad2a30129a54ea8319f656e9b6bad015926b" Oct 31 01:01:10.274556 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:10.274534 5222 scope.go:122] "RemoveContainer" containerID="6cb653f3827b816e3cc84f0faf3bae10b779fd6abab6730760c99e1d589b5515" Oct 31 01:01:10.283316 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:10.283289 5222 scope.go:122] "RemoveContainer" containerID="217be6d7ac777b9d93f152f122e0c627cb6028ba22975c4b5a4be8420441a3b8" Oct 31 01:01:10.292299 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:10.292210 5222 scope.go:122] "RemoveContainer" containerID="212a202fb631f69b536129a01a56c9fe3979c7ab86f69391a1e95212c31f65d7" Oct 31 01:01:10.301294 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:10.301253 5222 scope.go:122] "RemoveContainer" containerID="97d4dbc0448918ee3baa3b83612217f0bfbb7521c5699506f523ae1c9bed5db8" Oct 31 01:01:10.310110 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:10.310090 5222 scope.go:122] "RemoveContainer" containerID="0b7c7f06d2559c2aef5ff0e50b8cc4fbeef1940ca2da763853d2f6bd4248e92c" Oct 31 01:01:13.338830 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:13.338783 5222 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-da9d07-5848fd449-s4wwm" podUID="bc1b882b-b51a-4d28-a9ee-35362dd92d69" containerName="payload-processing" containerID="cri-o://b0bb94e6b83687089012e92190f7d97b981d645b7eb64472835acd134c959db8" gracePeriod=25 Oct 31 01:01:13.445303 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:13.445256 5222 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-da9d07-69bb55d779-hg9l2" podUID="1b4de77a-88d7-466f-a8a4-37c224ab4a3c" containerName="payload-pre-processing" containerID="cri-o://9e0e113793b61d5918a3a4d25a7690379e8f177c50465b69b9ff577361cbdd12" gracePeriod=25 Oct 31 01:01:13.844670 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:13.844630 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-784a983c-mttgm"] Oct 31 01:01:13.845024 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:13.845012 5222 cpu_manager.go:414] "containerMap: removing container" podUID="98201195-ac44-4539-92b7-fee5bacb2703" containerName="revoke-keys" Oct 31 01:01:13.845069 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:13.845026 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98201195-ac44-4539-92b7-fee5bacb2703" containerName="revoke-keys" Oct 31 01:01:13.845069 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:13.845055 5222 cpu_manager.go:414] "containerMap: removing container" podUID="d5eb6fe4-1be0-468b-91e0-d398870307b6" containerName="istio-proxy" Oct 31 01:01:13.845069 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:13.845060 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5eb6fe4-1be0-468b-91e0-d398870307b6" containerName="istio-proxy" Oct 31 01:01:13.845069 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:13.845068 5222 cpu_manager.go:414] "containerMap: removing container" podUID="d5eb6fe4-1be0-468b-91e0-d398870307b6" containerName="istio-proxy" Oct 31 01:01:13.845200 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:13.845073 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5eb6fe4-1be0-468b-91e0-d398870307b6" containerName="istio-proxy" Oct 31 01:01:13.845200 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:13.845130 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5eb6fe4-1be0-468b-91e0-d398870307b6" containerName="istio-proxy" Oct 31 01:01:13.845200 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:13.845141 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="98201195-ac44-4539-92b7-fee5bacb2703" containerName="revoke-keys" Oct 31 01:01:13.850904 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:13.850873 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-784a983c-mttgm" Oct 31 01:01:13.902051 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:13.902015 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-784a983c-mttgm"] Oct 31 01:01:13.905327 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:13.905295 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2fec8671-lhx4k"] Oct 31 01:01:13.905958 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:13.905932 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5eb6fe4-1be0-468b-91e0-d398870307b6" containerName="istio-proxy" Oct 31 01:01:13.910420 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:13.910400 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fec8671-lhx4k" Oct 31 01:01:13.950399 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:13.950332 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0599b1e3-1033-48ab-bd20-b5446d96dd13-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2fec8671-lhx4k\" (UID: \"0599b1e3-1033-48ab-bd20-b5446d96dd13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fec8671-lhx4k" Oct 31 01:01:13.950399 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:13.950397 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgf6m\" (UniqueName: \"kubernetes.io/projected/0599b1e3-1033-48ab-bd20-b5446d96dd13-kube-api-access-vgf6m\") pod \"maas-api-revoke-sub-2fec8671-lhx4k\" (UID: \"0599b1e3-1033-48ab-bd20-b5446d96dd13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fec8671-lhx4k" Oct 31 01:01:13.950624 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:13.950514 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d8b71ff-c6de-4b8c-8e78-175b48c7ed2e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-784a983c-mttgm\" (UID: \"0d8b71ff-c6de-4b8c-8e78-175b48c7ed2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-784a983c-mttgm" Oct 31 01:01:13.950624 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:13.950550 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d8b71ff-c6de-4b8c-8e78-175b48c7ed2e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-784a983c-mttgm\" (UID: \"0d8b71ff-c6de-4b8c-8e78-175b48c7ed2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-784a983c-mttgm" Oct 31 01:01:13.950693 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:13.950659 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0599b1e3-1033-48ab-bd20-b5446d96dd13-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2fec8671-lhx4k\" (UID: \"0599b1e3-1033-48ab-bd20-b5446d96dd13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fec8671-lhx4k" Oct 31 01:01:13.950693 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:13.950683 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hd7c\" (UniqueName: \"kubernetes.io/projected/0d8b71ff-c6de-4b8c-8e78-175b48c7ed2e-kube-api-access-2hd7c\") pod \"maas-api-revoke-sub-784a983c-mttgm\" (UID: \"0d8b71ff-c6de-4b8c-8e78-175b48c7ed2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-784a983c-mttgm" Oct 31 01:01:13.951429 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:13.951409 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:01:13.951547 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:13.951460 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:01:13.954465 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:13.954442 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfz9z\"" Oct 31 01:01:13.954564 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:13.954450 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:01:13.962294 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:13.962262 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2fec8671-lhx4k"] Oct 31 01:01:13.985733 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:13.985693 5222 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-da9d07-5848fd449-s4wwm" podUID="bc1b882b-b51a-4d28-a9ee-35362dd92d69" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:01:13.985907 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:13.985699 5222 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-da9d07-69bb55d779-hg9l2" podUID="1b4de77a-88d7-466f-a8a4-37c224ab4a3c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:01:14.052222 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:14.052187 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0599b1e3-1033-48ab-bd20-b5446d96dd13-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2fec8671-lhx4k\" (UID: \"0599b1e3-1033-48ab-bd20-b5446d96dd13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fec8671-lhx4k" Oct 31 01:01:14.052448 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:14.052231 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hd7c\" (UniqueName: \"kubernetes.io/projected/0d8b71ff-c6de-4b8c-8e78-175b48c7ed2e-kube-api-access-2hd7c\") pod \"maas-api-revoke-sub-784a983c-mttgm\" (UID: \"0d8b71ff-c6de-4b8c-8e78-175b48c7ed2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-784a983c-mttgm" Oct 31 01:01:14.052448 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:14.052283 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0599b1e3-1033-48ab-bd20-b5446d96dd13-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2fec8671-lhx4k\" (UID: \"0599b1e3-1033-48ab-bd20-b5446d96dd13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fec8671-lhx4k" Oct 31 01:01:14.052448 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:14.052304 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgf6m\" (UniqueName: \"kubernetes.io/projected/0599b1e3-1033-48ab-bd20-b5446d96dd13-kube-api-access-vgf6m\") pod \"maas-api-revoke-sub-2fec8671-lhx4k\" (UID: \"0599b1e3-1033-48ab-bd20-b5446d96dd13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fec8671-lhx4k" Oct 31 01:01:14.052448 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:14.052336 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d8b71ff-c6de-4b8c-8e78-175b48c7ed2e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-784a983c-mttgm\" (UID: \"0d8b71ff-c6de-4b8c-8e78-175b48c7ed2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-784a983c-mttgm" Oct 31 01:01:14.052448 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:14.052355 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d8b71ff-c6de-4b8c-8e78-175b48c7ed2e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-784a983c-mttgm\" (UID: \"0d8b71ff-c6de-4b8c-8e78-175b48c7ed2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-784a983c-mttgm" Oct 31 01:01:14.053206 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:14.053105 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d8b71ff-c6de-4b8c-8e78-175b48c7ed2e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-784a983c-mttgm\" (UID: \"0d8b71ff-c6de-4b8c-8e78-175b48c7ed2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-784a983c-mttgm" Oct 31 01:01:14.053206 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:14.053106 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0599b1e3-1033-48ab-bd20-b5446d96dd13-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2fec8671-lhx4k\" (UID: \"0599b1e3-1033-48ab-bd20-b5446d96dd13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fec8671-lhx4k" Oct 31 01:01:14.054905 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:14.054878 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d8b71ff-c6de-4b8c-8e78-175b48c7ed2e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-784a983c-mttgm\" (UID: \"0d8b71ff-c6de-4b8c-8e78-175b48c7ed2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-784a983c-mttgm" Oct 31 01:01:14.054905 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:14.054898 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0599b1e3-1033-48ab-bd20-b5446d96dd13-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2fec8671-lhx4k\" (UID: \"0599b1e3-1033-48ab-bd20-b5446d96dd13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fec8671-lhx4k" Oct 31 01:01:14.115216 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:14.115145 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgf6m\" (UniqueName: \"kubernetes.io/projected/0599b1e3-1033-48ab-bd20-b5446d96dd13-kube-api-access-vgf6m\") pod \"maas-api-revoke-sub-2fec8671-lhx4k\" (UID: \"0599b1e3-1033-48ab-bd20-b5446d96dd13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fec8671-lhx4k" Oct 31 01:01:14.115216 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:14.115186 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hd7c\" (UniqueName: \"kubernetes.io/projected/0d8b71ff-c6de-4b8c-8e78-175b48c7ed2e-kube-api-access-2hd7c\") pod \"maas-api-revoke-sub-784a983c-mttgm\" (UID: \"0d8b71ff-c6de-4b8c-8e78-175b48c7ed2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-784a983c-mttgm" Oct 31 01:01:14.164449 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:14.164363 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-784a983c-mttgm" Oct 31 01:01:14.223767 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:14.223731 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fec8671-lhx4k" Oct 31 01:01:14.537180 ip-10-0-137-56 kubenswrapper[5222]: W1031 01:01:14.537138 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d8b71ff_c6de_4b8c_8e78_175b48c7ed2e.slice/crio-487c273f5c53f9313d29f1314439fc902274513023b8d8a50b2e2b035fd7ba30 WatchSource:0}: Error finding container 487c273f5c53f9313d29f1314439fc902274513023b8d8a50b2e2b035fd7ba30: Status 404 returned error can't find the container with id 487c273f5c53f9313d29f1314439fc902274513023b8d8a50b2e2b035fd7ba30 Oct 31 01:01:14.538744 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:14.538709 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-784a983c-mttgm"] Oct 31 01:01:14.643847 ip-10-0-137-56 kubenswrapper[5222]: W1031 01:01:14.643805 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0599b1e3_1033_48ab_bd20_b5446d96dd13.slice/crio-38f7c086742b14ef42a9344d5df0ddf8559543952e38cfc83fbfc42219219a64 WatchSource:0}: Error finding container 38f7c086742b14ef42a9344d5df0ddf8559543952e38cfc83fbfc42219219a64: Status 404 returned error can't find the container with id 38f7c086742b14ef42a9344d5df0ddf8559543952e38cfc83fbfc42219219a64 Oct 31 01:01:14.645345 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:14.645318 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2fec8671-lhx4k"] Oct 31 01:01:15.060459 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:15.060420 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a0c57b4-d46f8"] Oct 31 01:01:15.064222 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:15.064202 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03773639-nsd5f"] Oct 31 01:01:15.066698 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:15.066677 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a0c57b4-d46f8"] Oct 31 01:01:15.111656 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:15.111575 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03773639-nsd5f"] Oct 31 01:01:15.258387 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:15.258338 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fec8671-lhx4k" event={"ID":"0599b1e3-1033-48ab-bd20-b5446d96dd13","Type":"ContainerStarted","Data":"9212714a7c7c4f861f19db50d851da5dbb6bab01f5758e2524c93e7832ba3bcd"} Oct 31 01:01:15.258604 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:15.258425 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fec8671-lhx4k" event={"ID":"0599b1e3-1033-48ab-bd20-b5446d96dd13","Type":"ContainerStarted","Data":"38f7c086742b14ef42a9344d5df0ddf8559543952e38cfc83fbfc42219219a64"} Oct 31 01:01:15.259949 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:15.259924 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-784a983c-mttgm" event={"ID":"0d8b71ff-c6de-4b8c-8e78-175b48c7ed2e","Type":"ContainerStarted","Data":"6d9c2a05f072ef51dbe04bb013eb50f5095fade5057b3271df9d539f53283cb4"} Oct 31 01:01:15.260062 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:15.259961 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-784a983c-mttgm" event={"ID":"0d8b71ff-c6de-4b8c-8e78-175b48c7ed2e","Type":"ContainerStarted","Data":"487c273f5c53f9313d29f1314439fc902274513023b8d8a50b2e2b035fd7ba30"} Oct 31 01:01:15.380315 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:15.380215 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fec8671-lhx4k" podStartSLOduration=2.3802003640000002 podStartE2EDuration="2.380200364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:01:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:01:15.37857097 +0000 UTC m=+2767.849740720" watchObservedRunningTime="2026-10-31 01:01:15.380200364 +0000 UTC m=+2767.851370128" Oct 31 01:01:15.496510 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:15.496446 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-784a983c-mttgm" podStartSLOduration=2.496419243 podStartE2EDuration="2.496419243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:01:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:01:15.496297991 +0000 UTC m=+2767.967467742" watchObservedRunningTime="2026-10-31 01:01:15.496419243 +0000 UTC m=+2767.967588989" Oct 31 01:01:15.656650 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:15.656572 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a1e9ba8-7189-4e75-a33e-028af75e3f87" path="/var/lib/kubelet/pods/2a1e9ba8-7189-4e75-a33e-028af75e3f87/volumes" Oct 31 01:01:15.657020 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:15.656984 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca49d252-2308-40a6-a13b-96500d6c1296" path="/var/lib/kubelet/pods/ca49d252-2308-40a6-a13b-96500d6c1296/volumes" Oct 31 01:01:16.266638 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:16.266596 5222 generic.go:356] "Generic (PLEG): container finished" podID="0599b1e3-1033-48ab-bd20-b5446d96dd13" containerID="9212714a7c7c4f861f19db50d851da5dbb6bab01f5758e2524c93e7832ba3bcd" exitCode=0 Oct 31 01:01:16.266865 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:16.266686 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fec8671-lhx4k" event={"ID":"0599b1e3-1033-48ab-bd20-b5446d96dd13","Type":"ContainerDied","Data":"9212714a7c7c4f861f19db50d851da5dbb6bab01f5758e2524c93e7832ba3bcd"} Oct 31 01:01:16.268284 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:16.268256 5222 generic.go:356] "Generic (PLEG): container finished" podID="0d8b71ff-c6de-4b8c-8e78-175b48c7ed2e" containerID="6d9c2a05f072ef51dbe04bb013eb50f5095fade5057b3271df9d539f53283cb4" exitCode=0 Oct 31 01:01:16.268449 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:16.268313 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-784a983c-mttgm" event={"ID":"0d8b71ff-c6de-4b8c-8e78-175b48c7ed2e","Type":"ContainerDied","Data":"6d9c2a05f072ef51dbe04bb013eb50f5095fade5057b3271df9d539f53283cb4"} Oct 31 01:01:16.399838 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:16.399798 5222 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-da9d07/simulator-w4-da9d07-kserve-7c7588b894-snqq4" podUID="3d37ba9a-a648-44b2-aa04-4acf2b503a08" containerName="main" containerID="cri-o://d9b2458873ff4883ca579cbebeb20cfff6e24cbc1808367ac031b35f91e04383" gracePeriod=45 Oct 31 01:01:16.658009 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:16.657985 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-da9d07/simulator-w4-da9d07-kserve-7c7588b894-snqq4" Oct 31 01:01:16.779539 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:16.779439 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d37ba9a-a648-44b2-aa04-4acf2b503a08-home\" (UniqueName: \"kubernetes.io/empty-dir/3d37ba9a-a648-44b2-aa04-4acf2b503a08-home\") pod \"3d37ba9a-a648-44b2-aa04-4acf2b503a08\" (UID: \"3d37ba9a-a648-44b2-aa04-4acf2b503a08\") " Oct 31 01:01:16.779719 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:16.779576 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d37ba9a-a648-44b2-aa04-4acf2b503a08-tls-certs\" (UniqueName: \"kubernetes.io/secret/3d37ba9a-a648-44b2-aa04-4acf2b503a08-tls-certs\") pod \"3d37ba9a-a648-44b2-aa04-4acf2b503a08\" (UID: \"3d37ba9a-a648-44b2-aa04-4acf2b503a08\") " Oct 31 01:01:16.779719 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:16.779625 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d37ba9a-a648-44b2-aa04-4acf2b503a08-kube-api-access-q8rjn\" (UniqueName: \"kubernetes.io/projected/3d37ba9a-a648-44b2-aa04-4acf2b503a08-kube-api-access-q8rjn\") pod \"3d37ba9a-a648-44b2-aa04-4acf2b503a08\" (UID: \"3d37ba9a-a648-44b2-aa04-4acf2b503a08\") " Oct 31 01:01:16.779719 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:16.779662 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d37ba9a-a648-44b2-aa04-4acf2b503a08-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d37ba9a-a648-44b2-aa04-4acf2b503a08-model-cache\") pod \"3d37ba9a-a648-44b2-aa04-4acf2b503a08\" (UID: \"3d37ba9a-a648-44b2-aa04-4acf2b503a08\") " Oct 31 01:01:16.779719 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:16.779693 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d37ba9a-a648-44b2-aa04-4acf2b503a08-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d37ba9a-a648-44b2-aa04-4acf2b503a08-tmp-dir\") pod \"3d37ba9a-a648-44b2-aa04-4acf2b503a08\" (UID: \"3d37ba9a-a648-44b2-aa04-4acf2b503a08\") " Oct 31 01:01:16.779983 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:16.779759 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d37ba9a-a648-44b2-aa04-4acf2b503a08-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d37ba9a-a648-44b2-aa04-4acf2b503a08-dshm\") pod \"3d37ba9a-a648-44b2-aa04-4acf2b503a08\" (UID: \"3d37ba9a-a648-44b2-aa04-4acf2b503a08\") " Oct 31 01:01:16.780762 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:16.780729 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d37ba9a-a648-44b2-aa04-4acf2b503a08-model-cache" pod "3d37ba9a-a648-44b2-aa04-4acf2b503a08" (UID: "3d37ba9a-a648-44b2-aa04-4acf2b503a08"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:01:16.780945 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:16.780886 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d37ba9a-a648-44b2-aa04-4acf2b503a08-tmp-dir" pod "3d37ba9a-a648-44b2-aa04-4acf2b503a08" (UID: "3d37ba9a-a648-44b2-aa04-4acf2b503a08"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:01:16.781705 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:16.781531 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d37ba9a-a648-44b2-aa04-4acf2b503a08-home" pod "3d37ba9a-a648-44b2-aa04-4acf2b503a08" (UID: "3d37ba9a-a648-44b2-aa04-4acf2b503a08"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:01:16.783895 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:16.783871 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d37ba9a-a648-44b2-aa04-4acf2b503a08-dshm" pod "3d37ba9a-a648-44b2-aa04-4acf2b503a08" (UID: "3d37ba9a-a648-44b2-aa04-4acf2b503a08"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:01:16.784184 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:16.784149 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d37ba9a-a648-44b2-aa04-4acf2b503a08-kube-api-access-q8rjn" pod "3d37ba9a-a648-44b2-aa04-4acf2b503a08" (UID: "3d37ba9a-a648-44b2-aa04-4acf2b503a08"). InnerVolumeSpecName "kube-api-access-q8rjn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:01:16.784280 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:16.784173 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d37ba9a-a648-44b2-aa04-4acf2b503a08-tls-certs" pod "3d37ba9a-a648-44b2-aa04-4acf2b503a08" (UID: "3d37ba9a-a648-44b2-aa04-4acf2b503a08"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:01:16.880666 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:16.880628 5222 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d37ba9a-a648-44b2-aa04-4acf2b503a08-home\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:01:16.880666 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:16.880662 5222 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d37ba9a-a648-44b2-aa04-4acf2b503a08-tls-certs\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:01:16.880666 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:16.880673 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8rjn\" (UniqueName: \"kubernetes.io/projected/3d37ba9a-a648-44b2-aa04-4acf2b503a08-kube-api-access-q8rjn\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:01:16.881032 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:16.880684 5222 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d37ba9a-a648-44b2-aa04-4acf2b503a08-model-cache\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:01:16.881032 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:16.880693 5222 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d37ba9a-a648-44b2-aa04-4acf2b503a08-tmp-dir\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:01:16.881032 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:16.880702 5222 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d37ba9a-a648-44b2-aa04-4acf2b503a08-dshm\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:01:17.275485 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:17.275450 5222 generic.go:356] "Generic (PLEG): container finished" podID="3d37ba9a-a648-44b2-aa04-4acf2b503a08" containerID="d9b2458873ff4883ca579cbebeb20cfff6e24cbc1808367ac031b35f91e04383" exitCode=0 Oct 31 01:01:17.275670 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:17.275500 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-da9d07/simulator-w4-da9d07-kserve-7c7588b894-snqq4" Oct 31 01:01:17.275670 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:17.275547 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-da9d07/simulator-w4-da9d07-kserve-7c7588b894-snqq4" event={"ID":"3d37ba9a-a648-44b2-aa04-4acf2b503a08","Type":"ContainerDied","Data":"d9b2458873ff4883ca579cbebeb20cfff6e24cbc1808367ac031b35f91e04383"} Oct 31 01:01:17.275670 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:17.275600 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-da9d07/simulator-w4-da9d07-kserve-7c7588b894-snqq4" event={"ID":"3d37ba9a-a648-44b2-aa04-4acf2b503a08","Type":"ContainerDied","Data":"75e6583c7520fa014f61f618fefb4a19255efdd97ca1fac2ed6e6e2bcf74c93e"} Oct 31 01:01:17.275670 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:17.275620 5222 scope.go:122] "RemoveContainer" containerID="d9b2458873ff4883ca579cbebeb20cfff6e24cbc1808367ac031b35f91e04383" Oct 31 01:01:17.289790 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:17.289765 5222 scope.go:122] "RemoveContainer" containerID="d9b2458873ff4883ca579cbebeb20cfff6e24cbc1808367ac031b35f91e04383" Oct 31 01:01:17.290305 ip-10-0-137-56 kubenswrapper[5222]: E1031 01:01:17.290254 5222 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9b2458873ff4883ca579cbebeb20cfff6e24cbc1808367ac031b35f91e04383\": container with ID starting with d9b2458873ff4883ca579cbebeb20cfff6e24cbc1808367ac031b35f91e04383 not found: ID does not exist" containerID="d9b2458873ff4883ca579cbebeb20cfff6e24cbc1808367ac031b35f91e04383" Oct 31 01:01:17.290414 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:17.290318 5222 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9b2458873ff4883ca579cbebeb20cfff6e24cbc1808367ac031b35f91e04383"} err="failed to get container status \"d9b2458873ff4883ca579cbebeb20cfff6e24cbc1808367ac031b35f91e04383\": rpc error: code = NotFound desc = could not find container \"d9b2458873ff4883ca579cbebeb20cfff6e24cbc1808367ac031b35f91e04383\": container with ID starting with d9b2458873ff4883ca579cbebeb20cfff6e24cbc1808367ac031b35f91e04383 not found: ID does not exist" Oct 31 01:01:17.409550 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:17.409521 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-784a983c-mttgm" Oct 31 01:01:17.446573 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:17.446546 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fec8671-lhx4k" Oct 31 01:01:17.456385 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:17.456339 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-da9d07/simulator-w4-da9d07-kserve-7c7588b894-snqq4"] Oct 31 01:01:17.460646 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:17.460618 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-da9d07/simulator-w4-da9d07-kserve-7c7588b894-snqq4"] Oct 31 01:01:17.486870 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:17.486839 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0599b1e3-1033-48ab-bd20-b5446d96dd13-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0599b1e3-1033-48ab-bd20-b5446d96dd13-maas-api-openshift-ca\") pod \"0599b1e3-1033-48ab-bd20-b5446d96dd13\" (UID: \"0599b1e3-1033-48ab-bd20-b5446d96dd13\") " Oct 31 01:01:17.487072 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:17.486894 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d8b71ff-c6de-4b8c-8e78-175b48c7ed2e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d8b71ff-c6de-4b8c-8e78-175b48c7ed2e-maas-api-xks-ca\") pod \"0d8b71ff-c6de-4b8c-8e78-175b48c7ed2e\" (UID: \"0d8b71ff-c6de-4b8c-8e78-175b48c7ed2e\") " Oct 31 01:01:17.487072 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:17.486917 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d8b71ff-c6de-4b8c-8e78-175b48c7ed2e-kube-api-access-2hd7c\" (UniqueName: \"kubernetes.io/projected/0d8b71ff-c6de-4b8c-8e78-175b48c7ed2e-kube-api-access-2hd7c\") pod \"0d8b71ff-c6de-4b8c-8e78-175b48c7ed2e\" (UID: \"0d8b71ff-c6de-4b8c-8e78-175b48c7ed2e\") " Oct 31 01:01:17.487072 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:17.486948 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0599b1e3-1033-48ab-bd20-b5446d96dd13-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0599b1e3-1033-48ab-bd20-b5446d96dd13-maas-api-xks-ca\") pod \"0599b1e3-1033-48ab-bd20-b5446d96dd13\" (UID: \"0599b1e3-1033-48ab-bd20-b5446d96dd13\") " Oct 31 01:01:17.487072 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:17.486977 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0599b1e3-1033-48ab-bd20-b5446d96dd13-kube-api-access-vgf6m\" (UniqueName: \"kubernetes.io/projected/0599b1e3-1033-48ab-bd20-b5446d96dd13-kube-api-access-vgf6m\") pod \"0599b1e3-1033-48ab-bd20-b5446d96dd13\" (UID: \"0599b1e3-1033-48ab-bd20-b5446d96dd13\") " Oct 31 01:01:17.487072 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:17.487026 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d8b71ff-c6de-4b8c-8e78-175b48c7ed2e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d8b71ff-c6de-4b8c-8e78-175b48c7ed2e-maas-api-openshift-ca\") pod \"0d8b71ff-c6de-4b8c-8e78-175b48c7ed2e\" (UID: \"0d8b71ff-c6de-4b8c-8e78-175b48c7ed2e\") " Oct 31 01:01:17.487436 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:17.487291 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0599b1e3-1033-48ab-bd20-b5446d96dd13-maas-api-openshift-ca" pod "0599b1e3-1033-48ab-bd20-b5446d96dd13" (UID: "0599b1e3-1033-48ab-bd20-b5446d96dd13"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:01:17.487507 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:17.487483 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d8b71ff-c6de-4b8c-8e78-175b48c7ed2e-maas-api-openshift-ca" pod "0d8b71ff-c6de-4b8c-8e78-175b48c7ed2e" (UID: "0d8b71ff-c6de-4b8c-8e78-175b48c7ed2e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:01:17.489333 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:17.489305 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0599b1e3-1033-48ab-bd20-b5446d96dd13-kube-api-access-vgf6m" pod "0599b1e3-1033-48ab-bd20-b5446d96dd13" (UID: "0599b1e3-1033-48ab-bd20-b5446d96dd13"). InnerVolumeSpecName "kube-api-access-vgf6m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:01:17.489474 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:17.489404 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d8b71ff-c6de-4b8c-8e78-175b48c7ed2e-kube-api-access-2hd7c" pod "0d8b71ff-c6de-4b8c-8e78-175b48c7ed2e" (UID: "0d8b71ff-c6de-4b8c-8e78-175b48c7ed2e"). InnerVolumeSpecName "kube-api-access-2hd7c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:01:17.489719 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:17.489696 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d8b71ff-c6de-4b8c-8e78-175b48c7ed2e-maas-api-xks-ca" pod "0d8b71ff-c6de-4b8c-8e78-175b48c7ed2e" (UID: "0d8b71ff-c6de-4b8c-8e78-175b48c7ed2e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:01:17.489786 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:17.489714 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0599b1e3-1033-48ab-bd20-b5446d96dd13-maas-api-xks-ca" pod "0599b1e3-1033-48ab-bd20-b5446d96dd13" (UID: "0599b1e3-1033-48ab-bd20-b5446d96dd13"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:01:17.588155 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:17.588078 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hd7c\" (UniqueName: \"kubernetes.io/projected/0d8b71ff-c6de-4b8c-8e78-175b48c7ed2e-kube-api-access-2hd7c\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:01:17.588155 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:17.588109 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0599b1e3-1033-48ab-bd20-b5446d96dd13-maas-api-xks-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:01:17.588155 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:17.588118 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgf6m\" (UniqueName: \"kubernetes.io/projected/0599b1e3-1033-48ab-bd20-b5446d96dd13-kube-api-access-vgf6m\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:01:17.588155 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:17.588128 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d8b71ff-c6de-4b8c-8e78-175b48c7ed2e-maas-api-openshift-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:01:17.588155 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:17.588137 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0599b1e3-1033-48ab-bd20-b5446d96dd13-maas-api-openshift-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:01:17.588155 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:17.588146 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d8b71ff-c6de-4b8c-8e78-175b48c7ed2e-maas-api-xks-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:01:17.654911 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:17.654876 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d37ba9a-a648-44b2-aa04-4acf2b503a08" path="/var/lib/kubelet/pods/3d37ba9a-a648-44b2-aa04-4acf2b503a08/volumes" Oct 31 01:01:18.282075 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:18.282041 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fec8671-lhx4k" Oct 31 01:01:18.282554 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:18.282040 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fec8671-lhx4k" event={"ID":"0599b1e3-1033-48ab-bd20-b5446d96dd13","Type":"ContainerDied","Data":"38f7c086742b14ef42a9344d5df0ddf8559543952e38cfc83fbfc42219219a64"} Oct 31 01:01:18.282554 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:18.282155 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="38f7c086742b14ef42a9344d5df0ddf8559543952e38cfc83fbfc42219219a64" Oct 31 01:01:18.283616 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:18.283596 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-784a983c-mttgm" Oct 31 01:01:18.283757 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:18.283597 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-784a983c-mttgm" event={"ID":"0d8b71ff-c6de-4b8c-8e78-175b48c7ed2e","Type":"ContainerDied","Data":"487c273f5c53f9313d29f1314439fc902274513023b8d8a50b2e2b035fd7ba30"} Oct 31 01:01:18.283757 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:18.283727 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="487c273f5c53f9313d29f1314439fc902274513023b8d8a50b2e2b035fd7ba30" Oct 31 01:01:23.986707 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:23.986664 5222 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-da9d07-69bb55d779-hg9l2" podUID="1b4de77a-88d7-466f-a8a4-37c224ab4a3c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:01:23.987129 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:23.986974 5222 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-da9d07-5848fd449-s4wwm" podUID="bc1b882b-b51a-4d28-a9ee-35362dd92d69" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:01:30.028058 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:30.028021 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-79tk8"] Oct 31 01:01:31.395663 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:31.395620 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-da9d07-8g95z"] Oct 31 01:01:31.446966 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:31.446922 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-da9d07-8g95z"] Oct 31 01:01:31.654769 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:31.654686 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98201195-ac44-4539-92b7-fee5bacb2703" path="/var/lib/kubelet/pods/98201195-ac44-4539-92b7-fee5bacb2703/volumes" Oct 31 01:01:32.379261 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:32.379229 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l9jbt"] Oct 31 01:01:32.379492 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:32.379469 5222 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l9jbt" podUID="a6ca1829-40e1-442b-b711-1d57b9900f4d" containerName="istio-proxy" containerID="cri-o://6c765868dfae014b2d1ae045ff92020b741de13cc5887a032fc66672618370d1" gracePeriod=30 Oct 31 01:01:32.430788 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:32.430750 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2rzpl"] Oct 31 01:01:32.431234 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:32.430958 5222 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2rzpl" podUID="5a711fef-fe18-4853-b9c5-c3138bdb9cbf" containerName="istio-proxy" containerID="cri-o://e950c3bd9cbbbc103d2bd5de16a582b3def0c5988d812ad96e3c9a64b6a234a3" gracePeriod=30 Oct 31 01:01:33.594355 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:33.594322 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-da9d07-5848fd449-s4wwm" Oct 31 01:01:33.689678 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:33.689654 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-da9d07-69bb55d779-hg9l2" Oct 31 01:01:33.765601 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:33.765556 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc1b882b-b51a-4d28-a9ee-35362dd92d69-kube-api-access-rdlk6\" (UniqueName: \"kubernetes.io/projected/bc1b882b-b51a-4d28-a9ee-35362dd92d69-kube-api-access-rdlk6\") pod \"bc1b882b-b51a-4d28-a9ee-35362dd92d69\" (UID: \"bc1b882b-b51a-4d28-a9ee-35362dd92d69\") " Oct 31 01:01:33.765767 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:33.765707 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc1b882b-b51a-4d28-a9ee-35362dd92d69-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc1b882b-b51a-4d28-a9ee-35362dd92d69-plugins-config-volume\") pod \"bc1b882b-b51a-4d28-a9ee-35362dd92d69\" (UID: \"bc1b882b-b51a-4d28-a9ee-35362dd92d69\") " Oct 31 01:01:33.766051 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:33.766026 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc1b882b-b51a-4d28-a9ee-35362dd92d69-plugins-config-volume" pod "bc1b882b-b51a-4d28-a9ee-35362dd92d69" (UID: "bc1b882b-b51a-4d28-a9ee-35362dd92d69"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:01:33.767966 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:33.767938 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc1b882b-b51a-4d28-a9ee-35362dd92d69-kube-api-access-rdlk6" pod "bc1b882b-b51a-4d28-a9ee-35362dd92d69" (UID: "bc1b882b-b51a-4d28-a9ee-35362dd92d69"). InnerVolumeSpecName "kube-api-access-rdlk6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:01:33.866280 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:33.866189 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b4de77a-88d7-466f-a8a4-37c224ab4a3c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b4de77a-88d7-466f-a8a4-37c224ab4a3c-plugins-config-volume\") pod \"1b4de77a-88d7-466f-a8a4-37c224ab4a3c\" (UID: \"1b4de77a-88d7-466f-a8a4-37c224ab4a3c\") " Oct 31 01:01:33.866280 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:33.866234 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b4de77a-88d7-466f-a8a4-37c224ab4a3c-kube-api-access-6s8ml\" (UniqueName: \"kubernetes.io/projected/1b4de77a-88d7-466f-a8a4-37c224ab4a3c-kube-api-access-6s8ml\") pod \"1b4de77a-88d7-466f-a8a4-37c224ab4a3c\" (UID: \"1b4de77a-88d7-466f-a8a4-37c224ab4a3c\") " Oct 31 01:01:33.866582 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:33.866567 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdlk6\" (UniqueName: \"kubernetes.io/projected/bc1b882b-b51a-4d28-a9ee-35362dd92d69-kube-api-access-rdlk6\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:01:33.866623 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:33.866584 5222 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc1b882b-b51a-4d28-a9ee-35362dd92d69-plugins-config-volume\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:01:33.866623 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:33.866606 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b4de77a-88d7-466f-a8a4-37c224ab4a3c-plugins-config-volume" pod "1b4de77a-88d7-466f-a8a4-37c224ab4a3c" (UID: "1b4de77a-88d7-466f-a8a4-37c224ab4a3c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:01:33.868656 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:33.868632 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b4de77a-88d7-466f-a8a4-37c224ab4a3c-kube-api-access-6s8ml" pod "1b4de77a-88d7-466f-a8a4-37c224ab4a3c" (UID: "1b4de77a-88d7-466f-a8a4-37c224ab4a3c"). InnerVolumeSpecName "kube-api-access-6s8ml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:01:33.967672 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:33.967632 5222 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b4de77a-88d7-466f-a8a4-37c224ab4a3c-plugins-config-volume\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:01:33.967672 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:33.967665 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6s8ml\" (UniqueName: \"kubernetes.io/projected/1b4de77a-88d7-466f-a8a4-37c224ab4a3c-kube-api-access-6s8ml\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:01:34.367277 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:34.367244 5222 generic.go:356] "Generic (PLEG): container finished" podID="bc1b882b-b51a-4d28-a9ee-35362dd92d69" containerID="b0bb94e6b83687089012e92190f7d97b981d645b7eb64472835acd134c959db8" exitCode=0 Oct 31 01:01:34.367504 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:34.367294 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-da9d07-5848fd449-s4wwm" Oct 31 01:01:34.367504 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:34.367323 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-da9d07-5848fd449-s4wwm" event={"ID":"bc1b882b-b51a-4d28-a9ee-35362dd92d69","Type":"ContainerDied","Data":"b0bb94e6b83687089012e92190f7d97b981d645b7eb64472835acd134c959db8"} Oct 31 01:01:34.367504 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:34.367390 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-da9d07-5848fd449-s4wwm" event={"ID":"bc1b882b-b51a-4d28-a9ee-35362dd92d69","Type":"ContainerDied","Data":"862ab2b8a85cdafc54e1333db78abf7056b30b930de2b3116028825629410af9"} Oct 31 01:01:34.367504 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:34.367414 5222 scope.go:122] "RemoveContainer" containerID="b0bb94e6b83687089012e92190f7d97b981d645b7eb64472835acd134c959db8" Oct 31 01:01:34.368868 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:34.368834 5222 generic.go:356] "Generic (PLEG): container finished" podID="1b4de77a-88d7-466f-a8a4-37c224ab4a3c" containerID="9e0e113793b61d5918a3a4d25a7690379e8f177c50465b69b9ff577361cbdd12" exitCode=0 Oct 31 01:01:34.368978 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:34.368877 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-da9d07-69bb55d779-hg9l2" Oct 31 01:01:34.368978 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:34.368894 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-da9d07-69bb55d779-hg9l2" event={"ID":"1b4de77a-88d7-466f-a8a4-37c224ab4a3c","Type":"ContainerDied","Data":"9e0e113793b61d5918a3a4d25a7690379e8f177c50465b69b9ff577361cbdd12"} Oct 31 01:01:34.368978 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:34.368943 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-da9d07-69bb55d779-hg9l2" event={"ID":"1b4de77a-88d7-466f-a8a4-37c224ab4a3c","Type":"ContainerDied","Data":"ac620df1fa3e62df015b12095fd4fa332bf0e49ec55441a6889bdce7ae348894"} Oct 31 01:01:34.378733 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:34.378710 5222 scope.go:122] "RemoveContainer" containerID="b0bb94e6b83687089012e92190f7d97b981d645b7eb64472835acd134c959db8" Oct 31 01:01:34.379107 ip-10-0-137-56 kubenswrapper[5222]: E1031 01:01:34.379083 5222 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0bb94e6b83687089012e92190f7d97b981d645b7eb64472835acd134c959db8\": container with ID starting with b0bb94e6b83687089012e92190f7d97b981d645b7eb64472835acd134c959db8 not found: ID does not exist" containerID="b0bb94e6b83687089012e92190f7d97b981d645b7eb64472835acd134c959db8" Oct 31 01:01:34.379203 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:34.379114 5222 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0bb94e6b83687089012e92190f7d97b981d645b7eb64472835acd134c959db8"} err="failed to get container status \"b0bb94e6b83687089012e92190f7d97b981d645b7eb64472835acd134c959db8\": rpc error: code = NotFound desc = could not find container \"b0bb94e6b83687089012e92190f7d97b981d645b7eb64472835acd134c959db8\": container with ID starting with b0bb94e6b83687089012e92190f7d97b981d645b7eb64472835acd134c959db8 not found: ID does not exist" Oct 31 01:01:34.379203 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:34.379131 5222 scope.go:122] "RemoveContainer" containerID="9e0e113793b61d5918a3a4d25a7690379e8f177c50465b69b9ff577361cbdd12" Oct 31 01:01:34.387826 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:34.387805 5222 scope.go:122] "RemoveContainer" containerID="9e0e113793b61d5918a3a4d25a7690379e8f177c50465b69b9ff577361cbdd12" Oct 31 01:01:34.388087 ip-10-0-137-56 kubenswrapper[5222]: E1031 01:01:34.388065 5222 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e0e113793b61d5918a3a4d25a7690379e8f177c50465b69b9ff577361cbdd12\": container with ID starting with 9e0e113793b61d5918a3a4d25a7690379e8f177c50465b69b9ff577361cbdd12 not found: ID does not exist" containerID="9e0e113793b61d5918a3a4d25a7690379e8f177c50465b69b9ff577361cbdd12" Oct 31 01:01:34.388131 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:34.388095 5222 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e0e113793b61d5918a3a4d25a7690379e8f177c50465b69b9ff577361cbdd12"} err="failed to get container status \"9e0e113793b61d5918a3a4d25a7690379e8f177c50465b69b9ff577361cbdd12\": rpc error: code = NotFound desc = could not find container \"9e0e113793b61d5918a3a4d25a7690379e8f177c50465b69b9ff577361cbdd12\": container with ID starting with 9e0e113793b61d5918a3a4d25a7690379e8f177c50465b69b9ff577361cbdd12 not found: ID does not exist" Oct 31 01:01:34.535875 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:34.535839 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-da9d07-69bb55d779-hg9l2"] Oct 31 01:01:34.586816 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:34.586778 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-da9d07-69bb55d779-hg9l2"] Oct 31 01:01:34.652956 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:34.652877 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-da9d07-5848fd449-s4wwm"] Oct 31 01:01:34.657930 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:34.657901 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-da9d07-5848fd449-s4wwm"] Oct 31 01:01:35.654286 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:35.654247 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b4de77a-88d7-466f-a8a4-37c224ab4a3c" path="/var/lib/kubelet/pods/1b4de77a-88d7-466f-a8a4-37c224ab4a3c/volumes" Oct 31 01:01:35.654808 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:35.654667 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc1b882b-b51a-4d28-a9ee-35362dd92d69" path="/var/lib/kubelet/pods/bc1b882b-b51a-4d28-a9ee-35362dd92d69/volumes" Oct 31 01:01:35.890165 ip-10-0-137-56 kubenswrapper[5222]: E1031 01:01:35.890128 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 01:01:37.698152 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:37.698122 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l9jbt" Oct 31 01:01:37.714527 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:37.714501 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2rzpl" Oct 31 01:01:37.811075 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:37.811041 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6ca1829-40e1-442b-b711-1d57b9900f4d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6ca1829-40e1-442b-b711-1d57b9900f4d-istio-data\") pod \"a6ca1829-40e1-442b-b711-1d57b9900f4d\" (UID: \"a6ca1829-40e1-442b-b711-1d57b9900f4d\") " Oct 31 01:01:37.811264 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:37.811085 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6ca1829-40e1-442b-b711-1d57b9900f4d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6ca1829-40e1-442b-b711-1d57b9900f4d-istiod-ca-cert\") pod \"a6ca1829-40e1-442b-b711-1d57b9900f4d\" (UID: \"a6ca1829-40e1-442b-b711-1d57b9900f4d\") " Oct 31 01:01:37.811264 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:37.811133 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6ca1829-40e1-442b-b711-1d57b9900f4d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6ca1829-40e1-442b-b711-1d57b9900f4d-istio-envoy\") pod \"a6ca1829-40e1-442b-b711-1d57b9900f4d\" (UID: \"a6ca1829-40e1-442b-b711-1d57b9900f4d\") " Oct 31 01:01:37.811264 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:37.811158 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6ca1829-40e1-442b-b711-1d57b9900f4d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6ca1829-40e1-442b-b711-1d57b9900f4d-workload-socket\") pod \"a6ca1829-40e1-442b-b711-1d57b9900f4d\" (UID: \"a6ca1829-40e1-442b-b711-1d57b9900f4d\") " Oct 31 01:01:37.811264 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:37.811241 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a6ca1829-40e1-442b-b711-1d57b9900f4d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6ca1829-40e1-442b-b711-1d57b9900f4d-istio-podinfo\") pod \"a6ca1829-40e1-442b-b711-1d57b9900f4d\" (UID: \"a6ca1829-40e1-442b-b711-1d57b9900f4d\") " Oct 31 01:01:37.811556 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:37.811274 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6ca1829-40e1-442b-b711-1d57b9900f4d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6ca1829-40e1-442b-b711-1d57b9900f4d-credential-socket\") pod \"a6ca1829-40e1-442b-b711-1d57b9900f4d\" (UID: \"a6ca1829-40e1-442b-b711-1d57b9900f4d\") " Oct 31 01:01:37.811556 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:37.811388 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6ca1829-40e1-442b-b711-1d57b9900f4d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6ca1829-40e1-442b-b711-1d57b9900f4d-workload-certs\") pod \"a6ca1829-40e1-442b-b711-1d57b9900f4d\" (UID: \"a6ca1829-40e1-442b-b711-1d57b9900f4d\") " Oct 31 01:01:37.811556 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:37.811419 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6ca1829-40e1-442b-b711-1d57b9900f4d-istio-token\" (UniqueName: \"kubernetes.io/projected/a6ca1829-40e1-442b-b711-1d57b9900f4d-istio-token\") pod \"a6ca1829-40e1-442b-b711-1d57b9900f4d\" (UID: \"a6ca1829-40e1-442b-b711-1d57b9900f4d\") " Oct 31 01:01:37.811556 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:37.811457 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6ca1829-40e1-442b-b711-1d57b9900f4d-istio-data" pod "a6ca1829-40e1-442b-b711-1d57b9900f4d" (UID: "a6ca1829-40e1-442b-b711-1d57b9900f4d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:01:37.811556 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:37.811466 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6ca1829-40e1-442b-b711-1d57b9900f4d-kube-api-access-tgdbr\" (UniqueName: \"kubernetes.io/projected/a6ca1829-40e1-442b-b711-1d57b9900f4d-kube-api-access-tgdbr\") pod \"a6ca1829-40e1-442b-b711-1d57b9900f4d\" (UID: \"a6ca1829-40e1-442b-b711-1d57b9900f4d\") " Oct 31 01:01:37.811827 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:37.811679 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6ca1829-40e1-442b-b711-1d57b9900f4d-istiod-ca-cert" pod "a6ca1829-40e1-442b-b711-1d57b9900f4d" (UID: "a6ca1829-40e1-442b-b711-1d57b9900f4d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:01:37.811885 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:37.811857 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6ca1829-40e1-442b-b711-1d57b9900f4d-credential-socket" pod "a6ca1829-40e1-442b-b711-1d57b9900f4d" (UID: "a6ca1829-40e1-442b-b711-1d57b9900f4d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:01:37.811885 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:37.811875 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6ca1829-40e1-442b-b711-1d57b9900f4d-workload-socket" pod "a6ca1829-40e1-442b-b711-1d57b9900f4d" (UID: "a6ca1829-40e1-442b-b711-1d57b9900f4d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:01:37.812177 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:37.812055 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6ca1829-40e1-442b-b711-1d57b9900f4d-workload-certs" pod "a6ca1829-40e1-442b-b711-1d57b9900f4d" (UID: "a6ca1829-40e1-442b-b711-1d57b9900f4d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:01:37.812177 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:37.812020 5222 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6ca1829-40e1-442b-b711-1d57b9900f4d-credential-socket\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:01:37.812177 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:37.812108 5222 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6ca1829-40e1-442b-b711-1d57b9900f4d-istio-data\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:01:37.812177 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:37.812123 5222 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6ca1829-40e1-442b-b711-1d57b9900f4d-istiod-ca-cert\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:01:37.812177 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:37.812137 5222 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6ca1829-40e1-442b-b711-1d57b9900f4d-workload-socket\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:01:37.813816 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:37.813787 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6ca1829-40e1-442b-b711-1d57b9900f4d-istio-envoy" pod "a6ca1829-40e1-442b-b711-1d57b9900f4d" (UID: "a6ca1829-40e1-442b-b711-1d57b9900f4d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:01:37.814064 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:37.814044 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6ca1829-40e1-442b-b711-1d57b9900f4d-istio-token" pod "a6ca1829-40e1-442b-b711-1d57b9900f4d" (UID: "a6ca1829-40e1-442b-b711-1d57b9900f4d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:01:37.814429 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:37.814407 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6ca1829-40e1-442b-b711-1d57b9900f4d-kube-api-access-tgdbr" pod "a6ca1829-40e1-442b-b711-1d57b9900f4d" (UID: "a6ca1829-40e1-442b-b711-1d57b9900f4d"). InnerVolumeSpecName "kube-api-access-tgdbr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:01:37.814551 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:37.814525 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a6ca1829-40e1-442b-b711-1d57b9900f4d-istio-podinfo" pod "a6ca1829-40e1-442b-b711-1d57b9900f4d" (UID: "a6ca1829-40e1-442b-b711-1d57b9900f4d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:01:37.913399 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:37.913355 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a711fef-fe18-4853-b9c5-c3138bdb9cbf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a711fef-fe18-4853-b9c5-c3138bdb9cbf-credential-socket\") pod \"5a711fef-fe18-4853-b9c5-c3138bdb9cbf\" (UID: \"5a711fef-fe18-4853-b9c5-c3138bdb9cbf\") " Oct 31 01:01:37.913586 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:37.913414 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a711fef-fe18-4853-b9c5-c3138bdb9cbf-kube-api-access-7hg8q\" (UniqueName: \"kubernetes.io/projected/5a711fef-fe18-4853-b9c5-c3138bdb9cbf-kube-api-access-7hg8q\") pod \"5a711fef-fe18-4853-b9c5-c3138bdb9cbf\" (UID: \"5a711fef-fe18-4853-b9c5-c3138bdb9cbf\") " Oct 31 01:01:37.913586 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:37.913434 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5a711fef-fe18-4853-b9c5-c3138bdb9cbf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a711fef-fe18-4853-b9c5-c3138bdb9cbf-istio-podinfo\") pod \"5a711fef-fe18-4853-b9c5-c3138bdb9cbf\" (UID: \"5a711fef-fe18-4853-b9c5-c3138bdb9cbf\") " Oct 31 01:01:37.913586 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:37.913505 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a711fef-fe18-4853-b9c5-c3138bdb9cbf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a711fef-fe18-4853-b9c5-c3138bdb9cbf-workload-socket\") pod \"5a711fef-fe18-4853-b9c5-c3138bdb9cbf\" (UID: \"5a711fef-fe18-4853-b9c5-c3138bdb9cbf\") " Oct 31 01:01:37.913731 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:37.913649 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a711fef-fe18-4853-b9c5-c3138bdb9cbf-credential-socket" pod "5a711fef-fe18-4853-b9c5-c3138bdb9cbf" (UID: "5a711fef-fe18-4853-b9c5-c3138bdb9cbf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:01:37.913731 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:37.913660 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a711fef-fe18-4853-b9c5-c3138bdb9cbf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a711fef-fe18-4853-b9c5-c3138bdb9cbf-istio-data\") pod \"5a711fef-fe18-4853-b9c5-c3138bdb9cbf\" (UID: \"5a711fef-fe18-4853-b9c5-c3138bdb9cbf\") " Oct 31 01:01:37.913731 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:37.913714 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a711fef-fe18-4853-b9c5-c3138bdb9cbf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a711fef-fe18-4853-b9c5-c3138bdb9cbf-workload-certs\") pod \"5a711fef-fe18-4853-b9c5-c3138bdb9cbf\" (UID: \"5a711fef-fe18-4853-b9c5-c3138bdb9cbf\") " Oct 31 01:01:37.913933 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:37.913744 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a711fef-fe18-4853-b9c5-c3138bdb9cbf-istio-token\" (UniqueName: \"kubernetes.io/projected/5a711fef-fe18-4853-b9c5-c3138bdb9cbf-istio-token\") pod \"5a711fef-fe18-4853-b9c5-c3138bdb9cbf\" (UID: \"5a711fef-fe18-4853-b9c5-c3138bdb9cbf\") " Oct 31 01:01:37.913933 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:37.913777 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a711fef-fe18-4853-b9c5-c3138bdb9cbf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a711fef-fe18-4853-b9c5-c3138bdb9cbf-istio-envoy\") pod \"5a711fef-fe18-4853-b9c5-c3138bdb9cbf\" (UID: \"5a711fef-fe18-4853-b9c5-c3138bdb9cbf\") " Oct 31 01:01:37.913933 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:37.913807 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a711fef-fe18-4853-b9c5-c3138bdb9cbf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a711fef-fe18-4853-b9c5-c3138bdb9cbf-istiod-ca-cert\") pod \"5a711fef-fe18-4853-b9c5-c3138bdb9cbf\" (UID: \"5a711fef-fe18-4853-b9c5-c3138bdb9cbf\") " Oct 31 01:01:37.913933 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:37.913856 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a711fef-fe18-4853-b9c5-c3138bdb9cbf-workload-socket" pod "5a711fef-fe18-4853-b9c5-c3138bdb9cbf" (UID: "5a711fef-fe18-4853-b9c5-c3138bdb9cbf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:01:37.913933 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:37.913896 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a711fef-fe18-4853-b9c5-c3138bdb9cbf-istio-data" pod "5a711fef-fe18-4853-b9c5-c3138bdb9cbf" (UID: "5a711fef-fe18-4853-b9c5-c3138bdb9cbf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:01:37.914244 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:37.913954 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a711fef-fe18-4853-b9c5-c3138bdb9cbf-workload-certs" pod "5a711fef-fe18-4853-b9c5-c3138bdb9cbf" (UID: "5a711fef-fe18-4853-b9c5-c3138bdb9cbf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:01:37.914244 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:37.914208 5222 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a711fef-fe18-4853-b9c5-c3138bdb9cbf-workload-certs\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:01:37.914244 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:37.914228 5222 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6ca1829-40e1-442b-b711-1d57b9900f4d-istio-envoy\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:01:37.914244 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:37.914243 5222 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6ca1829-40e1-442b-b711-1d57b9900f4d-istio-podinfo\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:01:37.914651 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:37.914259 5222 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a711fef-fe18-4853-b9c5-c3138bdb9cbf-credential-socket\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:01:37.914651 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:37.914271 5222 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a711fef-fe18-4853-b9c5-c3138bdb9cbf-workload-socket\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:01:37.914651 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:37.914279 5222 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6ca1829-40e1-442b-b711-1d57b9900f4d-workload-certs\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:01:37.914651 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:37.914290 5222 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6ca1829-40e1-442b-b711-1d57b9900f4d-istio-token\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:01:37.914651 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:37.914304 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgdbr\" (UniqueName: \"kubernetes.io/projected/a6ca1829-40e1-442b-b711-1d57b9900f4d-kube-api-access-tgdbr\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:01:37.914651 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:37.914243 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a711fef-fe18-4853-b9c5-c3138bdb9cbf-istiod-ca-cert" pod "5a711fef-fe18-4853-b9c5-c3138bdb9cbf" (UID: "5a711fef-fe18-4853-b9c5-c3138bdb9cbf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:01:37.914651 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:37.914318 5222 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a711fef-fe18-4853-b9c5-c3138bdb9cbf-istio-data\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:01:37.915928 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:37.915899 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5a711fef-fe18-4853-b9c5-c3138bdb9cbf-istio-podinfo" pod "5a711fef-fe18-4853-b9c5-c3138bdb9cbf" (UID: "5a711fef-fe18-4853-b9c5-c3138bdb9cbf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:01:37.916046 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:37.915930 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a711fef-fe18-4853-b9c5-c3138bdb9cbf-kube-api-access-7hg8q" pod "5a711fef-fe18-4853-b9c5-c3138bdb9cbf" (UID: "5a711fef-fe18-4853-b9c5-c3138bdb9cbf"). InnerVolumeSpecName "kube-api-access-7hg8q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:01:37.916112 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:37.916051 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a711fef-fe18-4853-b9c5-c3138bdb9cbf-istio-envoy" pod "5a711fef-fe18-4853-b9c5-c3138bdb9cbf" (UID: "5a711fef-fe18-4853-b9c5-c3138bdb9cbf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:01:37.916158 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:37.916147 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a711fef-fe18-4853-b9c5-c3138bdb9cbf-istio-token" pod "5a711fef-fe18-4853-b9c5-c3138bdb9cbf" (UID: "5a711fef-fe18-4853-b9c5-c3138bdb9cbf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:01:38.015034 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:38.015003 5222 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a711fef-fe18-4853-b9c5-c3138bdb9cbf-istio-token\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:01:38.015034 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:38.015034 5222 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a711fef-fe18-4853-b9c5-c3138bdb9cbf-istio-envoy\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:01:38.015226 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:38.015043 5222 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a711fef-fe18-4853-b9c5-c3138bdb9cbf-istiod-ca-cert\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:01:38.015226 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:38.015055 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hg8q\" (UniqueName: \"kubernetes.io/projected/5a711fef-fe18-4853-b9c5-c3138bdb9cbf-kube-api-access-7hg8q\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:01:38.015226 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:38.015064 5222 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a711fef-fe18-4853-b9c5-c3138bdb9cbf-istio-podinfo\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:01:38.394306 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:38.394214 5222 generic.go:356] "Generic (PLEG): container finished" podID="5a711fef-fe18-4853-b9c5-c3138bdb9cbf" containerID="e950c3bd9cbbbc103d2bd5de16a582b3def0c5988d812ad96e3c9a64b6a234a3" exitCode=0 Oct 31 01:01:38.394306 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:38.394273 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2rzpl" Oct 31 01:01:38.394599 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:38.394301 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2rzpl" event={"ID":"5a711fef-fe18-4853-b9c5-c3138bdb9cbf","Type":"ContainerDied","Data":"e950c3bd9cbbbc103d2bd5de16a582b3def0c5988d812ad96e3c9a64b6a234a3"} Oct 31 01:01:38.394599 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:38.394365 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2rzpl" event={"ID":"5a711fef-fe18-4853-b9c5-c3138bdb9cbf","Type":"ContainerDied","Data":"fb673a6e705ad63f4dad815e5779fbfe8711278aa01c81f46009f0e71d034ef0"} Oct 31 01:01:38.394599 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:38.394413 5222 scope.go:122] "RemoveContainer" containerID="e950c3bd9cbbbc103d2bd5de16a582b3def0c5988d812ad96e3c9a64b6a234a3" Oct 31 01:01:38.395807 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:38.395783 5222 generic.go:356] "Generic (PLEG): container finished" podID="a6ca1829-40e1-442b-b711-1d57b9900f4d" containerID="6c765868dfae014b2d1ae045ff92020b741de13cc5887a032fc66672618370d1" exitCode=0 Oct 31 01:01:38.395960 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:38.395841 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l9jbt" Oct 31 01:01:38.395960 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:38.395877 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l9jbt" event={"ID":"a6ca1829-40e1-442b-b711-1d57b9900f4d","Type":"ContainerDied","Data":"6c765868dfae014b2d1ae045ff92020b741de13cc5887a032fc66672618370d1"} Oct 31 01:01:38.395960 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:38.395913 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l9jbt" event={"ID":"a6ca1829-40e1-442b-b711-1d57b9900f4d","Type":"ContainerDied","Data":"9aa9327b3bb8fae82ae9ea22034af05b1a5505dc289f8b569edb8987809d7621"} Oct 31 01:01:38.404394 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:38.404358 5222 scope.go:122] "RemoveContainer" containerID="e950c3bd9cbbbc103d2bd5de16a582b3def0c5988d812ad96e3c9a64b6a234a3" Oct 31 01:01:38.404798 ip-10-0-137-56 kubenswrapper[5222]: E1031 01:01:38.404767 5222 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e950c3bd9cbbbc103d2bd5de16a582b3def0c5988d812ad96e3c9a64b6a234a3\": container with ID starting with e950c3bd9cbbbc103d2bd5de16a582b3def0c5988d812ad96e3c9a64b6a234a3 not found: ID does not exist" containerID="e950c3bd9cbbbc103d2bd5de16a582b3def0c5988d812ad96e3c9a64b6a234a3" Oct 31 01:01:38.404898 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:38.404809 5222 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e950c3bd9cbbbc103d2bd5de16a582b3def0c5988d812ad96e3c9a64b6a234a3"} err="failed to get container status \"e950c3bd9cbbbc103d2bd5de16a582b3def0c5988d812ad96e3c9a64b6a234a3\": rpc error: code = NotFound desc = could not find container \"e950c3bd9cbbbc103d2bd5de16a582b3def0c5988d812ad96e3c9a64b6a234a3\": container with ID starting with e950c3bd9cbbbc103d2bd5de16a582b3def0c5988d812ad96e3c9a64b6a234a3 not found: ID does not exist" Oct 31 01:01:38.404898 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:38.404832 5222 scope.go:122] "RemoveContainer" containerID="6c765868dfae014b2d1ae045ff92020b741de13cc5887a032fc66672618370d1" Oct 31 01:01:38.419403 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:38.419348 5222 scope.go:122] "RemoveContainer" containerID="6c765868dfae014b2d1ae045ff92020b741de13cc5887a032fc66672618370d1" Oct 31 01:01:38.419769 ip-10-0-137-56 kubenswrapper[5222]: E1031 01:01:38.419741 5222 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c765868dfae014b2d1ae045ff92020b741de13cc5887a032fc66672618370d1\": container with ID starting with 6c765868dfae014b2d1ae045ff92020b741de13cc5887a032fc66672618370d1 not found: ID does not exist" containerID="6c765868dfae014b2d1ae045ff92020b741de13cc5887a032fc66672618370d1" Oct 31 01:01:38.419837 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:38.419781 5222 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c765868dfae014b2d1ae045ff92020b741de13cc5887a032fc66672618370d1"} err="failed to get container status \"6c765868dfae014b2d1ae045ff92020b741de13cc5887a032fc66672618370d1\": rpc error: code = NotFound desc = could not find container \"6c765868dfae014b2d1ae045ff92020b741de13cc5887a032fc66672618370d1\": container with ID starting with 6c765868dfae014b2d1ae045ff92020b741de13cc5887a032fc66672618370d1 not found: ID does not exist" Oct 31 01:01:38.565062 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:38.565032 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l9jbt"] Oct 31 01:01:38.568623 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:38.568595 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l9jbt"] Oct 31 01:01:38.682837 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:38.682753 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2rzpl"] Oct 31 01:01:38.686984 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:38.686957 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2rzpl"] Oct 31 01:01:39.656933 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:39.656886 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a711fef-fe18-4853-b9c5-c3138bdb9cbf" path="/var/lib/kubelet/pods/5a711fef-fe18-4853-b9c5-c3138bdb9cbf/volumes" Oct 31 01:01:39.657530 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:39.657510 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6ca1829-40e1-442b-b711-1d57b9900f4d" path="/var/lib/kubelet/pods/a6ca1829-40e1-442b-b711-1d57b9900f4d/volumes" Oct 31 01:01:43.502154 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:43.502119 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-79tk8"] Oct 31 01:01:52.396218 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:01:52.396133 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-79tk8"] Oct 31 01:02:02.630064 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:02.630029 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-256br"] Oct 31 01:02:02.630516 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:02.630414 5222 cpu_manager.go:414] "containerMap: removing container" podUID="1b4de77a-88d7-466f-a8a4-37c224ab4a3c" containerName="payload-pre-processing" Oct 31 01:02:02.630516 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:02.630426 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b4de77a-88d7-466f-a8a4-37c224ab4a3c" containerName="payload-pre-processing" Oct 31 01:02:02.630516 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:02.630442 5222 cpu_manager.go:414] "containerMap: removing container" podUID="3d37ba9a-a648-44b2-aa04-4acf2b503a08" containerName="main" Oct 31 01:02:02.630516 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:02.630447 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d37ba9a-a648-44b2-aa04-4acf2b503a08" containerName="main" Oct 31 01:02:02.630516 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:02.630454 5222 cpu_manager.go:414] "containerMap: removing container" podUID="0599b1e3-1033-48ab-bd20-b5446d96dd13" containerName="revoke-keys" Oct 31 01:02:02.630516 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:02.630458 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0599b1e3-1033-48ab-bd20-b5446d96dd13" containerName="revoke-keys" Oct 31 01:02:02.630516 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:02.630466 5222 cpu_manager.go:414] "containerMap: removing container" podUID="a6ca1829-40e1-442b-b711-1d57b9900f4d" containerName="istio-proxy" Oct 31 01:02:02.630516 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:02.630471 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6ca1829-40e1-442b-b711-1d57b9900f4d" containerName="istio-proxy" Oct 31 01:02:02.630516 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:02.630477 5222 cpu_manager.go:414] "containerMap: removing container" podUID="5a711fef-fe18-4853-b9c5-c3138bdb9cbf" containerName="istio-proxy" Oct 31 01:02:02.630516 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:02.630481 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a711fef-fe18-4853-b9c5-c3138bdb9cbf" containerName="istio-proxy" Oct 31 01:02:02.630516 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:02.630489 5222 cpu_manager.go:414] "containerMap: removing container" podUID="0d8b71ff-c6de-4b8c-8e78-175b48c7ed2e" containerName="revoke-keys" Oct 31 01:02:02.630516 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:02.630494 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d8b71ff-c6de-4b8c-8e78-175b48c7ed2e" containerName="revoke-keys" Oct 31 01:02:02.630516 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:02.630514 5222 cpu_manager.go:414] "containerMap: removing container" podUID="bc1b882b-b51a-4d28-a9ee-35362dd92d69" containerName="payload-processing" Oct 31 01:02:02.630516 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:02.630519 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc1b882b-b51a-4d28-a9ee-35362dd92d69" containerName="payload-processing" Oct 31 01:02:02.631048 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:02.630575 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc1b882b-b51a-4d28-a9ee-35362dd92d69" containerName="payload-processing" Oct 31 01:02:02.631048 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:02.630588 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b4de77a-88d7-466f-a8a4-37c224ab4a3c" containerName="payload-pre-processing" Oct 31 01:02:02.631048 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:02.630595 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d37ba9a-a648-44b2-aa04-4acf2b503a08" containerName="main" Oct 31 01:02:02.631048 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:02.630600 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="0599b1e3-1033-48ab-bd20-b5446d96dd13" containerName="revoke-keys" Oct 31 01:02:02.631048 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:02.630605 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6ca1829-40e1-442b-b711-1d57b9900f4d" containerName="istio-proxy" Oct 31 01:02:02.631048 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:02.630610 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a711fef-fe18-4853-b9c5-c3138bdb9cbf" containerName="istio-proxy" Oct 31 01:02:02.631048 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:02.630615 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d8b71ff-c6de-4b8c-8e78-175b48c7ed2e" containerName="revoke-keys" Oct 31 01:02:02.633627 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:02.633608 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-256br" Oct 31 01:02:02.731489 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:02.731445 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-256br"] Oct 31 01:02:02.748702 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:02.748661 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a723d3c6-4cd4-4e23-bf6b-6f2dfb4a38f4-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-256br\" (UID: \"a723d3c6-4cd4-4e23-bf6b-6f2dfb4a38f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-256br" Oct 31 01:02:02.748702 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:02.748709 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a723d3c6-4cd4-4e23-bf6b-6f2dfb4a38f4-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-256br\" (UID: \"a723d3c6-4cd4-4e23-bf6b-6f2dfb4a38f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-256br" Oct 31 01:02:02.748953 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:02.748846 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a723d3c6-4cd4-4e23-bf6b-6f2dfb4a38f4-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-256br\" (UID: \"a723d3c6-4cd4-4e23-bf6b-6f2dfb4a38f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-256br" Oct 31 01:02:02.748953 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:02.748887 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a723d3c6-4cd4-4e23-bf6b-6f2dfb4a38f4-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-256br\" (UID: \"a723d3c6-4cd4-4e23-bf6b-6f2dfb4a38f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-256br" Oct 31 01:02:02.748953 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:02.748920 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6ln7\" (UniqueName: \"kubernetes.io/projected/a723d3c6-4cd4-4e23-bf6b-6f2dfb4a38f4-kube-api-access-l6ln7\") pod \"maas-default-gateway-openshift-default-6886b96f9-256br\" (UID: \"a723d3c6-4cd4-4e23-bf6b-6f2dfb4a38f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-256br" Oct 31 01:02:02.749085 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:02.748972 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a723d3c6-4cd4-4e23-bf6b-6f2dfb4a38f4-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-256br\" (UID: \"a723d3c6-4cd4-4e23-bf6b-6f2dfb4a38f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-256br" Oct 31 01:02:02.749085 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:02.749001 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a723d3c6-4cd4-4e23-bf6b-6f2dfb4a38f4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-256br\" (UID: \"a723d3c6-4cd4-4e23-bf6b-6f2dfb4a38f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-256br" Oct 31 01:02:02.749085 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:02.749022 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a723d3c6-4cd4-4e23-bf6b-6f2dfb4a38f4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-256br\" (UID: \"a723d3c6-4cd4-4e23-bf6b-6f2dfb4a38f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-256br" Oct 31 01:02:02.749085 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:02.749039 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a723d3c6-4cd4-4e23-bf6b-6f2dfb4a38f4-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-256br\" (UID: \"a723d3c6-4cd4-4e23-bf6b-6f2dfb4a38f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-256br" Oct 31 01:02:02.849882 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:02.849835 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a723d3c6-4cd4-4e23-bf6b-6f2dfb4a38f4-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-256br\" (UID: \"a723d3c6-4cd4-4e23-bf6b-6f2dfb4a38f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-256br" Oct 31 01:02:02.849882 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:02.849882 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6ln7\" (UniqueName: \"kubernetes.io/projected/a723d3c6-4cd4-4e23-bf6b-6f2dfb4a38f4-kube-api-access-l6ln7\") pod \"maas-default-gateway-openshift-default-6886b96f9-256br\" (UID: \"a723d3c6-4cd4-4e23-bf6b-6f2dfb4a38f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-256br" Oct 31 01:02:02.850163 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:02.849902 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a723d3c6-4cd4-4e23-bf6b-6f2dfb4a38f4-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-256br\" (UID: \"a723d3c6-4cd4-4e23-bf6b-6f2dfb4a38f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-256br" Oct 31 01:02:02.850163 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:02.849926 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a723d3c6-4cd4-4e23-bf6b-6f2dfb4a38f4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-256br\" (UID: \"a723d3c6-4cd4-4e23-bf6b-6f2dfb4a38f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-256br" Oct 31 01:02:02.850163 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:02.849951 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a723d3c6-4cd4-4e23-bf6b-6f2dfb4a38f4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-256br\" (UID: \"a723d3c6-4cd4-4e23-bf6b-6f2dfb4a38f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-256br" Oct 31 01:02:02.850163 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:02.849972 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a723d3c6-4cd4-4e23-bf6b-6f2dfb4a38f4-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-256br\" (UID: \"a723d3c6-4cd4-4e23-bf6b-6f2dfb4a38f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-256br" Oct 31 01:02:02.850163 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:02.850050 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a723d3c6-4cd4-4e23-bf6b-6f2dfb4a38f4-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-256br\" (UID: \"a723d3c6-4cd4-4e23-bf6b-6f2dfb4a38f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-256br" Oct 31 01:02:02.850163 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:02.850076 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a723d3c6-4cd4-4e23-bf6b-6f2dfb4a38f4-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-256br\" (UID: \"a723d3c6-4cd4-4e23-bf6b-6f2dfb4a38f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-256br" Oct 31 01:02:02.850525 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:02.850159 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a723d3c6-4cd4-4e23-bf6b-6f2dfb4a38f4-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-256br\" (UID: \"a723d3c6-4cd4-4e23-bf6b-6f2dfb4a38f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-256br" Oct 31 01:02:02.850525 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:02.850320 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a723d3c6-4cd4-4e23-bf6b-6f2dfb4a38f4-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-256br\" (UID: \"a723d3c6-4cd4-4e23-bf6b-6f2dfb4a38f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-256br" Oct 31 01:02:02.850635 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:02.850520 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a723d3c6-4cd4-4e23-bf6b-6f2dfb4a38f4-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-256br\" (UID: \"a723d3c6-4cd4-4e23-bf6b-6f2dfb4a38f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-256br" Oct 31 01:02:02.850635 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:02.850614 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a723d3c6-4cd4-4e23-bf6b-6f2dfb4a38f4-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-256br\" (UID: \"a723d3c6-4cd4-4e23-bf6b-6f2dfb4a38f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-256br" Oct 31 01:02:02.850748 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:02.850654 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a723d3c6-4cd4-4e23-bf6b-6f2dfb4a38f4-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-256br\" (UID: \"a723d3c6-4cd4-4e23-bf6b-6f2dfb4a38f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-256br" Oct 31 01:02:02.850748 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:02.850705 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a723d3c6-4cd4-4e23-bf6b-6f2dfb4a38f4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-256br\" (UID: \"a723d3c6-4cd4-4e23-bf6b-6f2dfb4a38f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-256br" Oct 31 01:02:02.852542 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:02.852523 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a723d3c6-4cd4-4e23-bf6b-6f2dfb4a38f4-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-256br\" (UID: \"a723d3c6-4cd4-4e23-bf6b-6f2dfb4a38f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-256br" Oct 31 01:02:02.852738 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:02.852714 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a723d3c6-4cd4-4e23-bf6b-6f2dfb4a38f4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-256br\" (UID: \"a723d3c6-4cd4-4e23-bf6b-6f2dfb4a38f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-256br" Oct 31 01:02:02.913419 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:02.913317 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a723d3c6-4cd4-4e23-bf6b-6f2dfb4a38f4-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-256br\" (UID: \"a723d3c6-4cd4-4e23-bf6b-6f2dfb4a38f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-256br" Oct 31 01:02:02.913565 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:02.913445 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6ln7\" (UniqueName: \"kubernetes.io/projected/a723d3c6-4cd4-4e23-bf6b-6f2dfb4a38f4-kube-api-access-l6ln7\") pod \"maas-default-gateway-openshift-default-6886b96f9-256br\" (UID: \"a723d3c6-4cd4-4e23-bf6b-6f2dfb4a38f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-256br" Oct 31 01:02:02.947218 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:02.947185 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-256br" Oct 31 01:02:03.369714 ip-10-0-137-56 kubenswrapper[5222]: W1031 01:02:03.369676 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda723d3c6_4cd4_4e23_bf6b_6f2dfb4a38f4.slice/crio-1faa3ffacbe8e72bfd03ec816d185699debd1d5f84f830eafb8975fac7cba096 WatchSource:0}: Error finding container 1faa3ffacbe8e72bfd03ec816d185699debd1d5f84f830eafb8975fac7cba096: Status 404 returned error can't find the container with id 1faa3ffacbe8e72bfd03ec816d185699debd1d5f84f830eafb8975fac7cba096 Oct 31 01:02:03.370749 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:03.370724 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-256br"] Oct 31 01:02:03.371927 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:03.371889 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 01:02:03.371981 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:03.371954 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 01:02:03.372018 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:03.371989 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 01:02:03.535608 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:03.535565 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-256br" event={"ID":"a723d3c6-4cd4-4e23-bf6b-6f2dfb4a38f4","Type":"ContainerStarted","Data":"cc02aa62e27a40319058cf160f28f45146ef6582fcffa25b3e098a0078edc688"} Oct 31 01:02:03.535800 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:03.535639 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-256br" event={"ID":"a723d3c6-4cd4-4e23-bf6b-6f2dfb4a38f4","Type":"ContainerStarted","Data":"1faa3ffacbe8e72bfd03ec816d185699debd1d5f84f830eafb8975fac7cba096"} Oct 31 01:02:03.659910 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:03.659815 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-256br" podStartSLOduration=1.659798449 podStartE2EDuration="1.659798449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:02:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:02:03.658945334 +0000 UTC m=+2816.130115084" watchObservedRunningTime="2026-10-31 01:02:03.659798449 +0000 UTC m=+2816.130968198" Oct 31 01:02:03.962482 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:03.957519 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-256br" Oct 31 01:02:04.959395 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:04.955419 5222 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-256br container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.248:15021/healthz/ready\": dial tcp 10.134.0.248:15021: connect: connection refused" start-of-body= Oct 31 01:02:04.959395 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:04.955493 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-256br" podUID="a723d3c6-4cd4-4e23-bf6b-6f2dfb4a38f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.248:15021/healthz/ready\": dial tcp 10.134.0.248:15021: connect: connection refused" Oct 31 01:02:05.890786 ip-10-0-137-56 kubenswrapper[5222]: E1031 01:02:05.890752 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 01:02:05.948969 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:05.948929 5222 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-256br container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.248:15021/healthz/ready\": dial tcp 10.134.0.248:15021: connect: connection refused" start-of-body= Oct 31 01:02:05.949290 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:05.949251 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-256br" podUID="a723d3c6-4cd4-4e23-bf6b-6f2dfb4a38f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.248:15021/healthz/ready\": dial tcp 10.134.0.248:15021: connect: connection refused" Oct 31 01:02:06.958889 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:06.958851 5222 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-256br container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.248:15021/healthz/ready\": dial tcp 10.134.0.248:15021: connect: connection refused" start-of-body= Oct 31 01:02:06.959553 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:06.959520 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-256br" podUID="a723d3c6-4cd4-4e23-bf6b-6f2dfb4a38f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.248:15021/healthz/ready\": dial tcp 10.134.0.248:15021: connect: connection refused" Oct 31 01:02:07.818947 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:07.818905 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-bxnmv"] Oct 31 01:02:07.822408 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:07.822359 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-bxnmv" Oct 31 01:02:07.883559 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:07.883526 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-bxnmv"] Oct 31 01:02:07.905384 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:07.905345 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9937a9f-f743-4587-9a27-88a51f33b245-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-bxnmv\" (UID: \"b9937a9f-f743-4587-9a27-88a51f33b245\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-bxnmv" Oct 31 01:02:07.905580 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:07.905476 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8nt6\" (UniqueName: \"kubernetes.io/projected/b9937a9f-f743-4587-9a27-88a51f33b245-kube-api-access-m8nt6\") pod \"maas-api-revoke-sub-85237e45-bxnmv\" (UID: \"b9937a9f-f743-4587-9a27-88a51f33b245\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-bxnmv" Oct 31 01:02:07.905580 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:07.905553 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9937a9f-f743-4587-9a27-88a51f33b245-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-bxnmv\" (UID: \"b9937a9f-f743-4587-9a27-88a51f33b245\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-bxnmv" Oct 31 01:02:07.933283 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:07.933248 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:02:07.933283 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:07.933258 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:02:07.933534 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:07.933261 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:02:07.933534 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:07.933308 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfz9z\"" Oct 31 01:02:07.961397 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:07.958833 5222 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-256br container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.248:15021/healthz/ready\": dial tcp 10.134.0.248:15021: connect: connection refused" start-of-body= Oct 31 01:02:07.961397 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:07.958900 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-256br" podUID="a723d3c6-4cd4-4e23-bf6b-6f2dfb4a38f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.248:15021/healthz/ready\": dial tcp 10.134.0.248:15021: connect: connection refused" Oct 31 01:02:08.006788 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:08.006744 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9937a9f-f743-4587-9a27-88a51f33b245-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-bxnmv\" (UID: \"b9937a9f-f743-4587-9a27-88a51f33b245\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-bxnmv" Oct 31 01:02:08.006978 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:08.006818 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8nt6\" (UniqueName: \"kubernetes.io/projected/b9937a9f-f743-4587-9a27-88a51f33b245-kube-api-access-m8nt6\") pod \"maas-api-revoke-sub-85237e45-bxnmv\" (UID: \"b9937a9f-f743-4587-9a27-88a51f33b245\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-bxnmv" Oct 31 01:02:08.006978 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:08.006891 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9937a9f-f743-4587-9a27-88a51f33b245-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-bxnmv\" (UID: \"b9937a9f-f743-4587-9a27-88a51f33b245\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-bxnmv" Oct 31 01:02:08.007544 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:08.007517 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9937a9f-f743-4587-9a27-88a51f33b245-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-bxnmv\" (UID: \"b9937a9f-f743-4587-9a27-88a51f33b245\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-bxnmv" Oct 31 01:02:08.009527 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:08.009483 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9937a9f-f743-4587-9a27-88a51f33b245-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-bxnmv\" (UID: \"b9937a9f-f743-4587-9a27-88a51f33b245\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-bxnmv" Oct 31 01:02:08.069944 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:08.069861 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8nt6\" (UniqueName: \"kubernetes.io/projected/b9937a9f-f743-4587-9a27-88a51f33b245-kube-api-access-m8nt6\") pod \"maas-api-revoke-sub-85237e45-bxnmv\" (UID: \"b9937a9f-f743-4587-9a27-88a51f33b245\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-bxnmv" Oct 31 01:02:08.135584 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:08.135546 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-bxnmv" Oct 31 01:02:08.533164 ip-10-0-137-56 kubenswrapper[5222]: W1031 01:02:08.533135 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9937a9f_f743_4587_9a27_88a51f33b245.slice/crio-f807ebe5faae7ec3350217c8ababc47069605bf057479bd295875a713e5f4d74 WatchSource:0}: Error finding container f807ebe5faae7ec3350217c8ababc47069605bf057479bd295875a713e5f4d74: Status 404 returned error can't find the container with id f807ebe5faae7ec3350217c8ababc47069605bf057479bd295875a713e5f4d74 Oct 31 01:02:08.533905 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:08.533782 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-bxnmv"] Oct 31 01:02:08.588068 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:08.588031 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-bxnmv" event={"ID":"b9937a9f-f743-4587-9a27-88a51f33b245","Type":"ContainerStarted","Data":"f807ebe5faae7ec3350217c8ababc47069605bf057479bd295875a713e5f4d74"} Oct 31 01:02:08.961520 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:08.961480 5222 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-256br container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.248:15021/healthz/ready\": dial tcp 10.134.0.248:15021: connect: connection refused" start-of-body= Oct 31 01:02:08.962299 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:08.962264 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-256br" podUID="a723d3c6-4cd4-4e23-bf6b-6f2dfb4a38f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.248:15021/healthz/ready\": dial tcp 10.134.0.248:15021: connect: connection refused" Oct 31 01:02:09.602704 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:09.602664 5222 generic.go:356] "Generic (PLEG): container finished" podID="b9937a9f-f743-4587-9a27-88a51f33b245" containerID="1aa45fae45d595e5cb643d2f46a6dab8bd0f9f585bfffad88464e09a6bcab198" exitCode=0 Oct 31 01:02:09.602887 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:09.602730 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-bxnmv" event={"ID":"b9937a9f-f743-4587-9a27-88a51f33b245","Type":"ContainerDied","Data":"1aa45fae45d595e5cb643d2f46a6dab8bd0f9f585bfffad88464e09a6bcab198"} Oct 31 01:02:09.948399 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:09.948270 5222 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-256br container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.248:15021/healthz/ready\": dial tcp 10.134.0.248:15021: connect: connection refused" start-of-body= Oct 31 01:02:09.948582 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:09.948347 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-256br" podUID="a723d3c6-4cd4-4e23-bf6b-6f2dfb4a38f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.248:15021/healthz/ready\": dial tcp 10.134.0.248:15021: connect: connection refused" Oct 31 01:02:10.381696 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:10.381659 5222 scope.go:122] "RemoveContainer" containerID="3d5893debb832fbff12b529fe662172e633bda295ea59f4b3cf55cd185e9f802" Oct 31 01:02:10.390492 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:10.390470 5222 scope.go:122] "RemoveContainer" containerID="8a361ecef0d6ddb72b61057cd2706fe90dc8fa4db98208e2c04ef70c72122c04" Oct 31 01:02:10.399583 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:10.399552 5222 scope.go:122] "RemoveContainer" containerID="29c14a3f4780eaafdc8305f58522d2dbc7ab57689dda24210abaf7059642a199" Oct 31 01:02:10.410435 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:10.410407 5222 scope.go:122] "RemoveContainer" containerID="2367c2763e7e539f66669c4ca65bcd1c2e193dd1ff670567ba15c0df69f39857" Oct 31 01:02:10.418758 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:10.418734 5222 scope.go:122] "RemoveContainer" containerID="c0ffd0c565dae97df963b3ef8f27203430423a2ddaca0f17ba1ca7cbe777e327" Oct 31 01:02:10.749031 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:10.749002 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-bxnmv" Oct 31 01:02:10.836418 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:10.836355 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9937a9f-f743-4587-9a27-88a51f33b245-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9937a9f-f743-4587-9a27-88a51f33b245-maas-api-xks-ca\") pod \"b9937a9f-f743-4587-9a27-88a51f33b245\" (UID: \"b9937a9f-f743-4587-9a27-88a51f33b245\") " Oct 31 01:02:10.836632 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:10.836474 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9937a9f-f743-4587-9a27-88a51f33b245-kube-api-access-m8nt6\" (UniqueName: \"kubernetes.io/projected/b9937a9f-f743-4587-9a27-88a51f33b245-kube-api-access-m8nt6\") pod \"b9937a9f-f743-4587-9a27-88a51f33b245\" (UID: \"b9937a9f-f743-4587-9a27-88a51f33b245\") " Oct 31 01:02:10.836632 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:10.836556 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9937a9f-f743-4587-9a27-88a51f33b245-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9937a9f-f743-4587-9a27-88a51f33b245-maas-api-openshift-ca\") pod \"b9937a9f-f743-4587-9a27-88a51f33b245\" (UID: \"b9937a9f-f743-4587-9a27-88a51f33b245\") " Oct 31 01:02:10.836916 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:10.836889 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9937a9f-f743-4587-9a27-88a51f33b245-maas-api-openshift-ca" pod "b9937a9f-f743-4587-9a27-88a51f33b245" (UID: "b9937a9f-f743-4587-9a27-88a51f33b245"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:02:10.838887 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:10.838861 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9937a9f-f743-4587-9a27-88a51f33b245-kube-api-access-m8nt6" pod "b9937a9f-f743-4587-9a27-88a51f33b245" (UID: "b9937a9f-f743-4587-9a27-88a51f33b245"). InnerVolumeSpecName "kube-api-access-m8nt6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:02:10.838987 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:10.838931 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9937a9f-f743-4587-9a27-88a51f33b245-maas-api-xks-ca" pod "b9937a9f-f743-4587-9a27-88a51f33b245" (UID: "b9937a9f-f743-4587-9a27-88a51f33b245"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:02:10.938509 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:10.938423 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8nt6\" (UniqueName: \"kubernetes.io/projected/b9937a9f-f743-4587-9a27-88a51f33b245-kube-api-access-m8nt6\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:02:10.938509 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:10.938457 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9937a9f-f743-4587-9a27-88a51f33b245-maas-api-openshift-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:02:10.938509 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:10.938469 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9937a9f-f743-4587-9a27-88a51f33b245-maas-api-xks-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:02:10.948304 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:10.948273 5222 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-256br container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.248:15021/healthz/ready\": dial tcp 10.134.0.248:15021: connect: connection refused" start-of-body= Oct 31 01:02:10.948498 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:10.948324 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-256br" podUID="a723d3c6-4cd4-4e23-bf6b-6f2dfb4a38f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.248:15021/healthz/ready\": dial tcp 10.134.0.248:15021: connect: connection refused" Oct 31 01:02:11.614331 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:11.614305 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-bxnmv" Oct 31 01:02:11.614823 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:11.614325 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-bxnmv" event={"ID":"b9937a9f-f743-4587-9a27-88a51f33b245","Type":"ContainerDied","Data":"f807ebe5faae7ec3350217c8ababc47069605bf057479bd295875a713e5f4d74"} Oct 31 01:02:11.614823 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:11.614412 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f807ebe5faae7ec3350217c8ababc47069605bf057479bd295875a713e5f4d74" Oct 31 01:02:11.952111 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:11.952025 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-256br" Oct 31 01:02:11.952335 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:11.952312 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-256br" Oct 31 01:02:11.953193 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:11.953176 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-256br" Oct 31 01:02:12.120359 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:12.120329 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-79tk8"] Oct 31 01:02:17.747456 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:17.747423 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jd2w5"] Oct 31 01:02:17.748010 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:17.747986 5222 cpu_manager.go:414] "containerMap: removing container" podUID="b9937a9f-f743-4587-9a27-88a51f33b245" containerName="revoke-keys" Oct 31 01:02:17.748092 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:17.748014 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9937a9f-f743-4587-9a27-88a51f33b245" containerName="revoke-keys" Oct 31 01:02:17.748159 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:17.748117 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9937a9f-f743-4587-9a27-88a51f33b245" containerName="revoke-keys" Oct 31 01:02:17.751525 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:17.751500 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jd2w5" Oct 31 01:02:17.858225 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:17.858193 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jd2w5"] Oct 31 01:02:17.921497 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:17.921457 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2ec2aa7-9ec8-41bd-b10f-e5ef7ecbd803-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-jd2w5\" (UID: \"d2ec2aa7-9ec8-41bd-b10f-e5ef7ecbd803\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jd2w5" Oct 31 01:02:17.921689 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:17.921507 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2ec2aa7-9ec8-41bd-b10f-e5ef7ecbd803-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-jd2w5\" (UID: \"d2ec2aa7-9ec8-41bd-b10f-e5ef7ecbd803\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jd2w5" Oct 31 01:02:17.921689 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:17.921535 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2ec2aa7-9ec8-41bd-b10f-e5ef7ecbd803-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-jd2w5\" (UID: \"d2ec2aa7-9ec8-41bd-b10f-e5ef7ecbd803\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jd2w5" Oct 31 01:02:17.921689 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:17.921621 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2ec2aa7-9ec8-41bd-b10f-e5ef7ecbd803-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-jd2w5\" (UID: \"d2ec2aa7-9ec8-41bd-b10f-e5ef7ecbd803\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jd2w5" Oct 31 01:02:17.921689 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:17.921664 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2ec2aa7-9ec8-41bd-b10f-e5ef7ecbd803-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-jd2w5\" (UID: \"d2ec2aa7-9ec8-41bd-b10f-e5ef7ecbd803\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jd2w5" Oct 31 01:02:17.921833 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:17.921712 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2ec2aa7-9ec8-41bd-b10f-e5ef7ecbd803-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-jd2w5\" (UID: \"d2ec2aa7-9ec8-41bd-b10f-e5ef7ecbd803\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jd2w5" Oct 31 01:02:17.921833 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:17.921812 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2ec2aa7-9ec8-41bd-b10f-e5ef7ecbd803-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-jd2w5\" (UID: \"d2ec2aa7-9ec8-41bd-b10f-e5ef7ecbd803\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jd2w5" Oct 31 01:02:17.921900 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:17.921856 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2ec2aa7-9ec8-41bd-b10f-e5ef7ecbd803-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-jd2w5\" (UID: \"d2ec2aa7-9ec8-41bd-b10f-e5ef7ecbd803\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jd2w5" Oct 31 01:02:17.921900 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:17.921875 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5h29f\" (UniqueName: \"kubernetes.io/projected/d2ec2aa7-9ec8-41bd-b10f-e5ef7ecbd803-kube-api-access-5h29f\") pod \"maas-default-gateway-openshift-default-6886b96f9-jd2w5\" (UID: \"d2ec2aa7-9ec8-41bd-b10f-e5ef7ecbd803\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jd2w5" Oct 31 01:02:18.022579 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:18.022534 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2ec2aa7-9ec8-41bd-b10f-e5ef7ecbd803-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-jd2w5\" (UID: \"d2ec2aa7-9ec8-41bd-b10f-e5ef7ecbd803\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jd2w5" Oct 31 01:02:18.022790 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:18.022620 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2ec2aa7-9ec8-41bd-b10f-e5ef7ecbd803-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-jd2w5\" (UID: \"d2ec2aa7-9ec8-41bd-b10f-e5ef7ecbd803\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jd2w5" Oct 31 01:02:18.022790 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:18.022680 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2ec2aa7-9ec8-41bd-b10f-e5ef7ecbd803-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-jd2w5\" (UID: \"d2ec2aa7-9ec8-41bd-b10f-e5ef7ecbd803\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jd2w5" Oct 31 01:02:18.022790 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:18.022724 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2ec2aa7-9ec8-41bd-b10f-e5ef7ecbd803-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-jd2w5\" (UID: \"d2ec2aa7-9ec8-41bd-b10f-e5ef7ecbd803\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jd2w5" Oct 31 01:02:18.022790 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:18.022751 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5h29f\" (UniqueName: \"kubernetes.io/projected/d2ec2aa7-9ec8-41bd-b10f-e5ef7ecbd803-kube-api-access-5h29f\") pod \"maas-default-gateway-openshift-default-6886b96f9-jd2w5\" (UID: \"d2ec2aa7-9ec8-41bd-b10f-e5ef7ecbd803\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jd2w5" Oct 31 01:02:18.023023 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:18.022794 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2ec2aa7-9ec8-41bd-b10f-e5ef7ecbd803-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-jd2w5\" (UID: \"d2ec2aa7-9ec8-41bd-b10f-e5ef7ecbd803\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jd2w5" Oct 31 01:02:18.023023 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:18.022838 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2ec2aa7-9ec8-41bd-b10f-e5ef7ecbd803-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-jd2w5\" (UID: \"d2ec2aa7-9ec8-41bd-b10f-e5ef7ecbd803\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jd2w5" Oct 31 01:02:18.023023 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:18.022873 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2ec2aa7-9ec8-41bd-b10f-e5ef7ecbd803-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-jd2w5\" (UID: \"d2ec2aa7-9ec8-41bd-b10f-e5ef7ecbd803\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jd2w5" Oct 31 01:02:18.023023 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:18.022909 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2ec2aa7-9ec8-41bd-b10f-e5ef7ecbd803-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-jd2w5\" (UID: \"d2ec2aa7-9ec8-41bd-b10f-e5ef7ecbd803\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jd2w5" Oct 31 01:02:18.023023 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:18.022992 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2ec2aa7-9ec8-41bd-b10f-e5ef7ecbd803-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-jd2w5\" (UID: \"d2ec2aa7-9ec8-41bd-b10f-e5ef7ecbd803\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jd2w5" Oct 31 01:02:18.023264 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:18.023090 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2ec2aa7-9ec8-41bd-b10f-e5ef7ecbd803-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-jd2w5\" (UID: \"d2ec2aa7-9ec8-41bd-b10f-e5ef7ecbd803\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jd2w5" Oct 31 01:02:18.023305 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:18.023271 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2ec2aa7-9ec8-41bd-b10f-e5ef7ecbd803-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-jd2w5\" (UID: \"d2ec2aa7-9ec8-41bd-b10f-e5ef7ecbd803\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jd2w5" Oct 31 01:02:18.023341 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:18.023298 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2ec2aa7-9ec8-41bd-b10f-e5ef7ecbd803-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-jd2w5\" (UID: \"d2ec2aa7-9ec8-41bd-b10f-e5ef7ecbd803\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jd2w5" Oct 31 01:02:18.023474 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:18.023455 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2ec2aa7-9ec8-41bd-b10f-e5ef7ecbd803-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-jd2w5\" (UID: \"d2ec2aa7-9ec8-41bd-b10f-e5ef7ecbd803\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jd2w5" Oct 31 01:02:18.025481 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:18.025452 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2ec2aa7-9ec8-41bd-b10f-e5ef7ecbd803-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-jd2w5\" (UID: \"d2ec2aa7-9ec8-41bd-b10f-e5ef7ecbd803\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jd2w5" Oct 31 01:02:18.025663 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:18.025645 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2ec2aa7-9ec8-41bd-b10f-e5ef7ecbd803-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-jd2w5\" (UID: \"d2ec2aa7-9ec8-41bd-b10f-e5ef7ecbd803\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jd2w5" Oct 31 01:02:18.085626 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:18.085587 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2ec2aa7-9ec8-41bd-b10f-e5ef7ecbd803-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-jd2w5\" (UID: \"d2ec2aa7-9ec8-41bd-b10f-e5ef7ecbd803\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jd2w5" Oct 31 01:02:18.085814 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:18.085790 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5h29f\" (UniqueName: \"kubernetes.io/projected/d2ec2aa7-9ec8-41bd-b10f-e5ef7ecbd803-kube-api-access-5h29f\") pod \"maas-default-gateway-openshift-default-6886b96f9-jd2w5\" (UID: \"d2ec2aa7-9ec8-41bd-b10f-e5ef7ecbd803\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jd2w5" Oct 31 01:02:18.365103 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:18.365017 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jd2w5" Oct 31 01:02:18.789027 ip-10-0-137-56 kubenswrapper[5222]: W1031 01:02:18.788990 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2ec2aa7_9ec8_41bd_b10f_e5ef7ecbd803.slice/crio-f682f1464c69978b9bdad244b2272cb118c1fcd2a7f28612093e123ea50d94e4 WatchSource:0}: Error finding container f682f1464c69978b9bdad244b2272cb118c1fcd2a7f28612093e123ea50d94e4: Status 404 returned error can't find the container with id f682f1464c69978b9bdad244b2272cb118c1fcd2a7f28612093e123ea50d94e4 Oct 31 01:02:18.789908 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:18.789882 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jd2w5"] Oct 31 01:02:18.791588 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:18.791551 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 01:02:18.791665 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:18.791622 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 01:02:18.791665 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:18.791652 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 01:02:19.668345 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:19.668307 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jd2w5" event={"ID":"d2ec2aa7-9ec8-41bd-b10f-e5ef7ecbd803","Type":"ContainerStarted","Data":"000d7e437329daf88640d36171366980d958e013219d33f9ff62b6e6dbe672db"} Oct 31 01:02:19.668557 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:19.668384 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jd2w5" event={"ID":"d2ec2aa7-9ec8-41bd-b10f-e5ef7ecbd803","Type":"ContainerStarted","Data":"f682f1464c69978b9bdad244b2272cb118c1fcd2a7f28612093e123ea50d94e4"} Oct 31 01:02:20.365437 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:20.365395 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jd2w5" Oct 31 01:02:20.367055 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:20.367023 5222 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-jd2w5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" start-of-body= Oct 31 01:02:20.367214 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:20.367080 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jd2w5" podUID="d2ec2aa7-9ec8-41bd-b10f-e5ef7ecbd803" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" Oct 31 01:02:21.365902 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:21.365860 5222 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-jd2w5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" start-of-body= Oct 31 01:02:21.366326 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:21.365926 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jd2w5" podUID="d2ec2aa7-9ec8-41bd-b10f-e5ef7ecbd803" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" Oct 31 01:02:22.366266 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:22.366227 5222 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-jd2w5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" start-of-body= Oct 31 01:02:22.366686 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:22.366290 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jd2w5" podUID="d2ec2aa7-9ec8-41bd-b10f-e5ef7ecbd803" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" Oct 31 01:02:23.366130 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:23.366085 5222 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-jd2w5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" start-of-body= Oct 31 01:02:23.366440 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:23.366147 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jd2w5" podUID="d2ec2aa7-9ec8-41bd-b10f-e5ef7ecbd803" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" Oct 31 01:02:24.365972 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:24.365931 5222 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-jd2w5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" start-of-body= Oct 31 01:02:24.366173 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:24.365989 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jd2w5" podUID="d2ec2aa7-9ec8-41bd-b10f-e5ef7ecbd803" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" Oct 31 01:02:25.365694 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:25.365652 5222 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-jd2w5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" start-of-body= Oct 31 01:02:25.366132 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:25.365738 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jd2w5" podUID="d2ec2aa7-9ec8-41bd-b10f-e5ef7ecbd803" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" Oct 31 01:02:27.055668 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:27.055628 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jd2w5" Oct 31 01:02:27.056833 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:27.056807 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jd2w5" Oct 31 01:02:27.057548 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:27.057528 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jd2w5" Oct 31 01:02:27.232286 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:02:27.232233 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jd2w5" podStartSLOduration=10.232220329 podStartE2EDuration="10.232220329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:02:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:02:19.853781223 +0000 UTC m=+2832.324950973" watchObservedRunningTime="2026-10-31 01:02:27.232220329 +0000 UTC m=+2839.703390115" Oct 31 01:02:35.889146 ip-10-0-137-56 kubenswrapper[5222]: E1031 01:02:35.889107 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 01:03:02.051279 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:02.051236 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f85ed32c-mz22p"] Oct 31 01:03:02.054217 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:02.054192 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f85ed32c-mz22p"] Oct 31 01:03:03.653947 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:03.653908 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24d0a0fc-07f1-4f6b-b7c8-8a89a97d5548" path="/var/lib/kubelet/pods/24d0a0fc-07f1-4f6b-b7c8-8a89a97d5548/volumes" Oct 31 01:03:05.888857 ip-10-0-137-56 kubenswrapper[5222]: E1031 01:03:05.888814 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 01:03:07.100756 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:07.100721 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-79tk8"] Oct 31 01:03:10.478330 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:10.478291 5222 scope.go:122] "RemoveContainer" containerID="a074f5e3d48b758450af11203cb545f3bdd3f57eeb91ac391c15227b4b9e5a8b" Oct 31 01:03:10.487338 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:10.487310 5222 scope.go:122] "RemoveContainer" containerID="5661ec1588b0c4cf6c3eedd761e3785334ad0921cc7db38a538235469d1d2432" Oct 31 01:03:15.654692 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:15.654619 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-drj9f"] Oct 31 01:03:15.659674 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:15.659650 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-drj9f" Oct 31 01:03:15.709581 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:15.709490 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-drj9f"] Oct 31 01:03:15.744153 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:15.744106 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqvjl\" (UniqueName: \"kubernetes.io/projected/20110920-0e40-41b2-9d39-fef24408d6c1-kube-api-access-vqvjl\") pod \"maas-api-revoke-sub-c1e271a9-drj9f\" (UID: \"20110920-0e40-41b2-9d39-fef24408d6c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-drj9f" Oct 31 01:03:15.744331 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:15.744267 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20110920-0e40-41b2-9d39-fef24408d6c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-drj9f\" (UID: \"20110920-0e40-41b2-9d39-fef24408d6c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-drj9f" Oct 31 01:03:15.744395 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:15.744343 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20110920-0e40-41b2-9d39-fef24408d6c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-drj9f\" (UID: \"20110920-0e40-41b2-9d39-fef24408d6c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-drj9f" Oct 31 01:03:15.758676 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:15.758636 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:03:15.758862 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:15.758740 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfz9z\"" Oct 31 01:03:15.764596 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:15.764567 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:03:15.764726 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:15.764599 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:03:15.845160 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:15.845123 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20110920-0e40-41b2-9d39-fef24408d6c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-drj9f\" (UID: \"20110920-0e40-41b2-9d39-fef24408d6c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-drj9f" Oct 31 01:03:15.845331 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:15.845189 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20110920-0e40-41b2-9d39-fef24408d6c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-drj9f\" (UID: \"20110920-0e40-41b2-9d39-fef24408d6c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-drj9f" Oct 31 01:03:15.845331 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:15.845251 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqvjl\" (UniqueName: \"kubernetes.io/projected/20110920-0e40-41b2-9d39-fef24408d6c1-kube-api-access-vqvjl\") pod \"maas-api-revoke-sub-c1e271a9-drj9f\" (UID: \"20110920-0e40-41b2-9d39-fef24408d6c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-drj9f" Oct 31 01:03:15.846007 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:15.845932 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20110920-0e40-41b2-9d39-fef24408d6c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-drj9f\" (UID: \"20110920-0e40-41b2-9d39-fef24408d6c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-drj9f" Oct 31 01:03:15.847760 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:15.847735 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20110920-0e40-41b2-9d39-fef24408d6c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-drj9f\" (UID: \"20110920-0e40-41b2-9d39-fef24408d6c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-drj9f" Oct 31 01:03:15.908270 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:15.908189 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqvjl\" (UniqueName: \"kubernetes.io/projected/20110920-0e40-41b2-9d39-fef24408d6c1-kube-api-access-vqvjl\") pod \"maas-api-revoke-sub-c1e271a9-drj9f\" (UID: \"20110920-0e40-41b2-9d39-fef24408d6c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-drj9f" Oct 31 01:03:15.973063 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:15.973025 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-drj9f" Oct 31 01:03:16.059826 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:16.059794 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6764d1c-ljhrm"] Oct 31 01:03:16.063092 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:16.063068 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6764d1c-ljhrm"] Oct 31 01:03:16.348360 ip-10-0-137-56 kubenswrapper[5222]: W1031 01:03:16.348328 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20110920_0e40_41b2_9d39_fef24408d6c1.slice/crio-d35b909ed46ef60663d21a8a5699d9441505280267da15c180339f9abd255180 WatchSource:0}: Error finding container d35b909ed46ef60663d21a8a5699d9441505280267da15c180339f9abd255180: Status 404 returned error can't find the container with id d35b909ed46ef60663d21a8a5699d9441505280267da15c180339f9abd255180 Oct 31 01:03:16.351002 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:16.350868 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-drj9f"] Oct 31 01:03:16.955427 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:16.955304 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-drj9f" event={"ID":"20110920-0e40-41b2-9d39-fef24408d6c1","Type":"ContainerStarted","Data":"5d2ec0ce2502501329875b65c9f7f40102415da981f2346d04176b1e8915ebae"} Oct 31 01:03:16.955427 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:16.955395 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-drj9f" event={"ID":"20110920-0e40-41b2-9d39-fef24408d6c1","Type":"ContainerStarted","Data":"d35b909ed46ef60663d21a8a5699d9441505280267da15c180339f9abd255180"} Oct 31 01:03:17.074688 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:17.074632 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-drj9f" podStartSLOduration=2.07461601 podStartE2EDuration="2.07461601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:03:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:03:17.073000198 +0000 UTC m=+2889.544170002" watchObservedRunningTime="2026-10-31 01:03:17.07461601 +0000 UTC m=+2889.545785756" Oct 31 01:03:17.654591 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:17.654560 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07ace044-4dbc-43cd-a259-81de418b36ab" path="/var/lib/kubelet/pods/07ace044-4dbc-43cd-a259-81de418b36ab/volumes" Oct 31 01:03:17.962356 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:17.962256 5222 generic.go:356] "Generic (PLEG): container finished" podID="20110920-0e40-41b2-9d39-fef24408d6c1" containerID="5d2ec0ce2502501329875b65c9f7f40102415da981f2346d04176b1e8915ebae" exitCode=0 Oct 31 01:03:17.962356 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:17.962335 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-drj9f" event={"ID":"20110920-0e40-41b2-9d39-fef24408d6c1","Type":"ContainerDied","Data":"5d2ec0ce2502501329875b65c9f7f40102415da981f2346d04176b1e8915ebae"} Oct 31 01:03:19.104975 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:19.104945 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-drj9f" Oct 31 01:03:19.180549 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:19.180515 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20110920-0e40-41b2-9d39-fef24408d6c1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20110920-0e40-41b2-9d39-fef24408d6c1-maas-api-openshift-ca\") pod \"20110920-0e40-41b2-9d39-fef24408d6c1\" (UID: \"20110920-0e40-41b2-9d39-fef24408d6c1\") " Oct 31 01:03:19.180744 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:19.180599 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20110920-0e40-41b2-9d39-fef24408d6c1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20110920-0e40-41b2-9d39-fef24408d6c1-maas-api-xks-ca\") pod \"20110920-0e40-41b2-9d39-fef24408d6c1\" (UID: \"20110920-0e40-41b2-9d39-fef24408d6c1\") " Oct 31 01:03:19.180744 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:19.180712 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20110920-0e40-41b2-9d39-fef24408d6c1-kube-api-access-vqvjl\" (UniqueName: \"kubernetes.io/projected/20110920-0e40-41b2-9d39-fef24408d6c1-kube-api-access-vqvjl\") pod \"20110920-0e40-41b2-9d39-fef24408d6c1\" (UID: \"20110920-0e40-41b2-9d39-fef24408d6c1\") " Oct 31 01:03:19.180937 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:19.180902 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20110920-0e40-41b2-9d39-fef24408d6c1-maas-api-openshift-ca" pod "20110920-0e40-41b2-9d39-fef24408d6c1" (UID: "20110920-0e40-41b2-9d39-fef24408d6c1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:03:19.181076 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:19.181027 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20110920-0e40-41b2-9d39-fef24408d6c1-maas-api-openshift-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:03:19.182917 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:19.182896 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20110920-0e40-41b2-9d39-fef24408d6c1-kube-api-access-vqvjl" pod "20110920-0e40-41b2-9d39-fef24408d6c1" (UID: "20110920-0e40-41b2-9d39-fef24408d6c1"). InnerVolumeSpecName "kube-api-access-vqvjl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:03:19.183002 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:19.182984 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20110920-0e40-41b2-9d39-fef24408d6c1-maas-api-xks-ca" pod "20110920-0e40-41b2-9d39-fef24408d6c1" (UID: "20110920-0e40-41b2-9d39-fef24408d6c1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:03:19.282502 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:19.282469 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqvjl\" (UniqueName: \"kubernetes.io/projected/20110920-0e40-41b2-9d39-fef24408d6c1-kube-api-access-vqvjl\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:03:19.282502 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:19.282498 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20110920-0e40-41b2-9d39-fef24408d6c1-maas-api-xks-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:03:19.974086 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:19.974000 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-drj9f" Oct 31 01:03:19.974086 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:19.974031 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-drj9f" event={"ID":"20110920-0e40-41b2-9d39-fef24408d6c1","Type":"ContainerDied","Data":"d35b909ed46ef60663d21a8a5699d9441505280267da15c180339f9abd255180"} Oct 31 01:03:19.974086 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:19.974076 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d35b909ed46ef60663d21a8a5699d9441505280267da15c180339f9abd255180" Oct 31 01:03:28.501736 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:28.501687 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-79tk8"] Oct 31 01:03:33.358411 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:33.358361 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zq2lm"] Oct 31 01:03:33.358962 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:33.358795 5222 cpu_manager.go:414] "containerMap: removing container" podUID="20110920-0e40-41b2-9d39-fef24408d6c1" containerName="revoke-keys" Oct 31 01:03:33.358962 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:33.358806 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20110920-0e40-41b2-9d39-fef24408d6c1" containerName="revoke-keys" Oct 31 01:03:33.358962 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:33.358872 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="20110920-0e40-41b2-9d39-fef24408d6c1" containerName="revoke-keys" Oct 31 01:03:33.362170 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:33.362148 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zq2lm" Oct 31 01:03:33.423274 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:33.423234 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d1be84f-73f0-4f54-ae9f-6d5f38367862-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-zq2lm\" (UID: \"3d1be84f-73f0-4f54-ae9f-6d5f38367862\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zq2lm" Oct 31 01:03:33.423483 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:33.423284 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d1be84f-73f0-4f54-ae9f-6d5f38367862-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-zq2lm\" (UID: \"3d1be84f-73f0-4f54-ae9f-6d5f38367862\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zq2lm" Oct 31 01:03:33.423483 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:33.423358 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d1be84f-73f0-4f54-ae9f-6d5f38367862-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-zq2lm\" (UID: \"3d1be84f-73f0-4f54-ae9f-6d5f38367862\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zq2lm" Oct 31 01:03:33.423483 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:33.423438 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d1be84f-73f0-4f54-ae9f-6d5f38367862-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-zq2lm\" (UID: \"3d1be84f-73f0-4f54-ae9f-6d5f38367862\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zq2lm" Oct 31 01:03:33.423598 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:33.423497 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d1be84f-73f0-4f54-ae9f-6d5f38367862-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-zq2lm\" (UID: \"3d1be84f-73f0-4f54-ae9f-6d5f38367862\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zq2lm" Oct 31 01:03:33.423598 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:33.423550 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d1be84f-73f0-4f54-ae9f-6d5f38367862-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-zq2lm\" (UID: \"3d1be84f-73f0-4f54-ae9f-6d5f38367862\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zq2lm" Oct 31 01:03:33.423598 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:33.423576 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d1be84f-73f0-4f54-ae9f-6d5f38367862-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-zq2lm\" (UID: \"3d1be84f-73f0-4f54-ae9f-6d5f38367862\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zq2lm" Oct 31 01:03:33.423700 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:33.423599 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42sjv\" (UniqueName: \"kubernetes.io/projected/3d1be84f-73f0-4f54-ae9f-6d5f38367862-kube-api-access-42sjv\") pod \"maas-default-gateway-openshift-default-6886b96f9-zq2lm\" (UID: \"3d1be84f-73f0-4f54-ae9f-6d5f38367862\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zq2lm" Oct 31 01:03:33.423700 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:33.423637 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d1be84f-73f0-4f54-ae9f-6d5f38367862-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-zq2lm\" (UID: \"3d1be84f-73f0-4f54-ae9f-6d5f38367862\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zq2lm" Oct 31 01:03:33.451696 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:33.451655 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zq2lm"] Oct 31 01:03:33.524404 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:33.524345 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d1be84f-73f0-4f54-ae9f-6d5f38367862-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-zq2lm\" (UID: \"3d1be84f-73f0-4f54-ae9f-6d5f38367862\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zq2lm" Oct 31 01:03:33.524602 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:33.524415 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d1be84f-73f0-4f54-ae9f-6d5f38367862-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-zq2lm\" (UID: \"3d1be84f-73f0-4f54-ae9f-6d5f38367862\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zq2lm" Oct 31 01:03:33.524602 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:33.524453 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d1be84f-73f0-4f54-ae9f-6d5f38367862-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-zq2lm\" (UID: \"3d1be84f-73f0-4f54-ae9f-6d5f38367862\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zq2lm" Oct 31 01:03:33.524602 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:33.524475 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d1be84f-73f0-4f54-ae9f-6d5f38367862-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-zq2lm\" (UID: \"3d1be84f-73f0-4f54-ae9f-6d5f38367862\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zq2lm" Oct 31 01:03:33.524602 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:33.524509 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d1be84f-73f0-4f54-ae9f-6d5f38367862-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-zq2lm\" (UID: \"3d1be84f-73f0-4f54-ae9f-6d5f38367862\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zq2lm" Oct 31 01:03:33.524602 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:33.524543 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d1be84f-73f0-4f54-ae9f-6d5f38367862-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-zq2lm\" (UID: \"3d1be84f-73f0-4f54-ae9f-6d5f38367862\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zq2lm" Oct 31 01:03:33.524602 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:33.524577 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42sjv\" (UniqueName: \"kubernetes.io/projected/3d1be84f-73f0-4f54-ae9f-6d5f38367862-kube-api-access-42sjv\") pod \"maas-default-gateway-openshift-default-6886b96f9-zq2lm\" (UID: \"3d1be84f-73f0-4f54-ae9f-6d5f38367862\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zq2lm" Oct 31 01:03:33.524952 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:33.524609 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d1be84f-73f0-4f54-ae9f-6d5f38367862-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-zq2lm\" (UID: \"3d1be84f-73f0-4f54-ae9f-6d5f38367862\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zq2lm" Oct 31 01:03:33.524952 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:33.524709 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d1be84f-73f0-4f54-ae9f-6d5f38367862-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-zq2lm\" (UID: \"3d1be84f-73f0-4f54-ae9f-6d5f38367862\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zq2lm" Oct 31 01:03:33.525080 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:33.524976 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d1be84f-73f0-4f54-ae9f-6d5f38367862-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-zq2lm\" (UID: \"3d1be84f-73f0-4f54-ae9f-6d5f38367862\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zq2lm" Oct 31 01:03:33.525146 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:33.525116 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d1be84f-73f0-4f54-ae9f-6d5f38367862-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-zq2lm\" (UID: \"3d1be84f-73f0-4f54-ae9f-6d5f38367862\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zq2lm" Oct 31 01:03:33.525211 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:33.525184 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d1be84f-73f0-4f54-ae9f-6d5f38367862-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-zq2lm\" (UID: \"3d1be84f-73f0-4f54-ae9f-6d5f38367862\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zq2lm" Oct 31 01:03:33.525285 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:33.525260 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d1be84f-73f0-4f54-ae9f-6d5f38367862-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-zq2lm\" (UID: \"3d1be84f-73f0-4f54-ae9f-6d5f38367862\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zq2lm" Oct 31 01:03:33.525339 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:33.525277 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d1be84f-73f0-4f54-ae9f-6d5f38367862-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-zq2lm\" (UID: \"3d1be84f-73f0-4f54-ae9f-6d5f38367862\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zq2lm" Oct 31 01:03:33.527060 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:33.527037 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d1be84f-73f0-4f54-ae9f-6d5f38367862-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-zq2lm\" (UID: \"3d1be84f-73f0-4f54-ae9f-6d5f38367862\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zq2lm" Oct 31 01:03:33.527306 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:33.527288 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d1be84f-73f0-4f54-ae9f-6d5f38367862-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-zq2lm\" (UID: \"3d1be84f-73f0-4f54-ae9f-6d5f38367862\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zq2lm" Oct 31 01:03:33.587077 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:33.587048 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d1be84f-73f0-4f54-ae9f-6d5f38367862-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-zq2lm\" (UID: \"3d1be84f-73f0-4f54-ae9f-6d5f38367862\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zq2lm" Oct 31 01:03:33.587333 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:33.587307 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42sjv\" (UniqueName: \"kubernetes.io/projected/3d1be84f-73f0-4f54-ae9f-6d5f38367862-kube-api-access-42sjv\") pod \"maas-default-gateway-openshift-default-6886b96f9-zq2lm\" (UID: \"3d1be84f-73f0-4f54-ae9f-6d5f38367862\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zq2lm" Oct 31 01:03:33.674589 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:33.674495 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zq2lm" Oct 31 01:03:34.100506 ip-10-0-137-56 kubenswrapper[5222]: W1031 01:03:34.100475 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d1be84f_73f0_4f54_ae9f_6d5f38367862.slice/crio-68929f44bca96cccd9db72b800b5d14fda78100d0823d7a042e59d93fb74f1a4 WatchSource:0}: Error finding container 68929f44bca96cccd9db72b800b5d14fda78100d0823d7a042e59d93fb74f1a4: Status 404 returned error can't find the container with id 68929f44bca96cccd9db72b800b5d14fda78100d0823d7a042e59d93fb74f1a4 Oct 31 01:03:34.102290 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:34.102263 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zq2lm"] Oct 31 01:03:34.102706 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:34.102669 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 01:03:34.102765 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:34.102753 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 01:03:34.102806 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:34.102785 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 01:03:35.069691 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:35.069650 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zq2lm" event={"ID":"3d1be84f-73f0-4f54-ae9f-6d5f38367862","Type":"ContainerStarted","Data":"fa0d66fbeaa0cbb920238b2e6fd5f74fe62aade4f1bb4fb0c031e8f6cd070261"} Oct 31 01:03:35.070197 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:35.069711 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zq2lm" event={"ID":"3d1be84f-73f0-4f54-ae9f-6d5f38367862","Type":"ContainerStarted","Data":"68929f44bca96cccd9db72b800b5d14fda78100d0823d7a042e59d93fb74f1a4"} Oct 31 01:03:35.304646 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:35.304586 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zq2lm" podStartSLOduration=2.304569825 podStartE2EDuration="2.304569825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:03:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:03:35.253184892 +0000 UTC m=+2907.724354646" watchObservedRunningTime="2026-10-31 01:03:35.304569825 +0000 UTC m=+2907.775739577" Oct 31 01:03:35.677689 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:35.677658 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zq2lm" Oct 31 01:03:35.678044 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:35.678020 5222 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-zq2lm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.252:15021/healthz/ready\": dial tcp 10.134.0.252:15021: connect: connection refused" start-of-body= Oct 31 01:03:35.678219 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:35.678194 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zq2lm" podUID="3d1be84f-73f0-4f54-ae9f-6d5f38367862" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.252:15021/healthz/ready\": dial tcp 10.134.0.252:15021: connect: connection refused" Oct 31 01:03:35.902316 ip-10-0-137-56 kubenswrapper[5222]: E1031 01:03:35.902273 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 01:03:36.689412 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:36.686818 5222 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-zq2lm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.252:15021/healthz/ready\": dial tcp 10.134.0.252:15021: connect: connection refused" start-of-body= Oct 31 01:03:36.689412 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:36.686885 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zq2lm" podUID="3d1be84f-73f0-4f54-ae9f-6d5f38367862" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.252:15021/healthz/ready\": dial tcp 10.134.0.252:15021: connect: connection refused" Oct 31 01:03:37.684394 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:37.683957 5222 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-zq2lm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.252:15021/healthz/ready\": dial tcp 10.134.0.252:15021: connect: connection refused" start-of-body= Oct 31 01:03:37.684394 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:37.684046 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zq2lm" podUID="3d1be84f-73f0-4f54-ae9f-6d5f38367862" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.252:15021/healthz/ready\": dial tcp 10.134.0.252:15021: connect: connection refused" Oct 31 01:03:38.675010 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:38.674970 5222 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-zq2lm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.252:15021/healthz/ready\": dial tcp 10.134.0.252:15021: connect: connection refused" start-of-body= Oct 31 01:03:38.675487 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:38.675027 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zq2lm" podUID="3d1be84f-73f0-4f54-ae9f-6d5f38367862" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.252:15021/healthz/ready\": dial tcp 10.134.0.252:15021: connect: connection refused" Oct 31 01:03:39.674882 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:39.674849 5222 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-zq2lm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.252:15021/healthz/ready\": dial tcp 10.134.0.252:15021: connect: connection refused" start-of-body= Oct 31 01:03:39.675131 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:39.674910 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zq2lm" podUID="3d1be84f-73f0-4f54-ae9f-6d5f38367862" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.252:15021/healthz/ready\": dial tcp 10.134.0.252:15021: connect: connection refused" Oct 31 01:03:40.675790 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:40.675750 5222 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-zq2lm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.252:15021/healthz/ready\": dial tcp 10.134.0.252:15021: connect: connection refused" start-of-body= Oct 31 01:03:40.676211 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:40.675814 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zq2lm" podUID="3d1be84f-73f0-4f54-ae9f-6d5f38367862" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.252:15021/healthz/ready\": dial tcp 10.134.0.252:15021: connect: connection refused" Oct 31 01:03:40.932673 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:40.932590 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-79tk8"] Oct 31 01:03:41.675680 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:41.675642 5222 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-zq2lm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.252:15021/healthz/ready\": dial tcp 10.134.0.252:15021: connect: connection refused" start-of-body= Oct 31 01:03:41.675868 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:41.675702 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zq2lm" podUID="3d1be84f-73f0-4f54-ae9f-6d5f38367862" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.252:15021/healthz/ready\": dial tcp 10.134.0.252:15021: connect: connection refused" Oct 31 01:03:43.716586 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:43.716542 5222 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-zq2lm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.252:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 01:03:43.717118 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:43.716620 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zq2lm" podUID="3d1be84f-73f0-4f54-ae9f-6d5f38367862" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.252:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 01:03:44.503083 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:44.503029 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zq2lm" Oct 31 01:03:44.503624 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:44.503604 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zq2lm" Oct 31 01:03:44.504254 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:44.504236 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zq2lm" Oct 31 01:03:47.360760 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:47.360727 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9bft5"] Oct 31 01:03:47.364534 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:47.364514 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9bft5" Oct 31 01:03:47.418216 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:47.418185 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9bft5"] Oct 31 01:03:47.448843 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:47.448808 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/800ff4e9-fe1c-403d-97b8-787dd370a036-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-9bft5\" (UID: \"800ff4e9-fe1c-403d-97b8-787dd370a036\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9bft5" Oct 31 01:03:47.449059 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:47.448850 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/800ff4e9-fe1c-403d-97b8-787dd370a036-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-9bft5\" (UID: \"800ff4e9-fe1c-403d-97b8-787dd370a036\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9bft5" Oct 31 01:03:47.449059 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:47.448992 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mrkw\" (UniqueName: \"kubernetes.io/projected/800ff4e9-fe1c-403d-97b8-787dd370a036-kube-api-access-6mrkw\") pod \"maas-api-revoke-sub-2e1f5b5f-9bft5\" (UID: \"800ff4e9-fe1c-403d-97b8-787dd370a036\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9bft5" Oct 31 01:03:47.467511 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:47.467475 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:03:47.467690 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:47.467593 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfz9z\"" Oct 31 01:03:47.467690 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:47.467593 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:03:47.467690 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:47.467661 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:03:47.549992 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:47.549956 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/800ff4e9-fe1c-403d-97b8-787dd370a036-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-9bft5\" (UID: \"800ff4e9-fe1c-403d-97b8-787dd370a036\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9bft5" Oct 31 01:03:47.549992 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:47.549997 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/800ff4e9-fe1c-403d-97b8-787dd370a036-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-9bft5\" (UID: \"800ff4e9-fe1c-403d-97b8-787dd370a036\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9bft5" Oct 31 01:03:47.550272 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:47.550068 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mrkw\" (UniqueName: \"kubernetes.io/projected/800ff4e9-fe1c-403d-97b8-787dd370a036-kube-api-access-6mrkw\") pod \"maas-api-revoke-sub-2e1f5b5f-9bft5\" (UID: \"800ff4e9-fe1c-403d-97b8-787dd370a036\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9bft5" Oct 31 01:03:47.550741 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:47.550716 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/800ff4e9-fe1c-403d-97b8-787dd370a036-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-9bft5\" (UID: \"800ff4e9-fe1c-403d-97b8-787dd370a036\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9bft5" Oct 31 01:03:47.552652 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:47.552626 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/800ff4e9-fe1c-403d-97b8-787dd370a036-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-9bft5\" (UID: \"800ff4e9-fe1c-403d-97b8-787dd370a036\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9bft5" Oct 31 01:03:47.612787 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:47.612706 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mrkw\" (UniqueName: \"kubernetes.io/projected/800ff4e9-fe1c-403d-97b8-787dd370a036-kube-api-access-6mrkw\") pod \"maas-api-revoke-sub-2e1f5b5f-9bft5\" (UID: \"800ff4e9-fe1c-403d-97b8-787dd370a036\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9bft5" Oct 31 01:03:47.680848 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:47.680807 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9bft5" Oct 31 01:03:48.052416 ip-10-0-137-56 kubenswrapper[5222]: W1031 01:03:48.052344 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod800ff4e9_fe1c_403d_97b8_787dd370a036.slice/crio-f6a0f320b84da65b2d6e1ba0f201789af1fd5179c9e6e1e59a605e8ad7280a81 WatchSource:0}: Error finding container f6a0f320b84da65b2d6e1ba0f201789af1fd5179c9e6e1e59a605e8ad7280a81: Status 404 returned error can't find the container with id f6a0f320b84da65b2d6e1ba0f201789af1fd5179c9e6e1e59a605e8ad7280a81 Oct 31 01:03:48.054269 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:48.054246 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9bft5"] Oct 31 01:03:48.152742 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:48.152701 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9bft5" event={"ID":"800ff4e9-fe1c-403d-97b8-787dd370a036","Type":"ContainerStarted","Data":"f6a0f320b84da65b2d6e1ba0f201789af1fd5179c9e6e1e59a605e8ad7280a81"} Oct 31 01:03:49.159106 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:49.159065 5222 generic.go:356] "Generic (PLEG): container finished" podID="800ff4e9-fe1c-403d-97b8-787dd370a036" containerID="0d93d2ddd041eddc0317ae881a545431ba5198f5b7e012b2b911bf12b723feb4" exitCode=0 Oct 31 01:03:49.159577 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:49.159107 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9bft5" event={"ID":"800ff4e9-fe1c-403d-97b8-787dd370a036","Type":"ContainerDied","Data":"0d93d2ddd041eddc0317ae881a545431ba5198f5b7e012b2b911bf12b723feb4"} Oct 31 01:03:50.302651 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:50.302620 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9bft5" Oct 31 01:03:50.376523 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:50.376488 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/800ff4e9-fe1c-403d-97b8-787dd370a036-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/800ff4e9-fe1c-403d-97b8-787dd370a036-maas-api-xks-ca\") pod \"800ff4e9-fe1c-403d-97b8-787dd370a036\" (UID: \"800ff4e9-fe1c-403d-97b8-787dd370a036\") " Oct 31 01:03:50.376671 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:50.376576 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/800ff4e9-fe1c-403d-97b8-787dd370a036-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/800ff4e9-fe1c-403d-97b8-787dd370a036-maas-api-openshift-ca\") pod \"800ff4e9-fe1c-403d-97b8-787dd370a036\" (UID: \"800ff4e9-fe1c-403d-97b8-787dd370a036\") " Oct 31 01:03:50.376671 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:50.376620 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/800ff4e9-fe1c-403d-97b8-787dd370a036-kube-api-access-6mrkw\" (UniqueName: \"kubernetes.io/projected/800ff4e9-fe1c-403d-97b8-787dd370a036-kube-api-access-6mrkw\") pod \"800ff4e9-fe1c-403d-97b8-787dd370a036\" (UID: \"800ff4e9-fe1c-403d-97b8-787dd370a036\") " Oct 31 01:03:50.377045 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:50.377010 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/800ff4e9-fe1c-403d-97b8-787dd370a036-maas-api-openshift-ca" pod "800ff4e9-fe1c-403d-97b8-787dd370a036" (UID: "800ff4e9-fe1c-403d-97b8-787dd370a036"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:03:50.378829 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:50.378794 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/800ff4e9-fe1c-403d-97b8-787dd370a036-kube-api-access-6mrkw" pod "800ff4e9-fe1c-403d-97b8-787dd370a036" (UID: "800ff4e9-fe1c-403d-97b8-787dd370a036"). InnerVolumeSpecName "kube-api-access-6mrkw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:03:50.378829 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:50.378820 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/800ff4e9-fe1c-403d-97b8-787dd370a036-maas-api-xks-ca" pod "800ff4e9-fe1c-403d-97b8-787dd370a036" (UID: "800ff4e9-fe1c-403d-97b8-787dd370a036"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:03:50.478083 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:50.477997 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/800ff4e9-fe1c-403d-97b8-787dd370a036-maas-api-openshift-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:03:50.478083 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:50.478029 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mrkw\" (UniqueName: \"kubernetes.io/projected/800ff4e9-fe1c-403d-97b8-787dd370a036-kube-api-access-6mrkw\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:03:50.478083 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:50.478041 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/800ff4e9-fe1c-403d-97b8-787dd370a036-maas-api-xks-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:03:51.171280 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:51.171240 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9bft5" event={"ID":"800ff4e9-fe1c-403d-97b8-787dd370a036","Type":"ContainerDied","Data":"f6a0f320b84da65b2d6e1ba0f201789af1fd5179c9e6e1e59a605e8ad7280a81"} Oct 31 01:03:51.171280 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:51.171275 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9bft5" Oct 31 01:03:51.171593 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:03:51.171295 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f6a0f320b84da65b2d6e1ba0f201789af1fd5179c9e6e1e59a605e8ad7280a81" Oct 31 01:04:00.041954 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:04:00.041919 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-7bxgn"] Oct 31 01:04:00.042394 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:04:00.042355 5222 cpu_manager.go:414] "containerMap: removing container" podUID="800ff4e9-fe1c-403d-97b8-787dd370a036" containerName="revoke-keys" Oct 31 01:04:00.042394 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:04:00.042380 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="800ff4e9-fe1c-403d-97b8-787dd370a036" containerName="revoke-keys" Oct 31 01:04:00.042480 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:04:00.042456 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="800ff4e9-fe1c-403d-97b8-787dd370a036" containerName="revoke-keys" Oct 31 01:04:00.045709 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:04:00.045682 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-7bxgn" Oct 31 01:04:00.099153 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:04:00.099103 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-7bxgn"] Oct 31 01:04:00.128245 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:04:00.128215 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:04:00.128245 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:04:00.128235 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:04:00.148309 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:04:00.148273 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:04:00.148514 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:04:00.148428 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfz9z\"" Oct 31 01:04:00.173889 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:04:00.173851 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7d6z\" (UniqueName: \"kubernetes.io/projected/884d98fd-e09a-480f-8195-07102988e30f-kube-api-access-c7d6z\") pod \"maas-api-revoke-sub-ec843018-7bxgn\" (UID: \"884d98fd-e09a-480f-8195-07102988e30f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-7bxgn" Oct 31 01:04:00.174065 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:04:00.173926 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/884d98fd-e09a-480f-8195-07102988e30f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-7bxgn\" (UID: \"884d98fd-e09a-480f-8195-07102988e30f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-7bxgn" Oct 31 01:04:00.174065 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:04:00.174016 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/884d98fd-e09a-480f-8195-07102988e30f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-7bxgn\" (UID: \"884d98fd-e09a-480f-8195-07102988e30f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-7bxgn" Oct 31 01:04:00.212010 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:04:00.211974 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-79tk8"] Oct 31 01:04:00.275224 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:04:00.275187 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/884d98fd-e09a-480f-8195-07102988e30f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-7bxgn\" (UID: \"884d98fd-e09a-480f-8195-07102988e30f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-7bxgn" Oct 31 01:04:00.275467 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:04:00.275252 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7d6z\" (UniqueName: \"kubernetes.io/projected/884d98fd-e09a-480f-8195-07102988e30f-kube-api-access-c7d6z\") pod \"maas-api-revoke-sub-ec843018-7bxgn\" (UID: \"884d98fd-e09a-480f-8195-07102988e30f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-7bxgn" Oct 31 01:04:00.275467 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:04:00.275340 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/884d98fd-e09a-480f-8195-07102988e30f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-7bxgn\" (UID: \"884d98fd-e09a-480f-8195-07102988e30f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-7bxgn" Oct 31 01:04:00.276002 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:04:00.275978 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/884d98fd-e09a-480f-8195-07102988e30f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-7bxgn\" (UID: \"884d98fd-e09a-480f-8195-07102988e30f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-7bxgn" Oct 31 01:04:00.277842 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:04:00.277824 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/884d98fd-e09a-480f-8195-07102988e30f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-7bxgn\" (UID: \"884d98fd-e09a-480f-8195-07102988e30f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-7bxgn" Oct 31 01:04:00.338385 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:04:00.338295 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7d6z\" (UniqueName: \"kubernetes.io/projected/884d98fd-e09a-480f-8195-07102988e30f-kube-api-access-c7d6z\") pod \"maas-api-revoke-sub-ec843018-7bxgn\" (UID: \"884d98fd-e09a-480f-8195-07102988e30f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-7bxgn" Oct 31 01:04:00.360045 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:04:00.360006 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-7bxgn" Oct 31 01:04:00.734314 ip-10-0-137-56 kubenswrapper[5222]: W1031 01:04:00.734282 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod884d98fd_e09a_480f_8195_07102988e30f.slice/crio-da4a89b3ab903cd8439309336e45def65b806e497c4dfbce12a9886b05559c4e WatchSource:0}: Error finding container da4a89b3ab903cd8439309336e45def65b806e497c4dfbce12a9886b05559c4e: Status 404 returned error can't find the container with id da4a89b3ab903cd8439309336e45def65b806e497c4dfbce12a9886b05559c4e Oct 31 01:04:00.737051 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:04:00.737026 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-7bxgn"] Oct 31 01:04:01.225480 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:04:01.225437 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-7bxgn" event={"ID":"884d98fd-e09a-480f-8195-07102988e30f","Type":"ContainerStarted","Data":"6444d642036e2752ac321f087d01a109a25775bc77b40b0711ee7ec61a3a97ac"} Oct 31 01:04:01.225930 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:04:01.225494 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-7bxgn" event={"ID":"884d98fd-e09a-480f-8195-07102988e30f","Type":"ContainerStarted","Data":"da4a89b3ab903cd8439309336e45def65b806e497c4dfbce12a9886b05559c4e"} Oct 31 01:04:01.350484 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:04:01.350340 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-7bxgn" podStartSLOduration=2.350322803 podStartE2EDuration="2.350322803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:03:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:04:01.349834149 +0000 UTC m=+2933.821003904" watchObservedRunningTime="2026-10-31 01:04:01.350322803 +0000 UTC m=+2933.821492552" Oct 31 01:04:02.231981 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:04:02.231948 5222 generic.go:356] "Generic (PLEG): container finished" podID="884d98fd-e09a-480f-8195-07102988e30f" containerID="6444d642036e2752ac321f087d01a109a25775bc77b40b0711ee7ec61a3a97ac" exitCode=0 Oct 31 01:04:02.231981 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:04:02.231984 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-7bxgn" event={"ID":"884d98fd-e09a-480f-8195-07102988e30f","Type":"ContainerDied","Data":"6444d642036e2752ac321f087d01a109a25775bc77b40b0711ee7ec61a3a97ac"} Oct 31 01:04:03.373398 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:04:03.373349 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-7bxgn" Oct 31 01:04:03.507400 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:04:03.507282 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/884d98fd-e09a-480f-8195-07102988e30f-kube-api-access-c7d6z\" (UniqueName: \"kubernetes.io/projected/884d98fd-e09a-480f-8195-07102988e30f-kube-api-access-c7d6z\") pod \"884d98fd-e09a-480f-8195-07102988e30f\" (UID: \"884d98fd-e09a-480f-8195-07102988e30f\") " Oct 31 01:04:03.507400 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:04:03.507326 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/884d98fd-e09a-480f-8195-07102988e30f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/884d98fd-e09a-480f-8195-07102988e30f-maas-api-openshift-ca\") pod \"884d98fd-e09a-480f-8195-07102988e30f\" (UID: \"884d98fd-e09a-480f-8195-07102988e30f\") " Oct 31 01:04:03.507672 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:04:03.507427 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/884d98fd-e09a-480f-8195-07102988e30f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/884d98fd-e09a-480f-8195-07102988e30f-maas-api-xks-ca\") pod \"884d98fd-e09a-480f-8195-07102988e30f\" (UID: \"884d98fd-e09a-480f-8195-07102988e30f\") " Oct 31 01:04:03.507789 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:04:03.507759 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/884d98fd-e09a-480f-8195-07102988e30f-maas-api-openshift-ca" pod "884d98fd-e09a-480f-8195-07102988e30f" (UID: "884d98fd-e09a-480f-8195-07102988e30f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:04:03.509724 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:04:03.509694 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/884d98fd-e09a-480f-8195-07102988e30f-maas-api-xks-ca" pod "884d98fd-e09a-480f-8195-07102988e30f" (UID: "884d98fd-e09a-480f-8195-07102988e30f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:04:03.509869 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:04:03.509811 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/884d98fd-e09a-480f-8195-07102988e30f-kube-api-access-c7d6z" pod "884d98fd-e09a-480f-8195-07102988e30f" (UID: "884d98fd-e09a-480f-8195-07102988e30f"). InnerVolumeSpecName "kube-api-access-c7d6z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:04:03.608745 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:04:03.608701 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7d6z\" (UniqueName: \"kubernetes.io/projected/884d98fd-e09a-480f-8195-07102988e30f-kube-api-access-c7d6z\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:04:03.608745 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:04:03.608742 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/884d98fd-e09a-480f-8195-07102988e30f-maas-api-openshift-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:04:03.608970 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:04:03.608757 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/884d98fd-e09a-480f-8195-07102988e30f-maas-api-xks-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:04:04.243860 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:04:04.243826 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-7bxgn" Oct 31 01:04:04.244070 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:04:04.243830 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-7bxgn" event={"ID":"884d98fd-e09a-480f-8195-07102988e30f","Type":"ContainerDied","Data":"da4a89b3ab903cd8439309336e45def65b806e497c4dfbce12a9886b05559c4e"} Oct 31 01:04:04.244070 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:04:04.243976 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="da4a89b3ab903cd8439309336e45def65b806e497c4dfbce12a9886b05559c4e" Oct 31 01:04:05.889033 ip-10-0-137-56 kubenswrapper[5222]: E1031 01:04:05.888996 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 01:04:10.515198 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:04:10.515157 5222 scope.go:122] "RemoveContainer" containerID="898a2269efdfc6a79c62282b99124defed8473fb32c16488971a9833fedafdcd" Oct 31 01:04:17.310965 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:04:17.310924 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-79tk8"] Oct 31 01:04:35.233889 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:04:35.233853 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-49nqr"] Oct 31 01:04:35.234311 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:04:35.234261 5222 cpu_manager.go:414] "containerMap: removing container" podUID="884d98fd-e09a-480f-8195-07102988e30f" containerName="revoke-keys" Oct 31 01:04:35.234311 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:04:35.234273 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="884d98fd-e09a-480f-8195-07102988e30f" containerName="revoke-keys" Oct 31 01:04:35.234403 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:04:35.234350 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="884d98fd-e09a-480f-8195-07102988e30f" containerName="revoke-keys" Oct 31 01:04:35.237221 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:04:35.237201 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-49nqr" Oct 31 01:04:35.276778 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:04:35.276738 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9scdt\" (UniqueName: \"kubernetes.io/projected/564df912-a1a2-45a2-b8a5-a3730da64a41-kube-api-access-9scdt\") pod \"maas-api-revoke-sub-046e1343-49nqr\" (UID: \"564df912-a1a2-45a2-b8a5-a3730da64a41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-49nqr" Oct 31 01:04:35.276974 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:04:35.276856 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/564df912-a1a2-45a2-b8a5-a3730da64a41-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-49nqr\" (UID: \"564df912-a1a2-45a2-b8a5-a3730da64a41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-49nqr" Oct 31 01:04:35.276974 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:04:35.276937 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/564df912-a1a2-45a2-b8a5-a3730da64a41-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-49nqr\" (UID: \"564df912-a1a2-45a2-b8a5-a3730da64a41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-49nqr" Oct 31 01:04:35.292307 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:04:35.292272 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-49nqr"] Oct 31 01:04:35.341229 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:04:35.341199 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:04:35.341229 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:04:35.341219 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:04:35.341229 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:04:35.341198 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:04:35.342228 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:04:35.342207 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfz9z\"" Oct 31 01:04:35.377805 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:04:35.377753 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/564df912-a1a2-45a2-b8a5-a3730da64a41-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-49nqr\" (UID: \"564df912-a1a2-45a2-b8a5-a3730da64a41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-49nqr" Oct 31 01:04:35.378010 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:04:35.377855 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9scdt\" (UniqueName: \"kubernetes.io/projected/564df912-a1a2-45a2-b8a5-a3730da64a41-kube-api-access-9scdt\") pod \"maas-api-revoke-sub-046e1343-49nqr\" (UID: \"564df912-a1a2-45a2-b8a5-a3730da64a41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-49nqr" Oct 31 01:04:35.378010 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:04:35.377919 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/564df912-a1a2-45a2-b8a5-a3730da64a41-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-49nqr\" (UID: \"564df912-a1a2-45a2-b8a5-a3730da64a41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-49nqr" Oct 31 01:04:35.378560 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:04:35.378538 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/564df912-a1a2-45a2-b8a5-a3730da64a41-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-49nqr\" (UID: \"564df912-a1a2-45a2-b8a5-a3730da64a41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-49nqr" Oct 31 01:04:35.380515 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:04:35.380497 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/564df912-a1a2-45a2-b8a5-a3730da64a41-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-49nqr\" (UID: \"564df912-a1a2-45a2-b8a5-a3730da64a41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-49nqr" Oct 31 01:04:35.440673 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:04:35.440646 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9scdt\" (UniqueName: \"kubernetes.io/projected/564df912-a1a2-45a2-b8a5-a3730da64a41-kube-api-access-9scdt\") pod \"maas-api-revoke-sub-046e1343-49nqr\" (UID: \"564df912-a1a2-45a2-b8a5-a3730da64a41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-49nqr" Oct 31 01:04:35.550612 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:04:35.550574 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-49nqr" Oct 31 01:04:35.889632 ip-10-0-137-56 kubenswrapper[5222]: E1031 01:04:35.889544 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 01:04:35.923195 ip-10-0-137-56 kubenswrapper[5222]: W1031 01:04:35.923149 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod564df912_a1a2_45a2_b8a5_a3730da64a41.slice/crio-8d5a854c0fc36fca6964f28bd57fdc6e36266f6a36e0d80ca8a5e31337529162 WatchSource:0}: Error finding container 8d5a854c0fc36fca6964f28bd57fdc6e36266f6a36e0d80ca8a5e31337529162: Status 404 returned error can't find the container with id 8d5a854c0fc36fca6964f28bd57fdc6e36266f6a36e0d80ca8a5e31337529162 Oct 31 01:04:35.925193 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:04:35.925171 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-49nqr"] Oct 31 01:04:36.477625 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:04:36.477527 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-49nqr" event={"ID":"564df912-a1a2-45a2-b8a5-a3730da64a41","Type":"ContainerStarted","Data":"85dad0f4844d7d6bf90ab882c0b8777a3afb33c83dfb671a0ab5f788393218d2"} Oct 31 01:04:36.477625 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:04:36.477585 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-49nqr" event={"ID":"564df912-a1a2-45a2-b8a5-a3730da64a41","Type":"ContainerStarted","Data":"8d5a854c0fc36fca6964f28bd57fdc6e36266f6a36e0d80ca8a5e31337529162"} Oct 31 01:04:36.596719 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:04:36.596656 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-49nqr" podStartSLOduration=1.596641762 podStartE2EDuration="1.596641762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:04:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:04:36.595235556 +0000 UTC m=+2969.066405305" watchObservedRunningTime="2026-10-31 01:04:36.596641762 +0000 UTC m=+2969.067811512" Oct 31 01:04:37.484513 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:04:37.484476 5222 generic.go:356] "Generic (PLEG): container finished" podID="564df912-a1a2-45a2-b8a5-a3730da64a41" containerID="85dad0f4844d7d6bf90ab882c0b8777a3afb33c83dfb671a0ab5f788393218d2" exitCode=0 Oct 31 01:04:37.484513 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:04:37.484514 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-49nqr" event={"ID":"564df912-a1a2-45a2-b8a5-a3730da64a41","Type":"ContainerDied","Data":"85dad0f4844d7d6bf90ab882c0b8777a3afb33c83dfb671a0ab5f788393218d2"} Oct 31 01:04:38.643460 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:04:38.643433 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-49nqr" Oct 31 01:04:38.712784 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:04:38.712740 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/564df912-a1a2-45a2-b8a5-a3730da64a41-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/564df912-a1a2-45a2-b8a5-a3730da64a41-maas-api-openshift-ca\") pod \"564df912-a1a2-45a2-b8a5-a3730da64a41\" (UID: \"564df912-a1a2-45a2-b8a5-a3730da64a41\") " Oct 31 01:04:38.712784 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:04:38.712799 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/564df912-a1a2-45a2-b8a5-a3730da64a41-kube-api-access-9scdt\" (UniqueName: \"kubernetes.io/projected/564df912-a1a2-45a2-b8a5-a3730da64a41-kube-api-access-9scdt\") pod \"564df912-a1a2-45a2-b8a5-a3730da64a41\" (UID: \"564df912-a1a2-45a2-b8a5-a3730da64a41\") " Oct 31 01:04:38.713079 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:04:38.712920 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/564df912-a1a2-45a2-b8a5-a3730da64a41-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/564df912-a1a2-45a2-b8a5-a3730da64a41-maas-api-xks-ca\") pod \"564df912-a1a2-45a2-b8a5-a3730da64a41\" (UID: \"564df912-a1a2-45a2-b8a5-a3730da64a41\") " Oct 31 01:04:38.713161 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:04:38.713137 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/564df912-a1a2-45a2-b8a5-a3730da64a41-maas-api-openshift-ca" pod "564df912-a1a2-45a2-b8a5-a3730da64a41" (UID: "564df912-a1a2-45a2-b8a5-a3730da64a41"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:04:38.715181 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:04:38.715147 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/564df912-a1a2-45a2-b8a5-a3730da64a41-maas-api-xks-ca" pod "564df912-a1a2-45a2-b8a5-a3730da64a41" (UID: "564df912-a1a2-45a2-b8a5-a3730da64a41"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:04:38.715181 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:04:38.715164 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/564df912-a1a2-45a2-b8a5-a3730da64a41-kube-api-access-9scdt" pod "564df912-a1a2-45a2-b8a5-a3730da64a41" (UID: "564df912-a1a2-45a2-b8a5-a3730da64a41"). InnerVolumeSpecName "kube-api-access-9scdt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:04:38.814064 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:04:38.814027 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/564df912-a1a2-45a2-b8a5-a3730da64a41-maas-api-openshift-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:04:38.814064 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:04:38.814059 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9scdt\" (UniqueName: \"kubernetes.io/projected/564df912-a1a2-45a2-b8a5-a3730da64a41-kube-api-access-9scdt\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:04:38.814064 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:04:38.814069 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/564df912-a1a2-45a2-b8a5-a3730da64a41-maas-api-xks-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:04:39.495985 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:04:39.495955 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-49nqr" Oct 31 01:04:39.496193 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:04:39.495955 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-49nqr" event={"ID":"564df912-a1a2-45a2-b8a5-a3730da64a41","Type":"ContainerDied","Data":"8d5a854c0fc36fca6964f28bd57fdc6e36266f6a36e0d80ca8a5e31337529162"} Oct 31 01:04:39.496193 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:04:39.496075 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8d5a854c0fc36fca6964f28bd57fdc6e36266f6a36e0d80ca8a5e31337529162" Oct 31 01:04:48.110331 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:04:48.110245 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-79tk8"] Oct 31 01:04:58.922527 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:04:58.922493 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-79tk8"] Oct 31 01:05:05.903523 ip-10-0-137-56 kubenswrapper[5222]: E1031 01:05:05.903474 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 01:05:08.235905 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:05:08.235865 5222 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h9hsf_0f8c3c03-a1fc-4376-98c6-feeeefa08068/ovn-acl-logging/0.log" Oct 31 01:05:08.241099 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:05:08.241062 5222 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h9hsf_0f8c3c03-a1fc-4376-98c6-feeeefa08068/ovn-acl-logging/0.log" Oct 31 01:05:10.536174 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:05:10.536138 5222 scope.go:122] "RemoveContainer" containerID="688574db526f610ab2349927ae41443fb03891d025ef268f8544a109c4099ee1" Oct 31 01:05:10.545335 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:05:10.545306 5222 scope.go:122] "RemoveContainer" containerID="73dbb9cb87b4b3eedb8a1e7f218c3c5f28c69f635a38d966f1acbf5eb83d3326" Oct 31 01:05:11.131250 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:05:11.131214 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-79tk8"] Oct 31 01:05:21.136456 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:05:21.136411 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d7652"] Oct 31 01:05:21.136992 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:05:21.136874 5222 cpu_manager.go:414] "containerMap: removing container" podUID="564df912-a1a2-45a2-b8a5-a3730da64a41" containerName="revoke-keys" Oct 31 01:05:21.136992 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:05:21.136888 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="564df912-a1a2-45a2-b8a5-a3730da64a41" containerName="revoke-keys" Oct 31 01:05:21.136992 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:05:21.136955 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="564df912-a1a2-45a2-b8a5-a3730da64a41" containerName="revoke-keys" Oct 31 01:05:21.141874 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:05:21.141851 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d7652" Oct 31 01:05:21.194313 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:05:21.194282 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d7652"] Oct 31 01:05:21.243866 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:05:21.243819 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:05:21.243866 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:05:21.243824 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:05:21.243866 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:05:21.243864 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:05:21.244124 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:05:21.243911 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfz9z\"" Oct 31 01:05:21.287059 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:05:21.287022 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsxm6\" (UniqueName: \"kubernetes.io/projected/cb60e8c4-ec26-4826-9380-b8d379e52805-kube-api-access-nsxm6\") pod \"maas-api-revoke-sub-ae594979-d7652\" (UID: \"cb60e8c4-ec26-4826-9380-b8d379e52805\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d7652" Oct 31 01:05:21.287248 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:05:21.287075 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb60e8c4-ec26-4826-9380-b8d379e52805-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-d7652\" (UID: \"cb60e8c4-ec26-4826-9380-b8d379e52805\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d7652" Oct 31 01:05:21.287302 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:05:21.287265 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb60e8c4-ec26-4826-9380-b8d379e52805-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-d7652\" (UID: \"cb60e8c4-ec26-4826-9380-b8d379e52805\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d7652" Oct 31 01:05:21.388422 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:05:21.388298 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb60e8c4-ec26-4826-9380-b8d379e52805-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-d7652\" (UID: \"cb60e8c4-ec26-4826-9380-b8d379e52805\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d7652" Oct 31 01:05:21.388422 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:05:21.388353 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsxm6\" (UniqueName: \"kubernetes.io/projected/cb60e8c4-ec26-4826-9380-b8d379e52805-kube-api-access-nsxm6\") pod \"maas-api-revoke-sub-ae594979-d7652\" (UID: \"cb60e8c4-ec26-4826-9380-b8d379e52805\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d7652" Oct 31 01:05:21.388639 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:05:21.388437 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb60e8c4-ec26-4826-9380-b8d379e52805-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-d7652\" (UID: \"cb60e8c4-ec26-4826-9380-b8d379e52805\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d7652" Oct 31 01:05:21.389160 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:05:21.389140 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb60e8c4-ec26-4826-9380-b8d379e52805-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-d7652\" (UID: \"cb60e8c4-ec26-4826-9380-b8d379e52805\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d7652" Oct 31 01:05:21.390977 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:05:21.390958 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb60e8c4-ec26-4826-9380-b8d379e52805-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-d7652\" (UID: \"cb60e8c4-ec26-4826-9380-b8d379e52805\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d7652" Oct 31 01:05:21.451214 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:05:21.451183 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsxm6\" (UniqueName: \"kubernetes.io/projected/cb60e8c4-ec26-4826-9380-b8d379e52805-kube-api-access-nsxm6\") pod \"maas-api-revoke-sub-ae594979-d7652\" (UID: \"cb60e8c4-ec26-4826-9380-b8d379e52805\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d7652" Oct 31 01:05:21.459014 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:05:21.458987 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d7652" Oct 31 01:05:21.831821 ip-10-0-137-56 kubenswrapper[5222]: W1031 01:05:21.831786 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb60e8c4_ec26_4826_9380_b8d379e52805.slice/crio-38577a7222c23ce5182845e3dae5f8199bc5a4e622c4dc32f6db2f757ef91a5f WatchSource:0}: Error finding container 38577a7222c23ce5182845e3dae5f8199bc5a4e622c4dc32f6db2f757ef91a5f: Status 404 returned error can't find the container with id 38577a7222c23ce5182845e3dae5f8199bc5a4e622c4dc32f6db2f757ef91a5f Oct 31 01:05:21.834205 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:05:21.834176 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d7652"] Oct 31 01:05:22.807952 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:05:22.807909 5222 generic.go:356] "Generic (PLEG): container finished" podID="cb60e8c4-ec26-4826-9380-b8d379e52805" containerID="9d2eb54c603eff9cd8ee017a9b8172c532b6d556a79df8e9fcfd89a3101becc4" exitCode=0 Oct 31 01:05:22.808413 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:05:22.807987 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d7652" event={"ID":"cb60e8c4-ec26-4826-9380-b8d379e52805","Type":"ContainerDied","Data":"9d2eb54c603eff9cd8ee017a9b8172c532b6d556a79df8e9fcfd89a3101becc4"} Oct 31 01:05:22.808413 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:05:22.808028 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d7652" event={"ID":"cb60e8c4-ec26-4826-9380-b8d379e52805","Type":"ContainerStarted","Data":"38577a7222c23ce5182845e3dae5f8199bc5a4e622c4dc32f6db2f757ef91a5f"} Oct 31 01:05:23.947693 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:05:23.947666 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d7652" Oct 31 01:05:24.114018 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:05:24.113928 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb60e8c4-ec26-4826-9380-b8d379e52805-kube-api-access-nsxm6\" (UniqueName: \"kubernetes.io/projected/cb60e8c4-ec26-4826-9380-b8d379e52805-kube-api-access-nsxm6\") pod \"cb60e8c4-ec26-4826-9380-b8d379e52805\" (UID: \"cb60e8c4-ec26-4826-9380-b8d379e52805\") " Oct 31 01:05:24.114184 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:05:24.114018 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb60e8c4-ec26-4826-9380-b8d379e52805-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb60e8c4-ec26-4826-9380-b8d379e52805-maas-api-xks-ca\") pod \"cb60e8c4-ec26-4826-9380-b8d379e52805\" (UID: \"cb60e8c4-ec26-4826-9380-b8d379e52805\") " Oct 31 01:05:24.114184 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:05:24.114066 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb60e8c4-ec26-4826-9380-b8d379e52805-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb60e8c4-ec26-4826-9380-b8d379e52805-maas-api-openshift-ca\") pod \"cb60e8c4-ec26-4826-9380-b8d379e52805\" (UID: \"cb60e8c4-ec26-4826-9380-b8d379e52805\") " Oct 31 01:05:24.114517 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:05:24.114484 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb60e8c4-ec26-4826-9380-b8d379e52805-maas-api-openshift-ca" pod "cb60e8c4-ec26-4826-9380-b8d379e52805" (UID: "cb60e8c4-ec26-4826-9380-b8d379e52805"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:05:24.116267 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:05:24.116238 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb60e8c4-ec26-4826-9380-b8d379e52805-maas-api-xks-ca" pod "cb60e8c4-ec26-4826-9380-b8d379e52805" (UID: "cb60e8c4-ec26-4826-9380-b8d379e52805"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:05:24.116267 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:05:24.116256 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb60e8c4-ec26-4826-9380-b8d379e52805-kube-api-access-nsxm6" pod "cb60e8c4-ec26-4826-9380-b8d379e52805" (UID: "cb60e8c4-ec26-4826-9380-b8d379e52805"). InnerVolumeSpecName "kube-api-access-nsxm6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:05:24.214953 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:05:24.214909 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nsxm6\" (UniqueName: \"kubernetes.io/projected/cb60e8c4-ec26-4826-9380-b8d379e52805-kube-api-access-nsxm6\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:05:24.214953 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:05:24.214945 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb60e8c4-ec26-4826-9380-b8d379e52805-maas-api-xks-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:05:24.214953 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:05:24.214955 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb60e8c4-ec26-4826-9380-b8d379e52805-maas-api-openshift-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:05:24.818775 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:05:24.818746 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d7652" Oct 31 01:05:24.818972 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:05:24.818774 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d7652" event={"ID":"cb60e8c4-ec26-4826-9380-b8d379e52805","Type":"ContainerDied","Data":"38577a7222c23ce5182845e3dae5f8199bc5a4e622c4dc32f6db2f757ef91a5f"} Oct 31 01:05:24.818972 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:05:24.818823 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="38577a7222c23ce5182845e3dae5f8199bc5a4e622c4dc32f6db2f757ef91a5f" Oct 31 01:05:34.127421 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:05:34.127389 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-79tk8"] Oct 31 01:05:35.889796 ip-10-0-137-56 kubenswrapper[5222]: E1031 01:05:35.889751 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 01:05:43.685193 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:05:43.685156 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-79tk8"] Oct 31 01:05:49.407975 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:05:49.407942 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hc4gw"] Oct 31 01:05:49.408422 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:05:49.408314 5222 cpu_manager.go:414] "containerMap: removing container" podUID="cb60e8c4-ec26-4826-9380-b8d379e52805" containerName="revoke-keys" Oct 31 01:05:49.408422 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:05:49.408325 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb60e8c4-ec26-4826-9380-b8d379e52805" containerName="revoke-keys" Oct 31 01:05:49.408522 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:05:49.408449 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb60e8c4-ec26-4826-9380-b8d379e52805" containerName="revoke-keys" Oct 31 01:05:49.411670 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:05:49.411646 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hc4gw" Oct 31 01:05:49.466300 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:05:49.466265 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hc4gw"] Oct 31 01:05:49.472937 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:05:49.472898 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf20b806-e779-41a5-983b-397ffa538b53-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-hc4gw\" (UID: \"bf20b806-e779-41a5-983b-397ffa538b53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hc4gw" Oct 31 01:05:49.473111 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:05:49.472957 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf20b806-e779-41a5-983b-397ffa538b53-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-hc4gw\" (UID: \"bf20b806-e779-41a5-983b-397ffa538b53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hc4gw" Oct 31 01:05:49.473111 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:05:49.473068 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcw88\" (UniqueName: \"kubernetes.io/projected/bf20b806-e779-41a5-983b-397ffa538b53-kube-api-access-rcw88\") pod \"maas-api-revoke-sub-a4aef7d7-hc4gw\" (UID: \"bf20b806-e779-41a5-983b-397ffa538b53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hc4gw" Oct 31 01:05:49.514679 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:05:49.514644 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:05:49.514679 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:05:49.514669 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:05:49.514901 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:05:49.514657 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:05:49.514901 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:05:49.514734 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfz9z\"" Oct 31 01:05:49.573661 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:05:49.573622 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf20b806-e779-41a5-983b-397ffa538b53-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-hc4gw\" (UID: \"bf20b806-e779-41a5-983b-397ffa538b53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hc4gw" Oct 31 01:05:49.573883 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:05:49.573673 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcw88\" (UniqueName: \"kubernetes.io/projected/bf20b806-e779-41a5-983b-397ffa538b53-kube-api-access-rcw88\") pod \"maas-api-revoke-sub-a4aef7d7-hc4gw\" (UID: \"bf20b806-e779-41a5-983b-397ffa538b53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hc4gw" Oct 31 01:05:49.573883 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:05:49.573736 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf20b806-e779-41a5-983b-397ffa538b53-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-hc4gw\" (UID: \"bf20b806-e779-41a5-983b-397ffa538b53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hc4gw" Oct 31 01:05:49.574342 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:05:49.574305 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf20b806-e779-41a5-983b-397ffa538b53-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-hc4gw\" (UID: \"bf20b806-e779-41a5-983b-397ffa538b53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hc4gw" Oct 31 01:05:49.576181 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:05:49.576159 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf20b806-e779-41a5-983b-397ffa538b53-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-hc4gw\" (UID: \"bf20b806-e779-41a5-983b-397ffa538b53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hc4gw" Oct 31 01:05:49.636570 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:05:49.636540 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcw88\" (UniqueName: \"kubernetes.io/projected/bf20b806-e779-41a5-983b-397ffa538b53-kube-api-access-rcw88\") pod \"maas-api-revoke-sub-a4aef7d7-hc4gw\" (UID: \"bf20b806-e779-41a5-983b-397ffa538b53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hc4gw" Oct 31 01:05:49.725549 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:05:49.725451 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hc4gw" Oct 31 01:05:50.102081 ip-10-0-137-56 kubenswrapper[5222]: W1031 01:05:50.102037 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf20b806_e779_41a5_983b_397ffa538b53.slice/crio-c759de722de7acb908dac3d0fd663d0c8bada5c0ee70ad1d652106dd3f6c3f16 WatchSource:0}: Error finding container c759de722de7acb908dac3d0fd663d0c8bada5c0ee70ad1d652106dd3f6c3f16: Status 404 returned error can't find the container with id c759de722de7acb908dac3d0fd663d0c8bada5c0ee70ad1d652106dd3f6c3f16 Oct 31 01:05:50.104301 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:05:50.104277 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hc4gw"] Oct 31 01:05:50.959903 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:05:50.959870 5222 generic.go:356] "Generic (PLEG): container finished" podID="bf20b806-e779-41a5-983b-397ffa538b53" containerID="2ac70ee1f335c32ef27619426ef2975390d7eee39c900f9939e0c7088d31d2fe" exitCode=0 Oct 31 01:05:50.960315 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:05:50.959952 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hc4gw" event={"ID":"bf20b806-e779-41a5-983b-397ffa538b53","Type":"ContainerDied","Data":"2ac70ee1f335c32ef27619426ef2975390d7eee39c900f9939e0c7088d31d2fe"} Oct 31 01:05:50.960315 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:05:50.960001 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hc4gw" event={"ID":"bf20b806-e779-41a5-983b-397ffa538b53","Type":"ContainerStarted","Data":"c759de722de7acb908dac3d0fd663d0c8bada5c0ee70ad1d652106dd3f6c3f16"} Oct 31 01:05:52.109449 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:05:52.109419 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hc4gw" Oct 31 01:05:52.199853 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:05:52.199816 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf20b806-e779-41a5-983b-397ffa538b53-kube-api-access-rcw88\" (UniqueName: \"kubernetes.io/projected/bf20b806-e779-41a5-983b-397ffa538b53-kube-api-access-rcw88\") pod \"bf20b806-e779-41a5-983b-397ffa538b53\" (UID: \"bf20b806-e779-41a5-983b-397ffa538b53\") " Oct 31 01:05:52.200044 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:05:52.199919 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf20b806-e779-41a5-983b-397ffa538b53-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf20b806-e779-41a5-983b-397ffa538b53-maas-api-xks-ca\") pod \"bf20b806-e779-41a5-983b-397ffa538b53\" (UID: \"bf20b806-e779-41a5-983b-397ffa538b53\") " Oct 31 01:05:52.200044 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:05:52.199964 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf20b806-e779-41a5-983b-397ffa538b53-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf20b806-e779-41a5-983b-397ffa538b53-maas-api-openshift-ca\") pod \"bf20b806-e779-41a5-983b-397ffa538b53\" (UID: \"bf20b806-e779-41a5-983b-397ffa538b53\") " Oct 31 01:05:52.200393 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:05:52.200341 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf20b806-e779-41a5-983b-397ffa538b53-maas-api-openshift-ca" pod "bf20b806-e779-41a5-983b-397ffa538b53" (UID: "bf20b806-e779-41a5-983b-397ffa538b53"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:05:52.202141 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:05:52.202116 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf20b806-e779-41a5-983b-397ffa538b53-kube-api-access-rcw88" pod "bf20b806-e779-41a5-983b-397ffa538b53" (UID: "bf20b806-e779-41a5-983b-397ffa538b53"). InnerVolumeSpecName "kube-api-access-rcw88". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:05:52.202252 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:05:52.202166 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf20b806-e779-41a5-983b-397ffa538b53-maas-api-xks-ca" pod "bf20b806-e779-41a5-983b-397ffa538b53" (UID: "bf20b806-e779-41a5-983b-397ffa538b53"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:05:52.301875 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:05:52.301836 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf20b806-e779-41a5-983b-397ffa538b53-maas-api-xks-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:05:52.302164 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:05:52.302143 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf20b806-e779-41a5-983b-397ffa538b53-maas-api-openshift-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:05:52.302262 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:05:52.302250 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcw88\" (UniqueName: \"kubernetes.io/projected/bf20b806-e779-41a5-983b-397ffa538b53-kube-api-access-rcw88\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:05:52.970450 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:05:52.970421 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hc4gw" Oct 31 01:05:52.970450 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:05:52.970441 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hc4gw" event={"ID":"bf20b806-e779-41a5-983b-397ffa538b53","Type":"ContainerDied","Data":"c759de722de7acb908dac3d0fd663d0c8bada5c0ee70ad1d652106dd3f6c3f16"} Oct 31 01:05:52.970672 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:05:52.970489 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c759de722de7acb908dac3d0fd663d0c8bada5c0ee70ad1d652106dd3f6c3f16" Oct 31 01:06:02.332541 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:06:02.332501 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-79tk8"] Oct 31 01:06:05.890321 ip-10-0-137-56 kubenswrapper[5222]: E1031 01:06:05.890285 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 01:06:10.571397 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:06:10.571218 5222 scope.go:122] "RemoveContainer" containerID="799b7351b0255223fd1530ba1515955e8ab19d3e77c3e979dbde54567137df7d" Oct 31 01:06:10.613887 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:06:10.613457 5222 scope.go:122] "RemoveContainer" containerID="114183d4066cf62037a2f026023764665f461ece8a5487d5c819b65bf0f8c134" Oct 31 01:06:14.652181 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:06:14.652099 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-79tk8"] Oct 31 01:06:18.074186 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:06:18.074148 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-784a983c-mttgm"] Oct 31 01:06:18.125842 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:06:18.125795 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2fec8671-lhx4k"] Oct 31 01:06:18.127403 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:06:18.127388 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-784a983c-mttgm"] Oct 31 01:06:18.128848 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:06:18.128825 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2fec8671-lhx4k"] Oct 31 01:06:19.654524 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:06:19.654490 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0599b1e3-1033-48ab-bd20-b5446d96dd13" path="/var/lib/kubelet/pods/0599b1e3-1033-48ab-bd20-b5446d96dd13/volumes" Oct 31 01:06:19.654922 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:06:19.654900 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d8b71ff-c6de-4b8c-8e78-175b48c7ed2e" path="/var/lib/kubelet/pods/0d8b71ff-c6de-4b8c-8e78-175b48c7ed2e/volumes" Oct 31 01:06:20.373918 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:06:20.373890 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-nqt8l"] Oct 31 01:06:20.374276 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:06:20.374263 5222 cpu_manager.go:414] "containerMap: removing container" podUID="bf20b806-e779-41a5-983b-397ffa538b53" containerName="revoke-keys" Oct 31 01:06:20.374332 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:06:20.374278 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf20b806-e779-41a5-983b-397ffa538b53" containerName="revoke-keys" Oct 31 01:06:20.374384 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:06:20.374351 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf20b806-e779-41a5-983b-397ffa538b53" containerName="revoke-keys" Oct 31 01:06:20.378877 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:06:20.378851 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-nqt8l" Oct 31 01:06:20.410459 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:06:20.410418 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmms2\" (UniqueName: \"kubernetes.io/projected/f73d8c55-49c1-4530-8ad7-08514b80cdfa-kube-api-access-rmms2\") pod \"maas-api-revoke-sub-79805ca9-nqt8l\" (UID: \"f73d8c55-49c1-4530-8ad7-08514b80cdfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-nqt8l" Oct 31 01:06:20.410459 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:06:20.410464 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f73d8c55-49c1-4530-8ad7-08514b80cdfa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-nqt8l\" (UID: \"f73d8c55-49c1-4530-8ad7-08514b80cdfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-nqt8l" Oct 31 01:06:20.410739 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:06:20.410486 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f73d8c55-49c1-4530-8ad7-08514b80cdfa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-nqt8l\" (UID: \"f73d8c55-49c1-4530-8ad7-08514b80cdfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-nqt8l" Oct 31 01:06:20.431880 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:06:20.431846 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-nqt8l"] Oct 31 01:06:20.481231 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:06:20.481192 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:06:20.481461 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:06:20.481245 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:06:20.481461 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:06:20.481295 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfz9z\"" Oct 31 01:06:20.485151 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:06:20.485134 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:06:20.511334 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:06:20.511301 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmms2\" (UniqueName: \"kubernetes.io/projected/f73d8c55-49c1-4530-8ad7-08514b80cdfa-kube-api-access-rmms2\") pod \"maas-api-revoke-sub-79805ca9-nqt8l\" (UID: \"f73d8c55-49c1-4530-8ad7-08514b80cdfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-nqt8l" Oct 31 01:06:20.511334 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:06:20.511336 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f73d8c55-49c1-4530-8ad7-08514b80cdfa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-nqt8l\" (UID: \"f73d8c55-49c1-4530-8ad7-08514b80cdfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-nqt8l" Oct 31 01:06:20.511564 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:06:20.511363 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f73d8c55-49c1-4530-8ad7-08514b80cdfa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-nqt8l\" (UID: \"f73d8c55-49c1-4530-8ad7-08514b80cdfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-nqt8l" Oct 31 01:06:20.512123 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:06:20.512097 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f73d8c55-49c1-4530-8ad7-08514b80cdfa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-nqt8l\" (UID: \"f73d8c55-49c1-4530-8ad7-08514b80cdfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-nqt8l" Oct 31 01:06:20.513924 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:06:20.513898 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f73d8c55-49c1-4530-8ad7-08514b80cdfa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-nqt8l\" (UID: \"f73d8c55-49c1-4530-8ad7-08514b80cdfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-nqt8l" Oct 31 01:06:20.574591 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:06:20.574563 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmms2\" (UniqueName: \"kubernetes.io/projected/f73d8c55-49c1-4530-8ad7-08514b80cdfa-kube-api-access-rmms2\") pod \"maas-api-revoke-sub-79805ca9-nqt8l\" (UID: \"f73d8c55-49c1-4530-8ad7-08514b80cdfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-nqt8l" Oct 31 01:06:20.692673 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:06:20.692584 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-nqt8l" Oct 31 01:06:21.067520 ip-10-0-137-56 kubenswrapper[5222]: W1031 01:06:21.067487 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf73d8c55_49c1_4530_8ad7_08514b80cdfa.slice/crio-605dd4f55a94f18cbc115b5800af0256e2831cd8a94001cba81fff19a10f83bc WatchSource:0}: Error finding container 605dd4f55a94f18cbc115b5800af0256e2831cd8a94001cba81fff19a10f83bc: Status 404 returned error can't find the container with id 605dd4f55a94f18cbc115b5800af0256e2831cd8a94001cba81fff19a10f83bc Oct 31 01:06:21.069631 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:06:21.069602 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-nqt8l"] Oct 31 01:06:21.184796 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:06:21.184753 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-nqt8l" event={"ID":"f73d8c55-49c1-4530-8ad7-08514b80cdfa","Type":"ContainerStarted","Data":"605dd4f55a94f18cbc115b5800af0256e2831cd8a94001cba81fff19a10f83bc"} Oct 31 01:06:22.191547 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:06:22.191452 5222 generic.go:356] "Generic (PLEG): container finished" podID="f73d8c55-49c1-4530-8ad7-08514b80cdfa" containerID="19b44b29e3994c0cbc9503f58e043183e75137fc2f4f66751d38002ef0b734f3" exitCode=0 Oct 31 01:06:22.191950 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:06:22.191540 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-nqt8l" event={"ID":"f73d8c55-49c1-4530-8ad7-08514b80cdfa","Type":"ContainerDied","Data":"19b44b29e3994c0cbc9503f58e043183e75137fc2f4f66751d38002ef0b734f3"} Oct 31 01:06:23.332847 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:06:23.332816 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-nqt8l" Oct 31 01:06:23.442883 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:06:23.442850 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f73d8c55-49c1-4530-8ad7-08514b80cdfa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f73d8c55-49c1-4530-8ad7-08514b80cdfa-maas-api-openshift-ca\") pod \"f73d8c55-49c1-4530-8ad7-08514b80cdfa\" (UID: \"f73d8c55-49c1-4530-8ad7-08514b80cdfa\") " Oct 31 01:06:23.443064 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:06:23.442999 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f73d8c55-49c1-4530-8ad7-08514b80cdfa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f73d8c55-49c1-4530-8ad7-08514b80cdfa-maas-api-xks-ca\") pod \"f73d8c55-49c1-4530-8ad7-08514b80cdfa\" (UID: \"f73d8c55-49c1-4530-8ad7-08514b80cdfa\") " Oct 31 01:06:23.443064 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:06:23.443040 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f73d8c55-49c1-4530-8ad7-08514b80cdfa-kube-api-access-rmms2\" (UniqueName: \"kubernetes.io/projected/f73d8c55-49c1-4530-8ad7-08514b80cdfa-kube-api-access-rmms2\") pod \"f73d8c55-49c1-4530-8ad7-08514b80cdfa\" (UID: \"f73d8c55-49c1-4530-8ad7-08514b80cdfa\") " Oct 31 01:06:23.443267 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:06:23.443234 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f73d8c55-49c1-4530-8ad7-08514b80cdfa-maas-api-openshift-ca" pod "f73d8c55-49c1-4530-8ad7-08514b80cdfa" (UID: "f73d8c55-49c1-4530-8ad7-08514b80cdfa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:06:23.443404 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:06:23.443349 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f73d8c55-49c1-4530-8ad7-08514b80cdfa-maas-api-openshift-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:06:23.445474 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:06:23.445447 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f73d8c55-49c1-4530-8ad7-08514b80cdfa-kube-api-access-rmms2" pod "f73d8c55-49c1-4530-8ad7-08514b80cdfa" (UID: "f73d8c55-49c1-4530-8ad7-08514b80cdfa"). InnerVolumeSpecName "kube-api-access-rmms2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:06:23.445562 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:06:23.445485 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f73d8c55-49c1-4530-8ad7-08514b80cdfa-maas-api-xks-ca" pod "f73d8c55-49c1-4530-8ad7-08514b80cdfa" (UID: "f73d8c55-49c1-4530-8ad7-08514b80cdfa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:06:23.544725 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:06:23.544693 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmms2\" (UniqueName: \"kubernetes.io/projected/f73d8c55-49c1-4530-8ad7-08514b80cdfa-kube-api-access-rmms2\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:06:23.544725 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:06:23.544719 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f73d8c55-49c1-4530-8ad7-08514b80cdfa-maas-api-xks-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:06:24.202454 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:06:24.202420 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-nqt8l" event={"ID":"f73d8c55-49c1-4530-8ad7-08514b80cdfa","Type":"ContainerDied","Data":"605dd4f55a94f18cbc115b5800af0256e2831cd8a94001cba81fff19a10f83bc"} Oct 31 01:06:24.202454 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:06:24.202440 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-nqt8l" Oct 31 01:06:24.202683 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:06:24.202469 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="605dd4f55a94f18cbc115b5800af0256e2831cd8a94001cba81fff19a10f83bc" Oct 31 01:06:35.012044 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:06:35.012005 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-79tk8"] Oct 31 01:06:35.902891 ip-10-0-137-56 kubenswrapper[5222]: E1031 01:06:35.902847 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 01:06:44.371672 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:06:44.371633 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rrfmz"] Oct 31 01:06:44.374009 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:06:44.372084 5222 cpu_manager.go:414] "containerMap: removing container" podUID="f73d8c55-49c1-4530-8ad7-08514b80cdfa" containerName="revoke-keys" Oct 31 01:06:44.374009 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:06:44.372096 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f73d8c55-49c1-4530-8ad7-08514b80cdfa" containerName="revoke-keys" Oct 31 01:06:44.374009 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:06:44.372181 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="f73d8c55-49c1-4530-8ad7-08514b80cdfa" containerName="revoke-keys" Oct 31 01:06:44.375127 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:06:44.375110 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rrfmz" Oct 31 01:06:44.429894 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:06:44.429858 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rrfmz"] Oct 31 01:06:44.438563 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:06:44.438531 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:06:44.478859 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:06:44.478827 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfz9z\"" Oct 31 01:06:44.479078 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:06:44.479058 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:06:44.479172 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:06:44.479094 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:06:44.483697 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:06:44.483672 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqb58\" (UniqueName: \"kubernetes.io/projected/4c897631-8986-4fd2-bab7-afeae8584d6f-kube-api-access-dqb58\") pod \"maas-api-revoke-sub-0b7e8edb-rrfmz\" (UID: \"4c897631-8986-4fd2-bab7-afeae8584d6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rrfmz" Oct 31 01:06:44.483789 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:06:44.483713 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c897631-8986-4fd2-bab7-afeae8584d6f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-rrfmz\" (UID: \"4c897631-8986-4fd2-bab7-afeae8584d6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rrfmz" Oct 31 01:06:44.483840 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:06:44.483800 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c897631-8986-4fd2-bab7-afeae8584d6f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-rrfmz\" (UID: \"4c897631-8986-4fd2-bab7-afeae8584d6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rrfmz" Oct 31 01:06:44.585363 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:06:44.585324 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c897631-8986-4fd2-bab7-afeae8584d6f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-rrfmz\" (UID: \"4c897631-8986-4fd2-bab7-afeae8584d6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rrfmz" Oct 31 01:06:44.585571 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:06:44.585458 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqb58\" (UniqueName: \"kubernetes.io/projected/4c897631-8986-4fd2-bab7-afeae8584d6f-kube-api-access-dqb58\") pod \"maas-api-revoke-sub-0b7e8edb-rrfmz\" (UID: \"4c897631-8986-4fd2-bab7-afeae8584d6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rrfmz" Oct 31 01:06:44.585571 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:06:44.585502 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c897631-8986-4fd2-bab7-afeae8584d6f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-rrfmz\" (UID: \"4c897631-8986-4fd2-bab7-afeae8584d6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rrfmz" Oct 31 01:06:44.586111 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:06:44.586087 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c897631-8986-4fd2-bab7-afeae8584d6f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-rrfmz\" (UID: \"4c897631-8986-4fd2-bab7-afeae8584d6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rrfmz" Oct 31 01:06:44.588011 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:06:44.587978 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c897631-8986-4fd2-bab7-afeae8584d6f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-rrfmz\" (UID: \"4c897631-8986-4fd2-bab7-afeae8584d6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rrfmz" Oct 31 01:06:44.648396 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:06:44.648298 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqb58\" (UniqueName: \"kubernetes.io/projected/4c897631-8986-4fd2-bab7-afeae8584d6f-kube-api-access-dqb58\") pod \"maas-api-revoke-sub-0b7e8edb-rrfmz\" (UID: \"4c897631-8986-4fd2-bab7-afeae8584d6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rrfmz" Oct 31 01:06:44.698425 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:06:44.698389 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rrfmz" Oct 31 01:06:45.072161 ip-10-0-137-56 kubenswrapper[5222]: W1031 01:06:45.072114 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c897631_8986_4fd2_bab7_afeae8584d6f.slice/crio-f0e812ed69291e6f7d8d2e7f0ec9b67c93c02f8eb68e629ebbf892d6c79f15c0 WatchSource:0}: Error finding container f0e812ed69291e6f7d8d2e7f0ec9b67c93c02f8eb68e629ebbf892d6c79f15c0: Status 404 returned error can't find the container with id f0e812ed69291e6f7d8d2e7f0ec9b67c93c02f8eb68e629ebbf892d6c79f15c0 Oct 31 01:06:45.074527 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:06:45.074499 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rrfmz"] Oct 31 01:06:45.323996 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:06:45.323891 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rrfmz" event={"ID":"4c897631-8986-4fd2-bab7-afeae8584d6f","Type":"ContainerStarted","Data":"fb3b71080fcf0fb81661ef6345110da84edc853f3fe7cdc6f7b6d09b061c01d4"} Oct 31 01:06:45.323996 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:06:45.323960 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rrfmz" event={"ID":"4c897631-8986-4fd2-bab7-afeae8584d6f","Type":"ContainerStarted","Data":"f0e812ed69291e6f7d8d2e7f0ec9b67c93c02f8eb68e629ebbf892d6c79f15c0"} Oct 31 01:06:45.443950 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:06:45.443867 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rrfmz" podStartSLOduration=1.443853442 podStartE2EDuration="1.443853442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:06:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:06:45.441956283 +0000 UTC m=+3097.913126038" watchObservedRunningTime="2026-10-31 01:06:45.443853442 +0000 UTC m=+3097.915023191" Oct 31 01:06:46.330239 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:06:46.330203 5222 generic.go:356] "Generic (PLEG): container finished" podID="4c897631-8986-4fd2-bab7-afeae8584d6f" containerID="fb3b71080fcf0fb81661ef6345110da84edc853f3fe7cdc6f7b6d09b061c01d4" exitCode=0 Oct 31 01:06:46.330239 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:06:46.330241 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rrfmz" event={"ID":"4c897631-8986-4fd2-bab7-afeae8584d6f","Type":"ContainerDied","Data":"fb3b71080fcf0fb81661ef6345110da84edc853f3fe7cdc6f7b6d09b061c01d4"} Oct 31 01:06:47.473744 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:06:47.473719 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rrfmz" Oct 31 01:06:47.616042 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:06:47.615936 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c897631-8986-4fd2-bab7-afeae8584d6f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c897631-8986-4fd2-bab7-afeae8584d6f-maas-api-xks-ca\") pod \"4c897631-8986-4fd2-bab7-afeae8584d6f\" (UID: \"4c897631-8986-4fd2-bab7-afeae8584d6f\") " Oct 31 01:06:47.616042 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:06:47.616010 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c897631-8986-4fd2-bab7-afeae8584d6f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c897631-8986-4fd2-bab7-afeae8584d6f-maas-api-openshift-ca\") pod \"4c897631-8986-4fd2-bab7-afeae8584d6f\" (UID: \"4c897631-8986-4fd2-bab7-afeae8584d6f\") " Oct 31 01:06:47.616319 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:06:47.616168 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c897631-8986-4fd2-bab7-afeae8584d6f-kube-api-access-dqb58\" (UniqueName: \"kubernetes.io/projected/4c897631-8986-4fd2-bab7-afeae8584d6f-kube-api-access-dqb58\") pod \"4c897631-8986-4fd2-bab7-afeae8584d6f\" (UID: \"4c897631-8986-4fd2-bab7-afeae8584d6f\") " Oct 31 01:06:47.616442 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:06:47.616406 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c897631-8986-4fd2-bab7-afeae8584d6f-maas-api-openshift-ca" pod "4c897631-8986-4fd2-bab7-afeae8584d6f" (UID: "4c897631-8986-4fd2-bab7-afeae8584d6f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:06:47.616575 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:06:47.616545 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c897631-8986-4fd2-bab7-afeae8584d6f-maas-api-openshift-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:06:47.618317 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:06:47.618284 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c897631-8986-4fd2-bab7-afeae8584d6f-maas-api-xks-ca" pod "4c897631-8986-4fd2-bab7-afeae8584d6f" (UID: "4c897631-8986-4fd2-bab7-afeae8584d6f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:06:47.618511 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:06:47.618490 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c897631-8986-4fd2-bab7-afeae8584d6f-kube-api-access-dqb58" pod "4c897631-8986-4fd2-bab7-afeae8584d6f" (UID: "4c897631-8986-4fd2-bab7-afeae8584d6f"). InnerVolumeSpecName "kube-api-access-dqb58". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:06:47.717940 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:06:47.717905 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqb58\" (UniqueName: \"kubernetes.io/projected/4c897631-8986-4fd2-bab7-afeae8584d6f-kube-api-access-dqb58\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:06:47.717940 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:06:47.717937 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c897631-8986-4fd2-bab7-afeae8584d6f-maas-api-xks-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:06:48.342085 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:06:48.342057 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rrfmz" Oct 31 01:06:48.342273 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:06:48.342093 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rrfmz" event={"ID":"4c897631-8986-4fd2-bab7-afeae8584d6f","Type":"ContainerDied","Data":"f0e812ed69291e6f7d8d2e7f0ec9b67c93c02f8eb68e629ebbf892d6c79f15c0"} Oct 31 01:06:48.342273 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:06:48.342147 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f0e812ed69291e6f7d8d2e7f0ec9b67c93c02f8eb68e629ebbf892d6c79f15c0" Oct 31 01:06:57.221432 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:06:57.221390 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-79tk8"] Oct 31 01:07:05.890136 ip-10-0-137-56 kubenswrapper[5222]: E1031 01:07:05.890095 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 01:07:10.417677 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:07:10.417634 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-79tk8"] Oct 31 01:07:10.718279 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:07:10.718198 5222 scope.go:122] "RemoveContainer" containerID="25552adba68fe453444a71d04742ecf327b4504d7743c4ef0a2a403f58e5ade7" Oct 31 01:07:10.728907 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:07:10.728884 5222 scope.go:122] "RemoveContainer" containerID="175fd51abffffaa516b5ca835acfadf78eadb4160657d97733e36b6119d45f94" Oct 31 01:07:11.051111 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:07:11.051075 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-bxnmv"] Oct 31 01:07:11.121942 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:07:11.121900 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-bxnmv"] Oct 31 01:07:11.682395 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:07:11.681899 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9937a9f-f743-4587-9a27-88a51f33b245" path="/var/lib/kubelet/pods/b9937a9f-f743-4587-9a27-88a51f33b245/volumes" Oct 31 01:07:13.001869 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:07:13.001819 5222 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-256br container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.248:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 01:07:13.003188 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:07:13.003145 5222 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-256br" podUID="a723d3c6-4cd4-4e23-bf6b-6f2dfb4a38f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.248:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 01:07:13.104766 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:07:13.104587 5222 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-jd2w5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.250:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 01:07:13.104766 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:07:13.104678 5222 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jd2w5" podUID="d2ec2aa7-9ec8-41bd-b10f-e5ef7ecbd803" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.250:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 01:07:20.824261 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:07:20.824220 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-79tk8"] Oct 31 01:07:35.889293 ip-10-0-137-56 kubenswrapper[5222]: E1031 01:07:35.889250 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 01:07:41.524965 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:07:41.524925 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-79tk8"] Oct 31 01:07:51.613785 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:07:51.613695 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-79tk8"] Oct 31 01:08:05.890016 ip-10-0-137-56 kubenswrapper[5222]: E1031 01:08:05.889979 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 01:08:05.990145 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:08:05.990113 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-79tk8"] Oct 31 01:08:10.776503 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:08:10.776470 5222 scope.go:122] "RemoveContainer" containerID="6d9c2a05f072ef51dbe04bb013eb50f5095fade5057b3271df9d539f53283cb4" Oct 31 01:08:10.785192 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:08:10.785164 5222 scope.go:122] "RemoveContainer" containerID="1aa45fae45d595e5cb643d2f46a6dab8bd0f9f585bfffad88464e09a6bcab198" Oct 31 01:08:10.793276 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:08:10.793256 5222 scope.go:122] "RemoveContainer" containerID="9212714a7c7c4f861f19db50d851da5dbb6bab01f5758e2524c93e7832ba3bcd" Oct 31 01:08:14.825589 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:08:14.825556 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-7h8z8"] Oct 31 01:08:14.825990 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:08:14.825957 5222 cpu_manager.go:414] "containerMap: removing container" podUID="4c897631-8986-4fd2-bab7-afeae8584d6f" containerName="revoke-keys" Oct 31 01:08:14.825990 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:08:14.825969 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c897631-8986-4fd2-bab7-afeae8584d6f" containerName="revoke-keys" Oct 31 01:08:14.826066 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:08:14.826049 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c897631-8986-4fd2-bab7-afeae8584d6f" containerName="revoke-keys" Oct 31 01:08:14.830570 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:08:14.830546 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-7h8z8" Oct 31 01:08:14.868938 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:08:14.868904 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72557662-9af4-4c0a-8b2f-f8282cbd9dfb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-7h8z8\" (UID: \"72557662-9af4-4c0a-8b2f-f8282cbd9dfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-7h8z8" Oct 31 01:08:14.869124 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:08:14.868973 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mdg8\" (UniqueName: \"kubernetes.io/projected/72557662-9af4-4c0a-8b2f-f8282cbd9dfb-kube-api-access-4mdg8\") pod \"maas-api-revoke-sub-e2fc9c5b-7h8z8\" (UID: \"72557662-9af4-4c0a-8b2f-f8282cbd9dfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-7h8z8" Oct 31 01:08:14.869124 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:08:14.869093 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72557662-9af4-4c0a-8b2f-f8282cbd9dfb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-7h8z8\" (UID: \"72557662-9af4-4c0a-8b2f-f8282cbd9dfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-7h8z8" Oct 31 01:08:14.883711 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:08:14.883674 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-7h8z8"] Oct 31 01:08:14.933015 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:08:14.932963 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:08:14.933015 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:08:14.933009 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:08:14.933235 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:08:14.932981 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:08:14.933235 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:08:14.933037 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfz9z\"" Oct 31 01:08:14.970301 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:08:14.970262 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72557662-9af4-4c0a-8b2f-f8282cbd9dfb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-7h8z8\" (UID: \"72557662-9af4-4c0a-8b2f-f8282cbd9dfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-7h8z8" Oct 31 01:08:14.970515 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:08:14.970344 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72557662-9af4-4c0a-8b2f-f8282cbd9dfb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-7h8z8\" (UID: \"72557662-9af4-4c0a-8b2f-f8282cbd9dfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-7h8z8" Oct 31 01:08:14.970515 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:08:14.970417 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mdg8\" (UniqueName: \"kubernetes.io/projected/72557662-9af4-4c0a-8b2f-f8282cbd9dfb-kube-api-access-4mdg8\") pod \"maas-api-revoke-sub-e2fc9c5b-7h8z8\" (UID: \"72557662-9af4-4c0a-8b2f-f8282cbd9dfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-7h8z8" Oct 31 01:08:14.971113 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:08:14.971091 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72557662-9af4-4c0a-8b2f-f8282cbd9dfb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-7h8z8\" (UID: \"72557662-9af4-4c0a-8b2f-f8282cbd9dfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-7h8z8" Oct 31 01:08:14.972883 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:08:14.972863 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72557662-9af4-4c0a-8b2f-f8282cbd9dfb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-7h8z8\" (UID: \"72557662-9af4-4c0a-8b2f-f8282cbd9dfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-7h8z8" Oct 31 01:08:15.033700 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:08:15.033662 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mdg8\" (UniqueName: \"kubernetes.io/projected/72557662-9af4-4c0a-8b2f-f8282cbd9dfb-kube-api-access-4mdg8\") pod \"maas-api-revoke-sub-e2fc9c5b-7h8z8\" (UID: \"72557662-9af4-4c0a-8b2f-f8282cbd9dfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-7h8z8" Oct 31 01:08:15.144157 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:08:15.144071 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-7h8z8" Oct 31 01:08:15.515045 ip-10-0-137-56 kubenswrapper[5222]: W1031 01:08:15.515009 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72557662_9af4_4c0a_8b2f_f8282cbd9dfb.slice/crio-37ce8eabe70580373ed806e90ed03f0bb7d53d15fbd6eeb35dbbe6967cb80cad WatchSource:0}: Error finding container 37ce8eabe70580373ed806e90ed03f0bb7d53d15fbd6eeb35dbbe6967cb80cad: Status 404 returned error can't find the container with id 37ce8eabe70580373ed806e90ed03f0bb7d53d15fbd6eeb35dbbe6967cb80cad Oct 31 01:08:15.517081 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:08:15.516832 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-7h8z8"] Oct 31 01:08:15.872952 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:08:15.872917 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-7h8z8" event={"ID":"72557662-9af4-4c0a-8b2f-f8282cbd9dfb","Type":"ContainerStarted","Data":"e99a4fb66f0fdf8660180e4ce2a0af279bbc66375c678ac9dbf052316400221d"} Oct 31 01:08:15.873358 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:08:15.872970 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-7h8z8" event={"ID":"72557662-9af4-4c0a-8b2f-f8282cbd9dfb","Type":"ContainerStarted","Data":"37ce8eabe70580373ed806e90ed03f0bb7d53d15fbd6eeb35dbbe6967cb80cad"} Oct 31 01:08:15.992891 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:08:15.992834 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-7h8z8" podStartSLOduration=1.992819569 podStartE2EDuration="1.992819569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:08:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:08:15.991010652 +0000 UTC m=+3188.462180402" watchObservedRunningTime="2026-10-31 01:08:15.992819569 +0000 UTC m=+3188.463989317" Oct 31 01:08:16.879858 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:08:16.879830 5222 generic.go:356] "Generic (PLEG): container finished" podID="72557662-9af4-4c0a-8b2f-f8282cbd9dfb" containerID="e99a4fb66f0fdf8660180e4ce2a0af279bbc66375c678ac9dbf052316400221d" exitCode=0 Oct 31 01:08:16.880330 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:08:16.879935 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-7h8z8" event={"ID":"72557662-9af4-4c0a-8b2f-f8282cbd9dfb","Type":"ContainerDied","Data":"e99a4fb66f0fdf8660180e4ce2a0af279bbc66375c678ac9dbf052316400221d"} Oct 31 01:08:18.018402 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:08:18.018354 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-7h8z8" Oct 31 01:08:18.101658 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:08:18.101621 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72557662-9af4-4c0a-8b2f-f8282cbd9dfb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72557662-9af4-4c0a-8b2f-f8282cbd9dfb-maas-api-xks-ca\") pod \"72557662-9af4-4c0a-8b2f-f8282cbd9dfb\" (UID: \"72557662-9af4-4c0a-8b2f-f8282cbd9dfb\") " Oct 31 01:08:18.101853 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:08:18.101666 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72557662-9af4-4c0a-8b2f-f8282cbd9dfb-kube-api-access-4mdg8\" (UniqueName: \"kubernetes.io/projected/72557662-9af4-4c0a-8b2f-f8282cbd9dfb-kube-api-access-4mdg8\") pod \"72557662-9af4-4c0a-8b2f-f8282cbd9dfb\" (UID: \"72557662-9af4-4c0a-8b2f-f8282cbd9dfb\") " Oct 31 01:08:18.101853 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:08:18.101702 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72557662-9af4-4c0a-8b2f-f8282cbd9dfb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72557662-9af4-4c0a-8b2f-f8282cbd9dfb-maas-api-openshift-ca\") pod \"72557662-9af4-4c0a-8b2f-f8282cbd9dfb\" (UID: \"72557662-9af4-4c0a-8b2f-f8282cbd9dfb\") " Oct 31 01:08:18.102135 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:08:18.102106 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72557662-9af4-4c0a-8b2f-f8282cbd9dfb-maas-api-openshift-ca" pod "72557662-9af4-4c0a-8b2f-f8282cbd9dfb" (UID: "72557662-9af4-4c0a-8b2f-f8282cbd9dfb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:08:18.103993 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:08:18.103970 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72557662-9af4-4c0a-8b2f-f8282cbd9dfb-maas-api-xks-ca" pod "72557662-9af4-4c0a-8b2f-f8282cbd9dfb" (UID: "72557662-9af4-4c0a-8b2f-f8282cbd9dfb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:08:18.104055 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:08:18.104012 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72557662-9af4-4c0a-8b2f-f8282cbd9dfb-kube-api-access-4mdg8" pod "72557662-9af4-4c0a-8b2f-f8282cbd9dfb" (UID: "72557662-9af4-4c0a-8b2f-f8282cbd9dfb"). InnerVolumeSpecName "kube-api-access-4mdg8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:08:18.203675 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:08:18.203581 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72557662-9af4-4c0a-8b2f-f8282cbd9dfb-maas-api-xks-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:08:18.203675 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:08:18.203615 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mdg8\" (UniqueName: \"kubernetes.io/projected/72557662-9af4-4c0a-8b2f-f8282cbd9dfb-kube-api-access-4mdg8\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:08:18.203675 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:08:18.203628 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72557662-9af4-4c0a-8b2f-f8282cbd9dfb-maas-api-openshift-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:08:18.891589 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:08:18.891561 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-7h8z8" Oct 31 01:08:18.891770 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:08:18.891563 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-7h8z8" event={"ID":"72557662-9af4-4c0a-8b2f-f8282cbd9dfb","Type":"ContainerDied","Data":"37ce8eabe70580373ed806e90ed03f0bb7d53d15fbd6eeb35dbbe6967cb80cad"} Oct 31 01:08:18.891770 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:08:18.891687 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="37ce8eabe70580373ed806e90ed03f0bb7d53d15fbd6eeb35dbbe6967cb80cad" Oct 31 01:08:20.062809 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:08:20.062770 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-drj9f"] Oct 31 01:08:20.114101 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:08:20.114063 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-drj9f"] Oct 31 01:08:21.654744 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:08:21.654698 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20110920-0e40-41b2-9d39-fef24408d6c1" path="/var/lib/kubelet/pods/20110920-0e40-41b2-9d39-fef24408d6c1/volumes" Oct 31 01:08:27.591774 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:08:27.591738 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-79tk8"] Oct 31 01:08:30.553577 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:08:30.553519 5222 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-zq2lm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.252:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 01:08:30.554275 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:08:30.554236 5222 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-zq2lm" podUID="3d1be84f-73f0-4f54-ae9f-6d5f38367862" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.252:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 01:08:35.890107 ip-10-0-137-56 kubenswrapper[5222]: E1031 01:08:35.890067 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 01:08:38.386560 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:08:38.386523 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-79tk8"] Oct 31 01:08:43.142727 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:08:43.142690 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2k4fm"] Oct 31 01:08:43.143121 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:08:43.143102 5222 cpu_manager.go:414] "containerMap: removing container" podUID="72557662-9af4-4c0a-8b2f-f8282cbd9dfb" containerName="revoke-keys" Oct 31 01:08:43.143121 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:08:43.143113 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72557662-9af4-4c0a-8b2f-f8282cbd9dfb" containerName="revoke-keys" Oct 31 01:08:43.143204 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:08:43.143183 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="72557662-9af4-4c0a-8b2f-f8282cbd9dfb" containerName="revoke-keys" Oct 31 01:08:43.146258 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:08:43.146241 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2k4fm" Oct 31 01:08:43.194736 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:08:43.194698 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c891896-6776-4453-b25e-5b7cca46548a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-2k4fm\" (UID: \"2c891896-6776-4453-b25e-5b7cca46548a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2k4fm" Oct 31 01:08:43.195107 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:08:43.195080 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpgnb\" (UniqueName: \"kubernetes.io/projected/2c891896-6776-4453-b25e-5b7cca46548a-kube-api-access-tpgnb\") pod \"maas-api-revoke-sub-3d4156f1-2k4fm\" (UID: \"2c891896-6776-4453-b25e-5b7cca46548a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2k4fm" Oct 31 01:08:43.195239 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:08:43.195122 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c891896-6776-4453-b25e-5b7cca46548a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-2k4fm\" (UID: \"2c891896-6776-4453-b25e-5b7cca46548a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2k4fm" Oct 31 01:08:43.200589 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:08:43.200556 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2k4fm"] Oct 31 01:08:43.249806 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:08:43.249769 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:08:43.249806 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:08:43.249768 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfz9z\"" Oct 31 01:08:43.249806 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:08:43.249803 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:08:43.251556 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:08:43.251534 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:08:43.296001 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:08:43.295962 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c891896-6776-4453-b25e-5b7cca46548a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-2k4fm\" (UID: \"2c891896-6776-4453-b25e-5b7cca46548a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2k4fm" Oct 31 01:08:43.296201 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:08:43.296107 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpgnb\" (UniqueName: \"kubernetes.io/projected/2c891896-6776-4453-b25e-5b7cca46548a-kube-api-access-tpgnb\") pod \"maas-api-revoke-sub-3d4156f1-2k4fm\" (UID: \"2c891896-6776-4453-b25e-5b7cca46548a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2k4fm" Oct 31 01:08:43.296201 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:08:43.296138 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c891896-6776-4453-b25e-5b7cca46548a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-2k4fm\" (UID: \"2c891896-6776-4453-b25e-5b7cca46548a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2k4fm" Oct 31 01:08:43.296921 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:08:43.296861 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c891896-6776-4453-b25e-5b7cca46548a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-2k4fm\" (UID: \"2c891896-6776-4453-b25e-5b7cca46548a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2k4fm" Oct 31 01:08:43.298635 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:08:43.298610 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c891896-6776-4453-b25e-5b7cca46548a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-2k4fm\" (UID: \"2c891896-6776-4453-b25e-5b7cca46548a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2k4fm" Oct 31 01:08:43.359499 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:08:43.359465 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpgnb\" (UniqueName: \"kubernetes.io/projected/2c891896-6776-4453-b25e-5b7cca46548a-kube-api-access-tpgnb\") pod \"maas-api-revoke-sub-3d4156f1-2k4fm\" (UID: \"2c891896-6776-4453-b25e-5b7cca46548a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2k4fm" Oct 31 01:08:43.459520 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:08:43.459435 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2k4fm" Oct 31 01:08:43.833967 ip-10-0-137-56 kubenswrapper[5222]: W1031 01:08:43.833930 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c891896_6776_4453_b25e_5b7cca46548a.slice/crio-1dc86c4601c3e409d5b0b6b19c9f33e2455cd5cc8b473050c816ac333f751304 WatchSource:0}: Error finding container 1dc86c4601c3e409d5b0b6b19c9f33e2455cd5cc8b473050c816ac333f751304: Status 404 returned error can't find the container with id 1dc86c4601c3e409d5b0b6b19c9f33e2455cd5cc8b473050c816ac333f751304 Oct 31 01:08:43.837096 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:08:43.837070 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2k4fm"] Oct 31 01:08:44.058263 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:08:44.058228 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2k4fm" event={"ID":"2c891896-6776-4453-b25e-5b7cca46548a","Type":"ContainerStarted","Data":"1dc86c4601c3e409d5b0b6b19c9f33e2455cd5cc8b473050c816ac333f751304"} Oct 31 01:08:45.073207 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:08:45.073166 5222 generic.go:356] "Generic (PLEG): container finished" podID="2c891896-6776-4453-b25e-5b7cca46548a" containerID="5f0d1958915d1347e9d896f70b0c1e0a0f9c442c2f3cf3418a40c331e6c81397" exitCode=0 Oct 31 01:08:45.073645 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:08:45.073254 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2k4fm" event={"ID":"2c891896-6776-4453-b25e-5b7cca46548a","Type":"ContainerDied","Data":"5f0d1958915d1347e9d896f70b0c1e0a0f9c442c2f3cf3418a40c331e6c81397"} Oct 31 01:08:46.213153 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:08:46.213126 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2k4fm" Oct 31 01:08:46.326894 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:08:46.326847 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c891896-6776-4453-b25e-5b7cca46548a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c891896-6776-4453-b25e-5b7cca46548a-maas-api-xks-ca\") pod \"2c891896-6776-4453-b25e-5b7cca46548a\" (UID: \"2c891896-6776-4453-b25e-5b7cca46548a\") " Oct 31 01:08:46.327119 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:08:46.326933 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c891896-6776-4453-b25e-5b7cca46548a-kube-api-access-tpgnb\" (UniqueName: \"kubernetes.io/projected/2c891896-6776-4453-b25e-5b7cca46548a-kube-api-access-tpgnb\") pod \"2c891896-6776-4453-b25e-5b7cca46548a\" (UID: \"2c891896-6776-4453-b25e-5b7cca46548a\") " Oct 31 01:08:46.327119 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:08:46.327027 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c891896-6776-4453-b25e-5b7cca46548a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c891896-6776-4453-b25e-5b7cca46548a-maas-api-openshift-ca\") pod \"2c891896-6776-4453-b25e-5b7cca46548a\" (UID: \"2c891896-6776-4453-b25e-5b7cca46548a\") " Oct 31 01:08:46.327694 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:08:46.327664 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c891896-6776-4453-b25e-5b7cca46548a-maas-api-openshift-ca" pod "2c891896-6776-4453-b25e-5b7cca46548a" (UID: "2c891896-6776-4453-b25e-5b7cca46548a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:08:46.329824 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:08:46.329764 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c891896-6776-4453-b25e-5b7cca46548a-maas-api-xks-ca" pod "2c891896-6776-4453-b25e-5b7cca46548a" (UID: "2c891896-6776-4453-b25e-5b7cca46548a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:08:46.329824 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:08:46.329782 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c891896-6776-4453-b25e-5b7cca46548a-kube-api-access-tpgnb" pod "2c891896-6776-4453-b25e-5b7cca46548a" (UID: "2c891896-6776-4453-b25e-5b7cca46548a"). InnerVolumeSpecName "kube-api-access-tpgnb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:08:46.428511 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:08:46.428464 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpgnb\" (UniqueName: \"kubernetes.io/projected/2c891896-6776-4453-b25e-5b7cca46548a-kube-api-access-tpgnb\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:08:46.428511 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:08:46.428505 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c891896-6776-4453-b25e-5b7cca46548a-maas-api-openshift-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:08:46.428511 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:08:46.428516 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c891896-6776-4453-b25e-5b7cca46548a-maas-api-xks-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:08:47.084541 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:08:47.084510 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2k4fm" Oct 31 01:08:47.084726 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:08:47.084540 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2k4fm" event={"ID":"2c891896-6776-4453-b25e-5b7cca46548a","Type":"ContainerDied","Data":"1dc86c4601c3e409d5b0b6b19c9f33e2455cd5cc8b473050c816ac333f751304"} Oct 31 01:08:47.084726 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:08:47.084587 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1dc86c4601c3e409d5b0b6b19c9f33e2455cd5cc8b473050c816ac333f751304" Oct 31 01:08:51.050961 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:08:51.050919 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9bft5"] Oct 31 01:08:51.102019 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:08:51.101987 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9bft5"] Oct 31 01:08:51.653702 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:08:51.653667 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="800ff4e9-fe1c-403d-97b8-787dd370a036" path="/var/lib/kubelet/pods/800ff4e9-fe1c-403d-97b8-787dd370a036/volumes" Oct 31 01:08:57.623266 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:08:57.623195 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-79tk8"] Oct 31 01:09:04.050004 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:09:04.049964 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-7bxgn"] Oct 31 01:09:04.101267 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:09:04.101228 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-7bxgn"] Oct 31 01:09:05.654768 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:09:05.654728 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="884d98fd-e09a-480f-8195-07102988e30f" path="/var/lib/kubelet/pods/884d98fd-e09a-480f-8195-07102988e30f/volumes" Oct 31 01:09:05.890401 ip-10-0-137-56 kubenswrapper[5222]: E1031 01:09:05.890340 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 01:09:07.793751 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:09:07.793705 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-79tk8"] Oct 31 01:09:15.625926 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:09:15.625838 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-863fb0-openshift-default-86c56576ccdwkvd"] Oct 31 01:09:15.626540 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:09:15.626496 5222 cpu_manager.go:414] "containerMap: removing container" podUID="2c891896-6776-4453-b25e-5b7cca46548a" containerName="revoke-keys" Oct 31 01:09:15.626540 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:09:15.626517 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c891896-6776-4453-b25e-5b7cca46548a" containerName="revoke-keys" Oct 31 01:09:15.626680 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:09:15.626647 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c891896-6776-4453-b25e-5b7cca46548a" containerName="revoke-keys" Oct 31 01:09:15.631046 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:09:15.631020 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-863fb0-openshift-default-86c56576ccdwkvd" Oct 31 01:09:15.740417 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:09:15.737941 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-863fb0-openshift-default-dockercfg-mhcq9\"" Oct 31 01:09:15.741117 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:09:15.741085 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-863fb0-openshift-default-86c56576ccdwkvd"] Oct 31 01:09:15.788073 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:09:15.788039 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17f59d3b-a204-4e4a-9944-1e1069424e8b-istiod-ca-cert\") pod \"e2e-worker-single-main-863fb0-openshift-default-86c56576ccdwkvd\" (UID: \"17f59d3b-a204-4e4a-9944-1e1069424e8b\") " pod="openshift-ingress/e2e-worker-single-main-863fb0-openshift-default-86c56576ccdwkvd" Oct 31 01:09:15.788266 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:09:15.788110 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17f59d3b-a204-4e4a-9944-1e1069424e8b-workload-socket\") pod \"e2e-worker-single-main-863fb0-openshift-default-86c56576ccdwkvd\" (UID: \"17f59d3b-a204-4e4a-9944-1e1069424e8b\") " pod="openshift-ingress/e2e-worker-single-main-863fb0-openshift-default-86c56576ccdwkvd" Oct 31 01:09:15.788266 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:09:15.788163 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17f59d3b-a204-4e4a-9944-1e1069424e8b-istio-envoy\") pod \"e2e-worker-single-main-863fb0-openshift-default-86c56576ccdwkvd\" (UID: \"17f59d3b-a204-4e4a-9944-1e1069424e8b\") " pod="openshift-ingress/e2e-worker-single-main-863fb0-openshift-default-86c56576ccdwkvd" Oct 31 01:09:15.788266 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:09:15.788246 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17f59d3b-a204-4e4a-9944-1e1069424e8b-istio-podinfo\") pod \"e2e-worker-single-main-863fb0-openshift-default-86c56576ccdwkvd\" (UID: \"17f59d3b-a204-4e4a-9944-1e1069424e8b\") " pod="openshift-ingress/e2e-worker-single-main-863fb0-openshift-default-86c56576ccdwkvd" Oct 31 01:09:15.788433 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:09:15.788292 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k57v5\" (UniqueName: \"kubernetes.io/projected/17f59d3b-a204-4e4a-9944-1e1069424e8b-kube-api-access-k57v5\") pod \"e2e-worker-single-main-863fb0-openshift-default-86c56576ccdwkvd\" (UID: \"17f59d3b-a204-4e4a-9944-1e1069424e8b\") " pod="openshift-ingress/e2e-worker-single-main-863fb0-openshift-default-86c56576ccdwkvd" Oct 31 01:09:15.788433 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:09:15.788334 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17f59d3b-a204-4e4a-9944-1e1069424e8b-istio-data\") pod \"e2e-worker-single-main-863fb0-openshift-default-86c56576ccdwkvd\" (UID: \"17f59d3b-a204-4e4a-9944-1e1069424e8b\") " pod="openshift-ingress/e2e-worker-single-main-863fb0-openshift-default-86c56576ccdwkvd" Oct 31 01:09:15.788433 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:09:15.788413 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17f59d3b-a204-4e4a-9944-1e1069424e8b-credential-socket\") pod \"e2e-worker-single-main-863fb0-openshift-default-86c56576ccdwkvd\" (UID: \"17f59d3b-a204-4e4a-9944-1e1069424e8b\") " pod="openshift-ingress/e2e-worker-single-main-863fb0-openshift-default-86c56576ccdwkvd" Oct 31 01:09:15.788550 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:09:15.788456 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17f59d3b-a204-4e4a-9944-1e1069424e8b-istio-token\") pod \"e2e-worker-single-main-863fb0-openshift-default-86c56576ccdwkvd\" (UID: \"17f59d3b-a204-4e4a-9944-1e1069424e8b\") " pod="openshift-ingress/e2e-worker-single-main-863fb0-openshift-default-86c56576ccdwkvd" Oct 31 01:09:15.788550 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:09:15.788539 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17f59d3b-a204-4e4a-9944-1e1069424e8b-workload-certs\") pod \"e2e-worker-single-main-863fb0-openshift-default-86c56576ccdwkvd\" (UID: \"17f59d3b-a204-4e4a-9944-1e1069424e8b\") " pod="openshift-ingress/e2e-worker-single-main-863fb0-openshift-default-86c56576ccdwkvd" Oct 31 01:09:15.889503 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:09:15.889410 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17f59d3b-a204-4e4a-9944-1e1069424e8b-istiod-ca-cert\") pod \"e2e-worker-single-main-863fb0-openshift-default-86c56576ccdwkvd\" (UID: \"17f59d3b-a204-4e4a-9944-1e1069424e8b\") " pod="openshift-ingress/e2e-worker-single-main-863fb0-openshift-default-86c56576ccdwkvd" Oct 31 01:09:15.889503 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:09:15.889491 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17f59d3b-a204-4e4a-9944-1e1069424e8b-workload-socket\") pod \"e2e-worker-single-main-863fb0-openshift-default-86c56576ccdwkvd\" (UID: \"17f59d3b-a204-4e4a-9944-1e1069424e8b\") " pod="openshift-ingress/e2e-worker-single-main-863fb0-openshift-default-86c56576ccdwkvd" Oct 31 01:09:15.889750 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:09:15.889507 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17f59d3b-a204-4e4a-9944-1e1069424e8b-istio-envoy\") pod \"e2e-worker-single-main-863fb0-openshift-default-86c56576ccdwkvd\" (UID: \"17f59d3b-a204-4e4a-9944-1e1069424e8b\") " pod="openshift-ingress/e2e-worker-single-main-863fb0-openshift-default-86c56576ccdwkvd" Oct 31 01:09:15.889750 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:09:15.889533 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17f59d3b-a204-4e4a-9944-1e1069424e8b-istio-podinfo\") pod \"e2e-worker-single-main-863fb0-openshift-default-86c56576ccdwkvd\" (UID: \"17f59d3b-a204-4e4a-9944-1e1069424e8b\") " pod="openshift-ingress/e2e-worker-single-main-863fb0-openshift-default-86c56576ccdwkvd" Oct 31 01:09:15.889750 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:09:15.889552 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k57v5\" (UniqueName: \"kubernetes.io/projected/17f59d3b-a204-4e4a-9944-1e1069424e8b-kube-api-access-k57v5\") pod \"e2e-worker-single-main-863fb0-openshift-default-86c56576ccdwkvd\" (UID: \"17f59d3b-a204-4e4a-9944-1e1069424e8b\") " pod="openshift-ingress/e2e-worker-single-main-863fb0-openshift-default-86c56576ccdwkvd" Oct 31 01:09:15.889750 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:09:15.889575 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17f59d3b-a204-4e4a-9944-1e1069424e8b-istio-data\") pod \"e2e-worker-single-main-863fb0-openshift-default-86c56576ccdwkvd\" (UID: \"17f59d3b-a204-4e4a-9944-1e1069424e8b\") " pod="openshift-ingress/e2e-worker-single-main-863fb0-openshift-default-86c56576ccdwkvd" Oct 31 01:09:15.889750 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:09:15.889602 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17f59d3b-a204-4e4a-9944-1e1069424e8b-credential-socket\") pod \"e2e-worker-single-main-863fb0-openshift-default-86c56576ccdwkvd\" (UID: \"17f59d3b-a204-4e4a-9944-1e1069424e8b\") " pod="openshift-ingress/e2e-worker-single-main-863fb0-openshift-default-86c56576ccdwkvd" Oct 31 01:09:15.889750 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:09:15.889635 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17f59d3b-a204-4e4a-9944-1e1069424e8b-istio-token\") pod \"e2e-worker-single-main-863fb0-openshift-default-86c56576ccdwkvd\" (UID: \"17f59d3b-a204-4e4a-9944-1e1069424e8b\") " pod="openshift-ingress/e2e-worker-single-main-863fb0-openshift-default-86c56576ccdwkvd" Oct 31 01:09:15.889750 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:09:15.889706 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17f59d3b-a204-4e4a-9944-1e1069424e8b-workload-certs\") pod \"e2e-worker-single-main-863fb0-openshift-default-86c56576ccdwkvd\" (UID: \"17f59d3b-a204-4e4a-9944-1e1069424e8b\") " pod="openshift-ingress/e2e-worker-single-main-863fb0-openshift-default-86c56576ccdwkvd" Oct 31 01:09:15.890131 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:09:15.890097 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17f59d3b-a204-4e4a-9944-1e1069424e8b-istio-data\") pod \"e2e-worker-single-main-863fb0-openshift-default-86c56576ccdwkvd\" (UID: \"17f59d3b-a204-4e4a-9944-1e1069424e8b\") " pod="openshift-ingress/e2e-worker-single-main-863fb0-openshift-default-86c56576ccdwkvd" Oct 31 01:09:15.890258 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:09:15.890229 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17f59d3b-a204-4e4a-9944-1e1069424e8b-credential-socket\") pod \"e2e-worker-single-main-863fb0-openshift-default-86c56576ccdwkvd\" (UID: \"17f59d3b-a204-4e4a-9944-1e1069424e8b\") " pod="openshift-ingress/e2e-worker-single-main-863fb0-openshift-default-86c56576ccdwkvd" Oct 31 01:09:15.890312 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:09:15.890240 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17f59d3b-a204-4e4a-9944-1e1069424e8b-istiod-ca-cert\") pod \"e2e-worker-single-main-863fb0-openshift-default-86c56576ccdwkvd\" (UID: \"17f59d3b-a204-4e4a-9944-1e1069424e8b\") " pod="openshift-ingress/e2e-worker-single-main-863fb0-openshift-default-86c56576ccdwkvd" Oct 31 01:09:15.890312 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:09:15.890271 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17f59d3b-a204-4e4a-9944-1e1069424e8b-workload-certs\") pod \"e2e-worker-single-main-863fb0-openshift-default-86c56576ccdwkvd\" (UID: \"17f59d3b-a204-4e4a-9944-1e1069424e8b\") " pod="openshift-ingress/e2e-worker-single-main-863fb0-openshift-default-86c56576ccdwkvd" Oct 31 01:09:15.890416 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:09:15.890339 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17f59d3b-a204-4e4a-9944-1e1069424e8b-workload-socket\") pod \"e2e-worker-single-main-863fb0-openshift-default-86c56576ccdwkvd\" (UID: \"17f59d3b-a204-4e4a-9944-1e1069424e8b\") " pod="openshift-ingress/e2e-worker-single-main-863fb0-openshift-default-86c56576ccdwkvd" Oct 31 01:09:15.892316 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:09:15.892289 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17f59d3b-a204-4e4a-9944-1e1069424e8b-istio-envoy\") pod \"e2e-worker-single-main-863fb0-openshift-default-86c56576ccdwkvd\" (UID: \"17f59d3b-a204-4e4a-9944-1e1069424e8b\") " pod="openshift-ingress/e2e-worker-single-main-863fb0-openshift-default-86c56576ccdwkvd" Oct 31 01:09:15.892501 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:09:15.892482 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17f59d3b-a204-4e4a-9944-1e1069424e8b-istio-podinfo\") pod \"e2e-worker-single-main-863fb0-openshift-default-86c56576ccdwkvd\" (UID: \"17f59d3b-a204-4e4a-9944-1e1069424e8b\") " pod="openshift-ingress/e2e-worker-single-main-863fb0-openshift-default-86c56576ccdwkvd" Oct 31 01:09:15.952760 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:09:15.952725 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17f59d3b-a204-4e4a-9944-1e1069424e8b-istio-token\") pod \"e2e-worker-single-main-863fb0-openshift-default-86c56576ccdwkvd\" (UID: \"17f59d3b-a204-4e4a-9944-1e1069424e8b\") " pod="openshift-ingress/e2e-worker-single-main-863fb0-openshift-default-86c56576ccdwkvd" Oct 31 01:09:15.952927 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:09:15.952728 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k57v5\" (UniqueName: \"kubernetes.io/projected/17f59d3b-a204-4e4a-9944-1e1069424e8b-kube-api-access-k57v5\") pod \"e2e-worker-single-main-863fb0-openshift-default-86c56576ccdwkvd\" (UID: \"17f59d3b-a204-4e4a-9944-1e1069424e8b\") " pod="openshift-ingress/e2e-worker-single-main-863fb0-openshift-default-86c56576ccdwkvd" Oct 31 01:09:16.243819 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:09:16.243719 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-863fb0-openshift-default-86c56576ccdwkvd" Oct 31 01:09:16.690580 ip-10-0-137-56 kubenswrapper[5222]: W1031 01:09:16.690545 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17f59d3b_a204_4e4a_9944_1e1069424e8b.slice/crio-c06591f8b55a4337af12d3db2de7175b98e307eb03d9405ec1f22823d87eba72 WatchSource:0}: Error finding container c06591f8b55a4337af12d3db2de7175b98e307eb03d9405ec1f22823d87eba72: Status 404 returned error can't find the container with id c06591f8b55a4337af12d3db2de7175b98e307eb03d9405ec1f22823d87eba72 Oct 31 01:09:16.692958 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:09:16.692931 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-863fb0-openshift-default-86c56576ccdwkvd"] Oct 31 01:09:16.693189 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:09:16.693163 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 01:09:16.693240 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:09:16.693231 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 01:09:16.693276 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:09:16.693261 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 01:09:17.274002 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:09:17.273961 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-863fb0-openshift-default-86c56576ccdwkvd" event={"ID":"17f59d3b-a204-4e4a-9944-1e1069424e8b","Type":"ContainerStarted","Data":"ee4bb99722a32245c0b22b409b764b4bad8e09c14a2427ecab7a278dbd42788e"} Oct 31 01:09:17.274214 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:09:17.274023 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-863fb0-openshift-default-86c56576ccdwkvd" event={"ID":"17f59d3b-a204-4e4a-9944-1e1069424e8b","Type":"ContainerStarted","Data":"c06591f8b55a4337af12d3db2de7175b98e307eb03d9405ec1f22823d87eba72"} Oct 31 01:09:17.402783 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:09:17.402708 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-863fb0-openshift-default-86c56576ccdwkvd" podStartSLOduration=2.402692645 podStartE2EDuration="2.402692645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:09:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:09:17.402013642 +0000 UTC m=+3249.873183391" watchObservedRunningTime="2026-10-31 01:09:17.402692645 +0000 UTC m=+3249.873862393" Oct 31 01:09:18.244116 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:09:18.244079 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-863fb0-openshift-default-86c56576ccdwkvd" Oct 31 01:09:18.249155 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:09:18.249131 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-863fb0-openshift-default-86c56576ccdwkvd" Oct 31 01:09:18.281241 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:09:18.281214 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-863fb0-openshift-default-86c56576ccdwkvd" Oct 31 01:09:18.282755 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:09:18.282720 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-863fb0-openshift-default-86c56576ccdwkvd" Oct 31 01:09:24.231043 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:09:24.231003 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-863fb0-5ff7d48b7sg7fr"] Oct 31 01:09:24.235116 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:09:24.235094 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-863fb0-5ff7d48b7sg7fr" Oct 31 01:09:24.286780 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:09:24.286730 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtfnz\" (UniqueName: \"kubernetes.io/projected/e0027499-1395-45c9-93e3-071127a25545-kube-api-access-qtfnz\") pod \"payload-processing-e2e-worker-single-main-863fb0-5ff7d48b7sg7fr\" (UID: \"e0027499-1395-45c9-93e3-071127a25545\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-863fb0-5ff7d48b7sg7fr" Oct 31 01:09:24.286977 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:09:24.286863 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0027499-1395-45c9-93e3-071127a25545-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-863fb0-5ff7d48b7sg7fr\" (UID: \"e0027499-1395-45c9-93e3-071127a25545\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-863fb0-5ff7d48b7sg7fr" Oct 31 01:09:24.289611 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:09:24.289581 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-863fb0-5ff7d48b7sg7fr"] Oct 31 01:09:24.337968 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:09:24.337937 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-863fb0\"" Oct 31 01:09:24.338163 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:09:24.338047 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-3d7bfc50-dockercfg-2q6l6\"" Oct 31 01:09:24.388404 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:09:24.388334 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtfnz\" (UniqueName: \"kubernetes.io/projected/e0027499-1395-45c9-93e3-071127a25545-kube-api-access-qtfnz\") pod \"payload-processing-e2e-worker-single-main-863fb0-5ff7d48b7sg7fr\" (UID: \"e0027499-1395-45c9-93e3-071127a25545\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-863fb0-5ff7d48b7sg7fr" Oct 31 01:09:24.388611 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:09:24.388475 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0027499-1395-45c9-93e3-071127a25545-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-863fb0-5ff7d48b7sg7fr\" (UID: \"e0027499-1395-45c9-93e3-071127a25545\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-863fb0-5ff7d48b7sg7fr" Oct 31 01:09:24.389218 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:09:24.389198 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0027499-1395-45c9-93e3-071127a25545-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-863fb0-5ff7d48b7sg7fr\" (UID: \"e0027499-1395-45c9-93e3-071127a25545\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-863fb0-5ff7d48b7sg7fr" Oct 31 01:09:24.451337 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:09:24.451297 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtfnz\" (UniqueName: \"kubernetes.io/projected/e0027499-1395-45c9-93e3-071127a25545-kube-api-access-qtfnz\") pod \"payload-processing-e2e-worker-single-main-863fb0-5ff7d48b7sg7fr\" (UID: \"e0027499-1395-45c9-93e3-071127a25545\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-863fb0-5ff7d48b7sg7fr" Oct 31 01:09:24.548700 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:09:24.548650 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-863fb0-5ff7d48b7sg7fr" Oct 31 01:09:24.925742 ip-10-0-137-56 kubenswrapper[5222]: W1031 01:09:24.925706 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0027499_1395_45c9_93e3_071127a25545.slice/crio-d6573f84b65941b99274d3a739c794077738421c397d56213e9f8b2417f1ba30 WatchSource:0}: Error finding container d6573f84b65941b99274d3a739c794077738421c397d56213e9f8b2417f1ba30: Status 404 returned error can't find the container with id d6573f84b65941b99274d3a739c794077738421c397d56213e9f8b2417f1ba30 Oct 31 01:09:24.925742 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:09:24.925711 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-863fb0-5ff7d48b7sg7fr"] Oct 31 01:09:25.318718 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:09:25.318675 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-863fb0-5ff7d48b7sg7fr" event={"ID":"e0027499-1395-45c9-93e3-071127a25545","Type":"ContainerStarted","Data":"12e8ab4119723261e1c64584349d0d1f8937a7d691bad807342a236c2722c04c"} Oct 31 01:09:25.319143 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:09:25.318758 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-863fb0-5ff7d48b7sg7fr" event={"ID":"e0027499-1395-45c9-93e3-071127a25545","Type":"ContainerStarted","Data":"d6573f84b65941b99274d3a739c794077738421c397d56213e9f8b2417f1ba30"} Oct 31 01:09:25.319143 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:09:25.318819 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-863fb0-5ff7d48b7sg7fr" Oct 31 01:09:25.439700 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:09:25.439638 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-863fb0-5ff7d48b7sg7fr" podStartSLOduration=1.439623291 podStartE2EDuration="1.439623291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:09:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:09:25.437419024 +0000 UTC m=+3257.908588772" watchObservedRunningTime="2026-10-31 01:09:25.439623291 +0000 UTC m=+3257.910793040" Oct 31 01:09:35.293276 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:09:35.293239 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-79tk8"] Oct 31 01:09:35.890490 ip-10-0-137-56 kubenswrapper[5222]: E1031 01:09:35.890455 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 01:09:36.326645 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:09:36.326617 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-863fb0-5ff7d48b7sg7fr" Oct 31 01:09:39.057873 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:09:39.057835 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-49nqr"] Oct 31 01:09:39.107926 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:09:39.107882 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-49nqr"] Oct 31 01:09:39.665651 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:09:39.665609 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="564df912-a1a2-45a2-b8a5-a3730da64a41" path="/var/lib/kubelet/pods/564df912-a1a2-45a2-b8a5-a3730da64a41/volumes" Oct 31 01:09:45.419615 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:09:45.419573 5222 generic.go:356] "Generic (PLEG): container finished" podID="17f59d3b-a204-4e4a-9944-1e1069424e8b" containerID="ee4bb99722a32245c0b22b409b764b4bad8e09c14a2427ecab7a278dbd42788e" exitCode=0 Oct 31 01:09:45.420404 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:09:45.420346 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-863fb0-openshift-default-86c56576ccdwkvd" event={"ID":"17f59d3b-a204-4e4a-9944-1e1069424e8b","Type":"ContainerDied","Data":"ee4bb99722a32245c0b22b409b764b4bad8e09c14a2427ecab7a278dbd42788e"} Oct 31 01:09:45.421105 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:09:45.421082 5222 scope.go:122] "RemoveContainer" containerID="ee4bb99722a32245c0b22b409b764b4bad8e09c14a2427ecab7a278dbd42788e" Oct 31 01:09:45.422919 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:09:45.422883 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 01:09:45.422991 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:09:45.422969 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 01:09:45.423033 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:09:45.423011 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 01:09:46.427976 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:09:46.427943 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-863fb0-openshift-default-86c56576ccdwkvd" event={"ID":"17f59d3b-a204-4e4a-9944-1e1069424e8b","Type":"ContainerStarted","Data":"8df2079b48acf7e095e6aca4145c51fd457b8b6e3b6e075a803bf2c1c1c934a4"} Oct 31 01:09:47.254438 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:09:47.253947 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-863fb0-openshift-default-86c56576ccdwkvd" Oct 31 01:09:47.254438 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:09:47.254094 5222 patch_prober.go:29] interesting pod/e2e-worker-single-main-863fb0-openshift-default-86c56576ccdwkvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.6:15021/healthz/ready\": dial tcp 10.134.1.6:15021: connect: connection refused" start-of-body= Oct 31 01:09:47.254438 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:09:47.254156 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-863fb0-openshift-default-86c56576ccdwkvd" podUID="17f59d3b-a204-4e4a-9944-1e1069424e8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.6:15021/healthz/ready\": dial tcp 10.134.1.6:15021: connect: connection refused" Oct 31 01:09:48.254558 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:09:48.254517 5222 patch_prober.go:29] interesting pod/e2e-worker-single-main-863fb0-openshift-default-86c56576ccdwkvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.6:15021/healthz/ready\": dial tcp 10.134.1.6:15021: connect: connection refused" start-of-body= Oct 31 01:09:48.255206 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:09:48.255172 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-863fb0-openshift-default-86c56576ccdwkvd" podUID="17f59d3b-a204-4e4a-9944-1e1069424e8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.6:15021/healthz/ready\": dial tcp 10.134.1.6:15021: connect: connection refused" Oct 31 01:09:48.281041 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:09:48.281001 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-863fb0-openshift-default-86c56576ccdwkvd" Oct 31 01:09:49.252396 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:09:49.244850 5222 patch_prober.go:29] interesting pod/e2e-worker-single-main-863fb0-openshift-default-86c56576ccdwkvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.6:15021/healthz/ready\": dial tcp 10.134.1.6:15021: connect: connection refused" start-of-body= Oct 31 01:09:49.252396 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:09:49.244930 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-863fb0-openshift-default-86c56576ccdwkvd" podUID="17f59d3b-a204-4e4a-9944-1e1069424e8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.6:15021/healthz/ready\": dial tcp 10.134.1.6:15021: connect: connection refused" Oct 31 01:09:50.252411 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:09:50.251818 5222 patch_prober.go:29] interesting pod/e2e-worker-single-main-863fb0-openshift-default-86c56576ccdwkvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.6:15021/healthz/ready\": dial tcp 10.134.1.6:15021: connect: connection refused" start-of-body= Oct 31 01:09:50.252411 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:09:50.251888 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-863fb0-openshift-default-86c56576ccdwkvd" podUID="17f59d3b-a204-4e4a-9944-1e1069424e8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.6:15021/healthz/ready\": dial tcp 10.134.1.6:15021: connect: connection refused" Oct 31 01:09:51.251826 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:09:51.251783 5222 patch_prober.go:29] interesting pod/e2e-worker-single-main-863fb0-openshift-default-86c56576ccdwkvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.6:15021/healthz/ready\": dial tcp 10.134.1.6:15021: connect: connection refused" start-of-body= Oct 31 01:09:51.252145 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:09:51.252114 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-863fb0-openshift-default-86c56576ccdwkvd" podUID="17f59d3b-a204-4e4a-9944-1e1069424e8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.6:15021/healthz/ready\": dial tcp 10.134.1.6:15021: connect: connection refused" Oct 31 01:09:52.248613 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:09:52.248581 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-863fb0-openshift-default-86c56576ccdwkvd" Oct 31 01:09:52.249690 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:09:52.249670 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-863fb0-openshift-default-86c56576ccdwkvd" Oct 31 01:09:53.286815 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:09:53.286776 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-79tk8"] Oct 31 01:10:05.890114 ip-10-0-137-56 kubenswrapper[5222]: E1031 01:10:05.890072 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 01:10:08.273750 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:10:08.273722 5222 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h9hsf_0f8c3c03-a1fc-4376-98c6-feeeefa08068/ovn-acl-logging/0.log" Oct 31 01:10:08.301584 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:10:08.301558 5222 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h9hsf_0f8c3c03-a1fc-4376-98c6-feeeefa08068/ovn-acl-logging/0.log" Oct 31 01:10:10.829747 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:10:10.829707 5222 scope.go:122] "RemoveContainer" containerID="0d93d2ddd041eddc0317ae881a545431ba5198f5b7e012b2b911bf12b723feb4" Oct 31 01:10:10.838614 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:10:10.838594 5222 scope.go:122] "RemoveContainer" containerID="5d2ec0ce2502501329875b65c9f7f40102415da981f2346d04176b1e8915ebae" Oct 31 01:10:10.847279 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:10:10.847252 5222 scope.go:122] "RemoveContainer" containerID="6444d642036e2752ac321f087d01a109a25775bc77b40b0711ee7ec61a3a97ac" Oct 31 01:10:19.328073 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:10:19.328039 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-79tk8"] Oct 31 01:10:25.050581 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:10:25.050545 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d7652"] Oct 31 01:10:25.101678 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:10:25.101643 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d7652"] Oct 31 01:10:25.653814 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:10:25.653774 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb60e8c4-ec26-4826-9380-b8d379e52805" path="/var/lib/kubelet/pods/cb60e8c4-ec26-4826-9380-b8d379e52805/volumes" Oct 31 01:10:35.889585 ip-10-0-137-56 kubenswrapper[5222]: E1031 01:10:35.889548 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 01:10:48.643848 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:10:48.633613 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-79tk8"] Oct 31 01:10:53.050303 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:10:53.050263 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hc4gw"] Oct 31 01:10:53.104078 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:10:53.104038 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hc4gw"] Oct 31 01:10:53.654606 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:10:53.654572 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf20b806-e779-41a5-983b-397ffa538b53" path="/var/lib/kubelet/pods/bf20b806-e779-41a5-983b-397ffa538b53/volumes" Oct 31 01:10:57.446589 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:10:57.446550 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-863fb0-5ff7d48b7sg7fr"] Oct 31 01:11:02.447663 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:11:02.447594 5222 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-863fb0-5ff7d48b7sg7fr" podUID="e0027499-1395-45c9-93e3-071127a25545" containerName="payload-processing" containerID="cri-o://12e8ab4119723261e1c64584349d0d1f8937a7d691bad807342a236c2722c04c" gracePeriod=25 Oct 31 01:11:04.495439 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:11:04.495389 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-79tk8"] Oct 31 01:11:05.894540 ip-10-0-137-56 kubenswrapper[5222]: E1031 01:11:05.894498 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 01:11:06.324898 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:11:06.324857 5222 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-863fb0-5ff7d48b7sg7fr" podUID="e0027499-1395-45c9-93e3-071127a25545" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:11:10.877639 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:11:10.877601 5222 scope.go:122] "RemoveContainer" containerID="85dad0f4844d7d6bf90ab882c0b8777a3afb33c83dfb671a0ab5f788393218d2" Oct 31 01:11:15.527383 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:11:15.527335 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-863fb0-openshift-default-86c56576ccdwkvd"] Oct 31 01:11:15.527837 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:11:15.527578 5222 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-863fb0-openshift-default-86c56576ccdwkvd" podUID="17f59d3b-a204-4e4a-9944-1e1069424e8b" containerName="istio-proxy" containerID="cri-o://8df2079b48acf7e095e6aca4145c51fd457b8b6e3b6e075a803bf2c1c1c934a4" gracePeriod=30 Oct 31 01:11:16.324828 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:11:16.324785 5222 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-863fb0-5ff7d48b7sg7fr" podUID="e0027499-1395-45c9-93e3-071127a25545" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:11:17.566777 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:11:17.566736 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2k4fm"] Oct 31 01:11:17.618192 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:11:17.618155 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2k4fm"] Oct 31 01:11:17.654306 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:11:17.654262 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c891896-6776-4453-b25e-5b7cca46548a" path="/var/lib/kubelet/pods/2c891896-6776-4453-b25e-5b7cca46548a/volumes" Oct 31 01:11:20.784389 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:11:20.784337 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-863fb0-openshift-default-86c56576ccdwkvd" Oct 31 01:11:20.924288 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:11:20.924193 5222 generic.go:356] "Generic (PLEG): container finished" podID="17f59d3b-a204-4e4a-9944-1e1069424e8b" containerID="8df2079b48acf7e095e6aca4145c51fd457b8b6e3b6e075a803bf2c1c1c934a4" exitCode=0 Oct 31 01:11:20.924288 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:11:20.924241 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-863fb0-openshift-default-86c56576ccdwkvd" Oct 31 01:11:20.924577 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:11:20.924282 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-863fb0-openshift-default-86c56576ccdwkvd" event={"ID":"17f59d3b-a204-4e4a-9944-1e1069424e8b","Type":"ContainerDied","Data":"8df2079b48acf7e095e6aca4145c51fd457b8b6e3b6e075a803bf2c1c1c934a4"} Oct 31 01:11:20.924577 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:11:20.924349 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-863fb0-openshift-default-86c56576ccdwkvd" event={"ID":"17f59d3b-a204-4e4a-9944-1e1069424e8b","Type":"ContainerDied","Data":"c06591f8b55a4337af12d3db2de7175b98e307eb03d9405ec1f22823d87eba72"} Oct 31 01:11:20.924577 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:11:20.924399 5222 scope.go:122] "RemoveContainer" containerID="8df2079b48acf7e095e6aca4145c51fd457b8b6e3b6e075a803bf2c1c1c934a4" Oct 31 01:11:20.933351 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:11:20.933326 5222 scope.go:122] "RemoveContainer" containerID="ee4bb99722a32245c0b22b409b764b4bad8e09c14a2427ecab7a278dbd42788e" Oct 31 01:11:20.941786 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:11:20.941754 5222 scope.go:122] "RemoveContainer" containerID="8df2079b48acf7e095e6aca4145c51fd457b8b6e3b6e075a803bf2c1c1c934a4" Oct 31 01:11:20.942074 ip-10-0-137-56 kubenswrapper[5222]: E1031 01:11:20.942056 5222 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8df2079b48acf7e095e6aca4145c51fd457b8b6e3b6e075a803bf2c1c1c934a4\": container with ID starting with 8df2079b48acf7e095e6aca4145c51fd457b8b6e3b6e075a803bf2c1c1c934a4 not found: ID does not exist" containerID="8df2079b48acf7e095e6aca4145c51fd457b8b6e3b6e075a803bf2c1c1c934a4" Oct 31 01:11:20.942121 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:11:20.942086 5222 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8df2079b48acf7e095e6aca4145c51fd457b8b6e3b6e075a803bf2c1c1c934a4"} err="failed to get container status \"8df2079b48acf7e095e6aca4145c51fd457b8b6e3b6e075a803bf2c1c1c934a4\": rpc error: code = NotFound desc = could not find container \"8df2079b48acf7e095e6aca4145c51fd457b8b6e3b6e075a803bf2c1c1c934a4\": container with ID starting with 8df2079b48acf7e095e6aca4145c51fd457b8b6e3b6e075a803bf2c1c1c934a4 not found: ID does not exist" Oct 31 01:11:20.942121 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:11:20.942105 5222 scope.go:122] "RemoveContainer" containerID="ee4bb99722a32245c0b22b409b764b4bad8e09c14a2427ecab7a278dbd42788e" Oct 31 01:11:20.942344 ip-10-0-137-56 kubenswrapper[5222]: E1031 01:11:20.942328 5222 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee4bb99722a32245c0b22b409b764b4bad8e09c14a2427ecab7a278dbd42788e\": container with ID starting with ee4bb99722a32245c0b22b409b764b4bad8e09c14a2427ecab7a278dbd42788e not found: ID does not exist" containerID="ee4bb99722a32245c0b22b409b764b4bad8e09c14a2427ecab7a278dbd42788e" Oct 31 01:11:20.942430 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:11:20.942346 5222 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee4bb99722a32245c0b22b409b764b4bad8e09c14a2427ecab7a278dbd42788e"} err="failed to get container status \"ee4bb99722a32245c0b22b409b764b4bad8e09c14a2427ecab7a278dbd42788e\": rpc error: code = NotFound desc = could not find container \"ee4bb99722a32245c0b22b409b764b4bad8e09c14a2427ecab7a278dbd42788e\": container with ID starting with ee4bb99722a32245c0b22b409b764b4bad8e09c14a2427ecab7a278dbd42788e not found: ID does not exist" Oct 31 01:11:20.955891 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:11:20.955865 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17f59d3b-a204-4e4a-9944-1e1069424e8b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17f59d3b-a204-4e4a-9944-1e1069424e8b-istiod-ca-cert\") pod \"17f59d3b-a204-4e4a-9944-1e1069424e8b\" (UID: \"17f59d3b-a204-4e4a-9944-1e1069424e8b\") " Oct 31 01:11:20.956040 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:11:20.955905 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17f59d3b-a204-4e4a-9944-1e1069424e8b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17f59d3b-a204-4e4a-9944-1e1069424e8b-workload-certs\") pod \"17f59d3b-a204-4e4a-9944-1e1069424e8b\" (UID: \"17f59d3b-a204-4e4a-9944-1e1069424e8b\") " Oct 31 01:11:20.956040 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:11:20.955940 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/17f59d3b-a204-4e4a-9944-1e1069424e8b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17f59d3b-a204-4e4a-9944-1e1069424e8b-istio-podinfo\") pod \"17f59d3b-a204-4e4a-9944-1e1069424e8b\" (UID: \"17f59d3b-a204-4e4a-9944-1e1069424e8b\") " Oct 31 01:11:20.956040 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:11:20.955980 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17f59d3b-a204-4e4a-9944-1e1069424e8b-kube-api-access-k57v5\" (UniqueName: \"kubernetes.io/projected/17f59d3b-a204-4e4a-9944-1e1069424e8b-kube-api-access-k57v5\") pod \"17f59d3b-a204-4e4a-9944-1e1069424e8b\" (UID: \"17f59d3b-a204-4e4a-9944-1e1069424e8b\") " Oct 31 01:11:20.956228 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:11:20.956068 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17f59d3b-a204-4e4a-9944-1e1069424e8b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17f59d3b-a204-4e4a-9944-1e1069424e8b-credential-socket\") pod \"17f59d3b-a204-4e4a-9944-1e1069424e8b\" (UID: \"17f59d3b-a204-4e4a-9944-1e1069424e8b\") " Oct 31 01:11:20.956228 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:11:20.956096 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17f59d3b-a204-4e4a-9944-1e1069424e8b-istio-token\" (UniqueName: \"kubernetes.io/projected/17f59d3b-a204-4e4a-9944-1e1069424e8b-istio-token\") pod \"17f59d3b-a204-4e4a-9944-1e1069424e8b\" (UID: \"17f59d3b-a204-4e4a-9944-1e1069424e8b\") " Oct 31 01:11:20.956228 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:11:20.956129 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17f59d3b-a204-4e4a-9944-1e1069424e8b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17f59d3b-a204-4e4a-9944-1e1069424e8b-workload-socket\") pod \"17f59d3b-a204-4e4a-9944-1e1069424e8b\" (UID: \"17f59d3b-a204-4e4a-9944-1e1069424e8b\") " Oct 31 01:11:20.956228 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:11:20.956186 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17f59d3b-a204-4e4a-9944-1e1069424e8b-workload-certs" pod "17f59d3b-a204-4e4a-9944-1e1069424e8b" (UID: "17f59d3b-a204-4e4a-9944-1e1069424e8b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:11:20.956228 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:11:20.956223 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17f59d3b-a204-4e4a-9944-1e1069424e8b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17f59d3b-a204-4e4a-9944-1e1069424e8b-istio-envoy\") pod \"17f59d3b-a204-4e4a-9944-1e1069424e8b\" (UID: \"17f59d3b-a204-4e4a-9944-1e1069424e8b\") " Oct 31 01:11:20.956543 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:11:20.956272 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17f59d3b-a204-4e4a-9944-1e1069424e8b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17f59d3b-a204-4e4a-9944-1e1069424e8b-istio-data\") pod \"17f59d3b-a204-4e4a-9944-1e1069424e8b\" (UID: \"17f59d3b-a204-4e4a-9944-1e1069424e8b\") " Oct 31 01:11:20.956543 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:11:20.956325 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17f59d3b-a204-4e4a-9944-1e1069424e8b-istiod-ca-cert" pod "17f59d3b-a204-4e4a-9944-1e1069424e8b" (UID: "17f59d3b-a204-4e4a-9944-1e1069424e8b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:11:20.956543 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:11:20.956330 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17f59d3b-a204-4e4a-9944-1e1069424e8b-credential-socket" pod "17f59d3b-a204-4e4a-9944-1e1069424e8b" (UID: "17f59d3b-a204-4e4a-9944-1e1069424e8b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:11:20.956720 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:11:20.956615 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17f59d3b-a204-4e4a-9944-1e1069424e8b-workload-socket" pod "17f59d3b-a204-4e4a-9944-1e1069424e8b" (UID: "17f59d3b-a204-4e4a-9944-1e1069424e8b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:11:20.956720 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:11:20.956663 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17f59d3b-a204-4e4a-9944-1e1069424e8b-istio-data" pod "17f59d3b-a204-4e4a-9944-1e1069424e8b" (UID: "17f59d3b-a204-4e4a-9944-1e1069424e8b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:11:20.956720 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:11:20.956693 5222 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17f59d3b-a204-4e4a-9944-1e1069424e8b-istiod-ca-cert\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:11:20.956720 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:11:20.956712 5222 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17f59d3b-a204-4e4a-9944-1e1069424e8b-workload-certs\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:11:20.957105 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:11:20.956725 5222 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17f59d3b-a204-4e4a-9944-1e1069424e8b-credential-socket\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:11:20.957105 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:11:20.956734 5222 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17f59d3b-a204-4e4a-9944-1e1069424e8b-workload-socket\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:11:20.958596 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:11:20.958547 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17f59d3b-a204-4e4a-9944-1e1069424e8b-istio-envoy" pod "17f59d3b-a204-4e4a-9944-1e1069424e8b" (UID: "17f59d3b-a204-4e4a-9944-1e1069424e8b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:11:20.958731 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:11:20.958648 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17f59d3b-a204-4e4a-9944-1e1069424e8b-istio-token" pod "17f59d3b-a204-4e4a-9944-1e1069424e8b" (UID: "17f59d3b-a204-4e4a-9944-1e1069424e8b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:11:20.958962 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:11:20.958933 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/17f59d3b-a204-4e4a-9944-1e1069424e8b-istio-podinfo" pod "17f59d3b-a204-4e4a-9944-1e1069424e8b" (UID: "17f59d3b-a204-4e4a-9944-1e1069424e8b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:11:20.959111 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:11:20.959047 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17f59d3b-a204-4e4a-9944-1e1069424e8b-kube-api-access-k57v5" pod "17f59d3b-a204-4e4a-9944-1e1069424e8b" (UID: "17f59d3b-a204-4e4a-9944-1e1069424e8b"). InnerVolumeSpecName "kube-api-access-k57v5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:11:21.058168 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:11:21.058126 5222 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17f59d3b-a204-4e4a-9944-1e1069424e8b-istio-envoy\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:11:21.058168 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:11:21.058161 5222 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17f59d3b-a204-4e4a-9944-1e1069424e8b-istio-data\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:11:21.058168 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:11:21.058171 5222 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17f59d3b-a204-4e4a-9944-1e1069424e8b-istio-podinfo\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:11:21.058168 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:11:21.058180 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k57v5\" (UniqueName: \"kubernetes.io/projected/17f59d3b-a204-4e4a-9944-1e1069424e8b-kube-api-access-k57v5\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:11:21.058504 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:11:21.058190 5222 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17f59d3b-a204-4e4a-9944-1e1069424e8b-istio-token\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:11:21.348639 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:11:21.348602 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-863fb0-openshift-default-86c56576ccdwkvd"] Oct 31 01:11:21.400433 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:11:21.400395 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-863fb0-openshift-default-86c56576ccdwkvd"] Oct 31 01:11:21.654720 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:11:21.654690 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17f59d3b-a204-4e4a-9944-1e1069424e8b" path="/var/lib/kubelet/pods/17f59d3b-a204-4e4a-9944-1e1069424e8b/volumes" Oct 31 01:11:22.695173 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:11:22.695149 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-863fb0-5ff7d48b7sg7fr" Oct 31 01:11:22.876552 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:11:22.876513 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0027499-1395-45c9-93e3-071127a25545-kube-api-access-qtfnz\" (UniqueName: \"kubernetes.io/projected/e0027499-1395-45c9-93e3-071127a25545-kube-api-access-qtfnz\") pod \"e0027499-1395-45c9-93e3-071127a25545\" (UID: \"e0027499-1395-45c9-93e3-071127a25545\") " Oct 31 01:11:22.876749 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:11:22.876570 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0027499-1395-45c9-93e3-071127a25545-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0027499-1395-45c9-93e3-071127a25545-plugins-config-volume\") pod \"e0027499-1395-45c9-93e3-071127a25545\" (UID: \"e0027499-1395-45c9-93e3-071127a25545\") " Oct 31 01:11:22.876979 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:11:22.876958 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0027499-1395-45c9-93e3-071127a25545-plugins-config-volume" pod "e0027499-1395-45c9-93e3-071127a25545" (UID: "e0027499-1395-45c9-93e3-071127a25545"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:11:22.878880 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:11:22.878859 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0027499-1395-45c9-93e3-071127a25545-kube-api-access-qtfnz" pod "e0027499-1395-45c9-93e3-071127a25545" (UID: "e0027499-1395-45c9-93e3-071127a25545"). InnerVolumeSpecName "kube-api-access-qtfnz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:11:22.938270 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:11:22.938231 5222 generic.go:356] "Generic (PLEG): container finished" podID="e0027499-1395-45c9-93e3-071127a25545" containerID="12e8ab4119723261e1c64584349d0d1f8937a7d691bad807342a236c2722c04c" exitCode=0 Oct 31 01:11:22.938505 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:11:22.938278 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-863fb0-5ff7d48b7sg7fr" Oct 31 01:11:22.938505 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:11:22.938311 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-863fb0-5ff7d48b7sg7fr" event={"ID":"e0027499-1395-45c9-93e3-071127a25545","Type":"ContainerDied","Data":"12e8ab4119723261e1c64584349d0d1f8937a7d691bad807342a236c2722c04c"} Oct 31 01:11:22.938505 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:11:22.938362 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-863fb0-5ff7d48b7sg7fr" event={"ID":"e0027499-1395-45c9-93e3-071127a25545","Type":"ContainerDied","Data":"d6573f84b65941b99274d3a739c794077738421c397d56213e9f8b2417f1ba30"} Oct 31 01:11:22.938505 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:11:22.938399 5222 scope.go:122] "RemoveContainer" containerID="12e8ab4119723261e1c64584349d0d1f8937a7d691bad807342a236c2722c04c" Oct 31 01:11:22.947630 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:11:22.947609 5222 scope.go:122] "RemoveContainer" containerID="12e8ab4119723261e1c64584349d0d1f8937a7d691bad807342a236c2722c04c" Oct 31 01:11:22.947916 ip-10-0-137-56 kubenswrapper[5222]: E1031 01:11:22.947897 5222 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12e8ab4119723261e1c64584349d0d1f8937a7d691bad807342a236c2722c04c\": container with ID starting with 12e8ab4119723261e1c64584349d0d1f8937a7d691bad807342a236c2722c04c not found: ID does not exist" containerID="12e8ab4119723261e1c64584349d0d1f8937a7d691bad807342a236c2722c04c" Oct 31 01:11:22.947965 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:11:22.947927 5222 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12e8ab4119723261e1c64584349d0d1f8937a7d691bad807342a236c2722c04c"} err="failed to get container status \"12e8ab4119723261e1c64584349d0d1f8937a7d691bad807342a236c2722c04c\": rpc error: code = NotFound desc = could not find container \"12e8ab4119723261e1c64584349d0d1f8937a7d691bad807342a236c2722c04c\": container with ID starting with 12e8ab4119723261e1c64584349d0d1f8937a7d691bad807342a236c2722c04c not found: ID does not exist" Oct 31 01:11:22.977726 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:11:22.977698 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtfnz\" (UniqueName: \"kubernetes.io/projected/e0027499-1395-45c9-93e3-071127a25545-kube-api-access-qtfnz\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:11:22.977726 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:11:22.977724 5222 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0027499-1395-45c9-93e3-071127a25545-plugins-config-volume\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:11:23.106573 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:11:23.106541 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-863fb0-5ff7d48b7sg7fr"] Oct 31 01:11:23.109440 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:11:23.109411 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-863fb0-5ff7d48b7sg7fr"] Oct 31 01:11:23.654138 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:11:23.654099 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0027499-1395-45c9-93e3-071127a25545" path="/var/lib/kubelet/pods/e0027499-1395-45c9-93e3-071127a25545/volumes" Oct 31 01:11:24.049743 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:11:24.049702 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-nqt8l"] Oct 31 01:11:24.052285 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:11:24.052255 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-nqt8l"] Oct 31 01:11:25.655116 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:11:25.655084 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f73d8c55-49c1-4530-8ad7-08514b80cdfa" path="/var/lib/kubelet/pods/f73d8c55-49c1-4530-8ad7-08514b80cdfa/volumes" Oct 31 01:11:35.889199 ip-10-0-137-56 kubenswrapper[5222]: E1031 01:11:35.889158 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 01:11:48.061220 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:11:48.061185 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rrfmz"] Oct 31 01:11:48.112423 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:11:48.112387 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rrfmz"] Oct 31 01:11:49.653903 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:11:49.653857 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c897631-8986-4fd2-bab7-afeae8584d6f" path="/var/lib/kubelet/pods/4c897631-8986-4fd2-bab7-afeae8584d6f/volumes" Oct 31 01:12:05.889547 ip-10-0-137-56 kubenswrapper[5222]: E1031 01:12:05.889506 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 01:12:10.898564 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:12:10.898528 5222 scope.go:122] "RemoveContainer" containerID="9d2eb54c603eff9cd8ee017a9b8172c532b6d556a79df8e9fcfd89a3101becc4" Oct 31 01:12:10.907275 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:12:10.907251 5222 scope.go:122] "RemoveContainer" containerID="2ac70ee1f335c32ef27619426ef2975390d7eee39c900f9939e0c7088d31d2fe" Oct 31 01:12:35.889916 ip-10-0-137-56 kubenswrapper[5222]: E1031 01:12:35.889875 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 01:13:03.810309 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:13:03.810275 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-79tk8"] Oct 31 01:13:06.097412 ip-10-0-137-56 kubenswrapper[5222]: E1031 01:13:06.095583 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 01:13:10.940762 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:13:10.940730 5222 scope.go:122] "RemoveContainer" containerID="fb3b71080fcf0fb81661ef6345110da84edc853f3fe7cdc6f7b6d09b061c01d4" Oct 31 01:13:10.949966 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:13:10.949940 5222 scope.go:122] "RemoveContainer" containerID="19b44b29e3994c0cbc9503f58e043183e75137fc2f4f66751d38002ef0b734f3" Oct 31 01:13:19.049968 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:13:19.049928 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-7h8z8"] Oct 31 01:13:19.101592 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:13:19.101548 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-7h8z8"] Oct 31 01:13:19.654712 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:13:19.654675 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72557662-9af4-4c0a-8b2f-f8282cbd9dfb" path="/var/lib/kubelet/pods/72557662-9af4-4c0a-8b2f-f8282cbd9dfb/volumes" Oct 31 01:13:21.426636 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:13:21.426600 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-79tk8"] Oct 31 01:13:31.196119 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:13:31.196084 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-79tk8"] Oct 31 01:13:35.889770 ip-10-0-137-56 kubenswrapper[5222]: E1031 01:13:35.889732 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 01:13:37.618528 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:13:37.618493 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nhmqk"] Oct 31 01:13:37.618948 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:13:37.618887 5222 cpu_manager.go:414] "containerMap: removing container" podUID="17f59d3b-a204-4e4a-9944-1e1069424e8b" containerName="istio-proxy" Oct 31 01:13:37.618948 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:13:37.618900 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17f59d3b-a204-4e4a-9944-1e1069424e8b" containerName="istio-proxy" Oct 31 01:13:37.618948 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:13:37.618914 5222 cpu_manager.go:414] "containerMap: removing container" podUID="e0027499-1395-45c9-93e3-071127a25545" containerName="payload-processing" Oct 31 01:13:37.618948 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:13:37.618919 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0027499-1395-45c9-93e3-071127a25545" containerName="payload-processing" Oct 31 01:13:37.619097 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:13:37.618980 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0027499-1395-45c9-93e3-071127a25545" containerName="payload-processing" Oct 31 01:13:37.619097 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:13:37.618991 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="17f59d3b-a204-4e4a-9944-1e1069424e8b" containerName="istio-proxy" Oct 31 01:13:37.622023 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:13:37.622005 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nhmqk" Oct 31 01:13:37.676159 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:13:37.676128 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nhmqk"] Oct 31 01:13:37.691320 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:13:37.691280 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd488448-cbbb-490e-a43b-ae78cb4f0f3b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-nhmqk\" (UID: \"cd488448-cbbb-490e-a43b-ae78cb4f0f3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nhmqk" Oct 31 01:13:37.691541 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:13:37.691475 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlvrl\" (UniqueName: \"kubernetes.io/projected/cd488448-cbbb-490e-a43b-ae78cb4f0f3b-kube-api-access-mlvrl\") pod \"maas-api-revoke-sub-d2bc93bc-nhmqk\" (UID: \"cd488448-cbbb-490e-a43b-ae78cb4f0f3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nhmqk" Oct 31 01:13:37.691666 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:13:37.691628 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd488448-cbbb-490e-a43b-ae78cb4f0f3b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-nhmqk\" (UID: \"cd488448-cbbb-490e-a43b-ae78cb4f0f3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nhmqk" Oct 31 01:13:37.725820 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:13:37.725788 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:13:37.725820 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:13:37.725788 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:13:37.726081 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:13:37.725832 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:13:37.726943 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:13:37.726915 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfz9z\"" Oct 31 01:13:37.793245 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:13:37.793216 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd488448-cbbb-490e-a43b-ae78cb4f0f3b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-nhmqk\" (UID: \"cd488448-cbbb-490e-a43b-ae78cb4f0f3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nhmqk" Oct 31 01:13:37.793458 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:13:37.793258 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlvrl\" (UniqueName: \"kubernetes.io/projected/cd488448-cbbb-490e-a43b-ae78cb4f0f3b-kube-api-access-mlvrl\") pod \"maas-api-revoke-sub-d2bc93bc-nhmqk\" (UID: \"cd488448-cbbb-490e-a43b-ae78cb4f0f3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nhmqk" Oct 31 01:13:37.793458 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:13:37.793417 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd488448-cbbb-490e-a43b-ae78cb4f0f3b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-nhmqk\" (UID: \"cd488448-cbbb-490e-a43b-ae78cb4f0f3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nhmqk" Oct 31 01:13:37.793910 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:13:37.793891 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd488448-cbbb-490e-a43b-ae78cb4f0f3b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-nhmqk\" (UID: \"cd488448-cbbb-490e-a43b-ae78cb4f0f3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nhmqk" Oct 31 01:13:37.795936 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:13:37.795908 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd488448-cbbb-490e-a43b-ae78cb4f0f3b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-nhmqk\" (UID: \"cd488448-cbbb-490e-a43b-ae78cb4f0f3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nhmqk" Oct 31 01:13:37.856512 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:13:37.856472 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlvrl\" (UniqueName: \"kubernetes.io/projected/cd488448-cbbb-490e-a43b-ae78cb4f0f3b-kube-api-access-mlvrl\") pod \"maas-api-revoke-sub-d2bc93bc-nhmqk\" (UID: \"cd488448-cbbb-490e-a43b-ae78cb4f0f3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nhmqk" Oct 31 01:13:37.934236 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:13:37.934146 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nhmqk" Oct 31 01:13:38.304107 ip-10-0-137-56 kubenswrapper[5222]: W1031 01:13:38.304071 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd488448_cbbb_490e_a43b_ae78cb4f0f3b.slice/crio-b0d2e426660c7aa0ce724046ac07e99ca6869dc949fd03aaca92ace66cf9733f WatchSource:0}: Error finding container b0d2e426660c7aa0ce724046ac07e99ca6869dc949fd03aaca92ace66cf9733f: Status 404 returned error can't find the container with id b0d2e426660c7aa0ce724046ac07e99ca6869dc949fd03aaca92ace66cf9733f Oct 31 01:13:38.306194 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:13:38.306169 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nhmqk"] Oct 31 01:13:38.668255 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:13:38.668147 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nhmqk" event={"ID":"cd488448-cbbb-490e-a43b-ae78cb4f0f3b","Type":"ContainerStarted","Data":"f18f9c653b78fe5a3e7c7ed97b783c260e4986510fceb0f2fcdc09d1e78cada1"} Oct 31 01:13:38.668255 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:13:38.668227 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nhmqk" event={"ID":"cd488448-cbbb-490e-a43b-ae78cb4f0f3b","Type":"ContainerStarted","Data":"b0d2e426660c7aa0ce724046ac07e99ca6869dc949fd03aaca92ace66cf9733f"} Oct 31 01:13:38.787742 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:13:38.787684 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nhmqk" podStartSLOduration=1.787669616 podStartE2EDuration="1.787669616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:13:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:13:38.786986846 +0000 UTC m=+3511.258156595" watchObservedRunningTime="2026-10-31 01:13:38.787669616 +0000 UTC m=+3511.258839364" Oct 31 01:13:39.673873 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:13:39.673839 5222 generic.go:356] "Generic (PLEG): container finished" podID="cd488448-cbbb-490e-a43b-ae78cb4f0f3b" containerID="f18f9c653b78fe5a3e7c7ed97b783c260e4986510fceb0f2fcdc09d1e78cada1" exitCode=0 Oct 31 01:13:39.673873 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:13:39.673875 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nhmqk" event={"ID":"cd488448-cbbb-490e-a43b-ae78cb4f0f3b","Type":"ContainerDied","Data":"f18f9c653b78fe5a3e7c7ed97b783c260e4986510fceb0f2fcdc09d1e78cada1"} Oct 31 01:13:40.813887 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:13:40.813860 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nhmqk" Oct 31 01:13:40.930037 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:13:40.929950 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd488448-cbbb-490e-a43b-ae78cb4f0f3b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd488448-cbbb-490e-a43b-ae78cb4f0f3b-maas-api-xks-ca\") pod \"cd488448-cbbb-490e-a43b-ae78cb4f0f3b\" (UID: \"cd488448-cbbb-490e-a43b-ae78cb4f0f3b\") " Oct 31 01:13:40.930216 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:13:40.930071 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd488448-cbbb-490e-a43b-ae78cb4f0f3b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd488448-cbbb-490e-a43b-ae78cb4f0f3b-maas-api-openshift-ca\") pod \"cd488448-cbbb-490e-a43b-ae78cb4f0f3b\" (UID: \"cd488448-cbbb-490e-a43b-ae78cb4f0f3b\") " Oct 31 01:13:40.930216 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:13:40.930180 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd488448-cbbb-490e-a43b-ae78cb4f0f3b-kube-api-access-mlvrl\" (UniqueName: \"kubernetes.io/projected/cd488448-cbbb-490e-a43b-ae78cb4f0f3b-kube-api-access-mlvrl\") pod \"cd488448-cbbb-490e-a43b-ae78cb4f0f3b\" (UID: \"cd488448-cbbb-490e-a43b-ae78cb4f0f3b\") " Oct 31 01:13:40.930559 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:13:40.930524 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd488448-cbbb-490e-a43b-ae78cb4f0f3b-maas-api-openshift-ca" pod "cd488448-cbbb-490e-a43b-ae78cb4f0f3b" (UID: "cd488448-cbbb-490e-a43b-ae78cb4f0f3b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:13:40.932510 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:13:40.932478 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd488448-cbbb-490e-a43b-ae78cb4f0f3b-maas-api-xks-ca" pod "cd488448-cbbb-490e-a43b-ae78cb4f0f3b" (UID: "cd488448-cbbb-490e-a43b-ae78cb4f0f3b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:13:40.932806 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:13:40.932781 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd488448-cbbb-490e-a43b-ae78cb4f0f3b-kube-api-access-mlvrl" pod "cd488448-cbbb-490e-a43b-ae78cb4f0f3b" (UID: "cd488448-cbbb-490e-a43b-ae78cb4f0f3b"). InnerVolumeSpecName "kube-api-access-mlvrl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:13:41.031007 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:13:41.030973 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd488448-cbbb-490e-a43b-ae78cb4f0f3b-maas-api-xks-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:13:41.031007 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:13:41.031002 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd488448-cbbb-490e-a43b-ae78cb4f0f3b-maas-api-openshift-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:13:41.031007 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:13:41.031015 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlvrl\" (UniqueName: \"kubernetes.io/projected/cd488448-cbbb-490e-a43b-ae78cb4f0f3b-kube-api-access-mlvrl\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:13:41.684358 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:13:41.684333 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nhmqk" Oct 31 01:13:41.684561 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:13:41.684324 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nhmqk" event={"ID":"cd488448-cbbb-490e-a43b-ae78cb4f0f3b","Type":"ContainerDied","Data":"b0d2e426660c7aa0ce724046ac07e99ca6869dc949fd03aaca92ace66cf9733f"} Oct 31 01:13:41.684561 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:13:41.684452 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b0d2e426660c7aa0ce724046ac07e99ca6869dc949fd03aaca92ace66cf9733f" Oct 31 01:13:50.588777 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:13:50.588686 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-79tk8"] Oct 31 01:13:50.648929 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:13:50.648895 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-abe7bd-openshift-default-7d97f8b7c75x2j7"] Oct 31 01:13:50.649277 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:13:50.649265 5222 cpu_manager.go:414] "containerMap: removing container" podUID="cd488448-cbbb-490e-a43b-ae78cb4f0f3b" containerName="revoke-keys" Oct 31 01:13:50.649322 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:13:50.649279 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd488448-cbbb-490e-a43b-ae78cb4f0f3b" containerName="revoke-keys" Oct 31 01:13:50.649322 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:13:50.649296 5222 cpu_manager.go:414] "containerMap: removing container" podUID="17f59d3b-a204-4e4a-9944-1e1069424e8b" containerName="istio-proxy" Oct 31 01:13:50.649322 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:13:50.649301 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17f59d3b-a204-4e4a-9944-1e1069424e8b" containerName="istio-proxy" Oct 31 01:13:50.649461 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:13:50.649388 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd488448-cbbb-490e-a43b-ae78cb4f0f3b" containerName="revoke-keys" Oct 31 01:13:50.649461 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:13:50.649402 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="17f59d3b-a204-4e4a-9944-1e1069424e8b" containerName="istio-proxy" Oct 31 01:13:50.652667 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:13:50.652638 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-abe7bd-openshift-default-7d97f8b7c75x2j7" Oct 31 01:13:50.706679 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:13:50.706637 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-abe7bd-openshift-default-dockercfg-nwfs5\"" Oct 31 01:13:50.730225 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:13:50.730194 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fb9a8e6-d32e-42d6-8348-5249d6a655f4-istio-data\") pod \"e2e-worker-single-main-abe7bd-openshift-default-7d97f8b7c75x2j7\" (UID: \"8fb9a8e6-d32e-42d6-8348-5249d6a655f4\") " pod="openshift-ingress/e2e-worker-single-main-abe7bd-openshift-default-7d97f8b7c75x2j7" Oct 31 01:13:50.730433 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:13:50.730232 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fb9a8e6-d32e-42d6-8348-5249d6a655f4-credential-socket\") pod \"e2e-worker-single-main-abe7bd-openshift-default-7d97f8b7c75x2j7\" (UID: \"8fb9a8e6-d32e-42d6-8348-5249d6a655f4\") " pod="openshift-ingress/e2e-worker-single-main-abe7bd-openshift-default-7d97f8b7c75x2j7" Oct 31 01:13:50.730433 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:13:50.730263 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fb9a8e6-d32e-42d6-8348-5249d6a655f4-istio-podinfo\") pod \"e2e-worker-single-main-abe7bd-openshift-default-7d97f8b7c75x2j7\" (UID: \"8fb9a8e6-d32e-42d6-8348-5249d6a655f4\") " pod="openshift-ingress/e2e-worker-single-main-abe7bd-openshift-default-7d97f8b7c75x2j7" Oct 31 01:13:50.730525 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:13:50.730463 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fb9a8e6-d32e-42d6-8348-5249d6a655f4-istiod-ca-cert\") pod \"e2e-worker-single-main-abe7bd-openshift-default-7d97f8b7c75x2j7\" (UID: \"8fb9a8e6-d32e-42d6-8348-5249d6a655f4\") " pod="openshift-ingress/e2e-worker-single-main-abe7bd-openshift-default-7d97f8b7c75x2j7" Oct 31 01:13:50.730572 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:13:50.730525 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fb9a8e6-d32e-42d6-8348-5249d6a655f4-istio-token\") pod \"e2e-worker-single-main-abe7bd-openshift-default-7d97f8b7c75x2j7\" (UID: \"8fb9a8e6-d32e-42d6-8348-5249d6a655f4\") " pod="openshift-ingress/e2e-worker-single-main-abe7bd-openshift-default-7d97f8b7c75x2j7" Oct 31 01:13:50.730572 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:13:50.730553 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fb9a8e6-d32e-42d6-8348-5249d6a655f4-workload-socket\") pod \"e2e-worker-single-main-abe7bd-openshift-default-7d97f8b7c75x2j7\" (UID: \"8fb9a8e6-d32e-42d6-8348-5249d6a655f4\") " pod="openshift-ingress/e2e-worker-single-main-abe7bd-openshift-default-7d97f8b7c75x2j7" Oct 31 01:13:50.730689 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:13:50.730651 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kg6lh\" (UniqueName: \"kubernetes.io/projected/8fb9a8e6-d32e-42d6-8348-5249d6a655f4-kube-api-access-kg6lh\") pod \"e2e-worker-single-main-abe7bd-openshift-default-7d97f8b7c75x2j7\" (UID: \"8fb9a8e6-d32e-42d6-8348-5249d6a655f4\") " pod="openshift-ingress/e2e-worker-single-main-abe7bd-openshift-default-7d97f8b7c75x2j7" Oct 31 01:13:50.730751 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:13:50.730706 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fb9a8e6-d32e-42d6-8348-5249d6a655f4-workload-certs\") pod \"e2e-worker-single-main-abe7bd-openshift-default-7d97f8b7c75x2j7\" (UID: \"8fb9a8e6-d32e-42d6-8348-5249d6a655f4\") " pod="openshift-ingress/e2e-worker-single-main-abe7bd-openshift-default-7d97f8b7c75x2j7" Oct 31 01:13:50.730751 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:13:50.730744 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fb9a8e6-d32e-42d6-8348-5249d6a655f4-istio-envoy\") pod \"e2e-worker-single-main-abe7bd-openshift-default-7d97f8b7c75x2j7\" (UID: \"8fb9a8e6-d32e-42d6-8348-5249d6a655f4\") " pod="openshift-ingress/e2e-worker-single-main-abe7bd-openshift-default-7d97f8b7c75x2j7" Oct 31 01:13:50.759849 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:13:50.759809 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-abe7bd-openshift-default-7d97f8b7c75x2j7"] Oct 31 01:13:50.831950 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:13:50.831905 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kg6lh\" (UniqueName: \"kubernetes.io/projected/8fb9a8e6-d32e-42d6-8348-5249d6a655f4-kube-api-access-kg6lh\") pod \"e2e-worker-single-main-abe7bd-openshift-default-7d97f8b7c75x2j7\" (UID: \"8fb9a8e6-d32e-42d6-8348-5249d6a655f4\") " pod="openshift-ingress/e2e-worker-single-main-abe7bd-openshift-default-7d97f8b7c75x2j7" Oct 31 01:13:50.832174 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:13:50.831959 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fb9a8e6-d32e-42d6-8348-5249d6a655f4-workload-certs\") pod \"e2e-worker-single-main-abe7bd-openshift-default-7d97f8b7c75x2j7\" (UID: \"8fb9a8e6-d32e-42d6-8348-5249d6a655f4\") " pod="openshift-ingress/e2e-worker-single-main-abe7bd-openshift-default-7d97f8b7c75x2j7" Oct 31 01:13:50.832174 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:13:50.831987 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fb9a8e6-d32e-42d6-8348-5249d6a655f4-istio-envoy\") pod \"e2e-worker-single-main-abe7bd-openshift-default-7d97f8b7c75x2j7\" (UID: \"8fb9a8e6-d32e-42d6-8348-5249d6a655f4\") " pod="openshift-ingress/e2e-worker-single-main-abe7bd-openshift-default-7d97f8b7c75x2j7" Oct 31 01:13:50.832174 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:13:50.832038 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fb9a8e6-d32e-42d6-8348-5249d6a655f4-istio-data\") pod \"e2e-worker-single-main-abe7bd-openshift-default-7d97f8b7c75x2j7\" (UID: \"8fb9a8e6-d32e-42d6-8348-5249d6a655f4\") " pod="openshift-ingress/e2e-worker-single-main-abe7bd-openshift-default-7d97f8b7c75x2j7" Oct 31 01:13:50.832174 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:13:50.832065 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fb9a8e6-d32e-42d6-8348-5249d6a655f4-credential-socket\") pod \"e2e-worker-single-main-abe7bd-openshift-default-7d97f8b7c75x2j7\" (UID: \"8fb9a8e6-d32e-42d6-8348-5249d6a655f4\") " pod="openshift-ingress/e2e-worker-single-main-abe7bd-openshift-default-7d97f8b7c75x2j7" Oct 31 01:13:50.832174 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:13:50.832111 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fb9a8e6-d32e-42d6-8348-5249d6a655f4-istio-podinfo\") pod \"e2e-worker-single-main-abe7bd-openshift-default-7d97f8b7c75x2j7\" (UID: \"8fb9a8e6-d32e-42d6-8348-5249d6a655f4\") " pod="openshift-ingress/e2e-worker-single-main-abe7bd-openshift-default-7d97f8b7c75x2j7" Oct 31 01:13:50.832519 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:13:50.832191 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fb9a8e6-d32e-42d6-8348-5249d6a655f4-istiod-ca-cert\") pod \"e2e-worker-single-main-abe7bd-openshift-default-7d97f8b7c75x2j7\" (UID: \"8fb9a8e6-d32e-42d6-8348-5249d6a655f4\") " pod="openshift-ingress/e2e-worker-single-main-abe7bd-openshift-default-7d97f8b7c75x2j7" Oct 31 01:13:50.832519 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:13:50.832261 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fb9a8e6-d32e-42d6-8348-5249d6a655f4-istio-token\") pod \"e2e-worker-single-main-abe7bd-openshift-default-7d97f8b7c75x2j7\" (UID: \"8fb9a8e6-d32e-42d6-8348-5249d6a655f4\") " pod="openshift-ingress/e2e-worker-single-main-abe7bd-openshift-default-7d97f8b7c75x2j7" Oct 31 01:13:50.832519 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:13:50.832290 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fb9a8e6-d32e-42d6-8348-5249d6a655f4-workload-socket\") pod \"e2e-worker-single-main-abe7bd-openshift-default-7d97f8b7c75x2j7\" (UID: \"8fb9a8e6-d32e-42d6-8348-5249d6a655f4\") " pod="openshift-ingress/e2e-worker-single-main-abe7bd-openshift-default-7d97f8b7c75x2j7" Oct 31 01:13:50.832519 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:13:50.832447 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fb9a8e6-d32e-42d6-8348-5249d6a655f4-workload-certs\") pod \"e2e-worker-single-main-abe7bd-openshift-default-7d97f8b7c75x2j7\" (UID: \"8fb9a8e6-d32e-42d6-8348-5249d6a655f4\") " pod="openshift-ingress/e2e-worker-single-main-abe7bd-openshift-default-7d97f8b7c75x2j7" Oct 31 01:13:50.832756 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:13:50.832568 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fb9a8e6-d32e-42d6-8348-5249d6a655f4-istio-data\") pod \"e2e-worker-single-main-abe7bd-openshift-default-7d97f8b7c75x2j7\" (UID: \"8fb9a8e6-d32e-42d6-8348-5249d6a655f4\") " pod="openshift-ingress/e2e-worker-single-main-abe7bd-openshift-default-7d97f8b7c75x2j7" Oct 31 01:13:50.832756 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:13:50.832727 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fb9a8e6-d32e-42d6-8348-5249d6a655f4-credential-socket\") pod \"e2e-worker-single-main-abe7bd-openshift-default-7d97f8b7c75x2j7\" (UID: \"8fb9a8e6-d32e-42d6-8348-5249d6a655f4\") " pod="openshift-ingress/e2e-worker-single-main-abe7bd-openshift-default-7d97f8b7c75x2j7" Oct 31 01:13:50.832843 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:13:50.832777 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fb9a8e6-d32e-42d6-8348-5249d6a655f4-workload-socket\") pod \"e2e-worker-single-main-abe7bd-openshift-default-7d97f8b7c75x2j7\" (UID: \"8fb9a8e6-d32e-42d6-8348-5249d6a655f4\") " pod="openshift-ingress/e2e-worker-single-main-abe7bd-openshift-default-7d97f8b7c75x2j7" Oct 31 01:13:50.832946 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:13:50.832924 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fb9a8e6-d32e-42d6-8348-5249d6a655f4-istiod-ca-cert\") pod \"e2e-worker-single-main-abe7bd-openshift-default-7d97f8b7c75x2j7\" (UID: \"8fb9a8e6-d32e-42d6-8348-5249d6a655f4\") " pod="openshift-ingress/e2e-worker-single-main-abe7bd-openshift-default-7d97f8b7c75x2j7" Oct 31 01:13:50.834656 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:13:50.834633 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fb9a8e6-d32e-42d6-8348-5249d6a655f4-istio-envoy\") pod \"e2e-worker-single-main-abe7bd-openshift-default-7d97f8b7c75x2j7\" (UID: \"8fb9a8e6-d32e-42d6-8348-5249d6a655f4\") " pod="openshift-ingress/e2e-worker-single-main-abe7bd-openshift-default-7d97f8b7c75x2j7" Oct 31 01:13:50.834779 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:13:50.834762 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fb9a8e6-d32e-42d6-8348-5249d6a655f4-istio-podinfo\") pod \"e2e-worker-single-main-abe7bd-openshift-default-7d97f8b7c75x2j7\" (UID: \"8fb9a8e6-d32e-42d6-8348-5249d6a655f4\") " pod="openshift-ingress/e2e-worker-single-main-abe7bd-openshift-default-7d97f8b7c75x2j7" Oct 31 01:13:50.900204 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:13:50.900120 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kg6lh\" (UniqueName: \"kubernetes.io/projected/8fb9a8e6-d32e-42d6-8348-5249d6a655f4-kube-api-access-kg6lh\") pod \"e2e-worker-single-main-abe7bd-openshift-default-7d97f8b7c75x2j7\" (UID: \"8fb9a8e6-d32e-42d6-8348-5249d6a655f4\") " pod="openshift-ingress/e2e-worker-single-main-abe7bd-openshift-default-7d97f8b7c75x2j7" Oct 31 01:13:50.921419 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:13:50.916939 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fb9a8e6-d32e-42d6-8348-5249d6a655f4-istio-token\") pod \"e2e-worker-single-main-abe7bd-openshift-default-7d97f8b7c75x2j7\" (UID: \"8fb9a8e6-d32e-42d6-8348-5249d6a655f4\") " pod="openshift-ingress/e2e-worker-single-main-abe7bd-openshift-default-7d97f8b7c75x2j7" Oct 31 01:13:51.009392 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:13:51.003580 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-abe7bd-openshift-default-7d97f8b7c75x2j7" Oct 31 01:13:51.448629 ip-10-0-137-56 kubenswrapper[5222]: W1031 01:13:51.448598 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fb9a8e6_d32e_42d6_8348_5249d6a655f4.slice/crio-77bfd3603820c62c7ad046bfa27c80f068391be2db248340ba1081f7eda944a1 WatchSource:0}: Error finding container 77bfd3603820c62c7ad046bfa27c80f068391be2db248340ba1081f7eda944a1: Status 404 returned error can't find the container with id 77bfd3603820c62c7ad046bfa27c80f068391be2db248340ba1081f7eda944a1 Oct 31 01:13:51.450522 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:13:51.450500 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-abe7bd-openshift-default-7d97f8b7c75x2j7"] Oct 31 01:13:51.451203 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:13:51.451172 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 01:13:51.451261 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:13:51.451232 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 01:13:51.451312 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:13:51.451260 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 01:13:51.731674 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:13:51.731579 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-abe7bd-openshift-default-7d97f8b7c75x2j7" event={"ID":"8fb9a8e6-d32e-42d6-8348-5249d6a655f4","Type":"ContainerStarted","Data":"9325b01f25a16be5ee2f74e151437bf0b0144016582a188ecbf4d99266ef2a10"} Oct 31 01:13:51.731674 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:13:51.731648 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-abe7bd-openshift-default-7d97f8b7c75x2j7" event={"ID":"8fb9a8e6-d32e-42d6-8348-5249d6a655f4","Type":"ContainerStarted","Data":"77bfd3603820c62c7ad046bfa27c80f068391be2db248340ba1081f7eda944a1"} Oct 31 01:13:51.855490 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:13:51.855438 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-abe7bd-openshift-default-7d97f8b7c75x2j7" podStartSLOduration=1.8554242730000001 podStartE2EDuration="1.855424273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:13:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:13:51.85433943 +0000 UTC m=+3524.325509179" watchObservedRunningTime="2026-10-31 01:13:51.855424273 +0000 UTC m=+3524.326594022" Oct 31 01:13:52.011439 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:13:52.011344 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-abe7bd-openshift-default-7d97f8b7c75x2j7" Oct 31 01:13:52.016730 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:13:52.016702 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-abe7bd-openshift-default-7d97f8b7c75x2j7" Oct 31 01:13:52.736724 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:13:52.736697 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-abe7bd-openshift-default-7d97f8b7c75x2j7" Oct 31 01:13:52.737826 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:13:52.737809 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-abe7bd-openshift-default-7d97f8b7c75x2j7" Oct 31 01:13:58.746694 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:13:58.746651 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-abe7bd-dff668477pzlvc"] Oct 31 01:13:58.751065 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:13:58.751032 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-abe7bd-dff668477pzlvc" Oct 31 01:13:58.803978 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:13:58.803942 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-abe7bd-dff668477pzlvc"] Oct 31 01:13:58.838198 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:13:58.838162 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4flrn\" (UniqueName: \"kubernetes.io/projected/e7a9db10-c5b4-4a95-b67a-84a98fbbb553-kube-api-access-4flrn\") pod \"payload-processing-e2e-worker-single-main-abe7bd-dff668477pzlvc\" (UID: \"e7a9db10-c5b4-4a95-b67a-84a98fbbb553\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-abe7bd-dff668477pzlvc" Oct 31 01:13:58.838418 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:13:58.838278 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7a9db10-c5b4-4a95-b67a-84a98fbbb553-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-abe7bd-dff668477pzlvc\" (UID: \"e7a9db10-c5b4-4a95-b67a-84a98fbbb553\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-abe7bd-dff668477pzlvc" Oct 31 01:13:58.853163 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:13:58.853132 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-abe7bd\"" Oct 31 01:13:58.853303 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:13:58.853250 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-8ce4f150-dockercfg-4p79p\"" Oct 31 01:13:58.938926 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:13:58.938888 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4flrn\" (UniqueName: \"kubernetes.io/projected/e7a9db10-c5b4-4a95-b67a-84a98fbbb553-kube-api-access-4flrn\") pod \"payload-processing-e2e-worker-single-main-abe7bd-dff668477pzlvc\" (UID: \"e7a9db10-c5b4-4a95-b67a-84a98fbbb553\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-abe7bd-dff668477pzlvc" Oct 31 01:13:58.939112 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:13:58.938961 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7a9db10-c5b4-4a95-b67a-84a98fbbb553-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-abe7bd-dff668477pzlvc\" (UID: \"e7a9db10-c5b4-4a95-b67a-84a98fbbb553\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-abe7bd-dff668477pzlvc" Oct 31 01:13:58.939773 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:13:58.939751 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7a9db10-c5b4-4a95-b67a-84a98fbbb553-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-abe7bd-dff668477pzlvc\" (UID: \"e7a9db10-c5b4-4a95-b67a-84a98fbbb553\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-abe7bd-dff668477pzlvc" Oct 31 01:13:59.002022 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:13:59.001943 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4flrn\" (UniqueName: \"kubernetes.io/projected/e7a9db10-c5b4-4a95-b67a-84a98fbbb553-kube-api-access-4flrn\") pod \"payload-processing-e2e-worker-single-main-abe7bd-dff668477pzlvc\" (UID: \"e7a9db10-c5b4-4a95-b67a-84a98fbbb553\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-abe7bd-dff668477pzlvc" Oct 31 01:13:59.065103 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:13:59.065065 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-abe7bd-dff668477pzlvc" Oct 31 01:13:59.436027 ip-10-0-137-56 kubenswrapper[5222]: W1031 01:13:59.435989 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7a9db10_c5b4_4a95_b67a_84a98fbbb553.slice/crio-8e3856997f12c7b931803f2d93cf7b930281b42e43188c4b583e22772c486221 WatchSource:0}: Error finding container 8e3856997f12c7b931803f2d93cf7b930281b42e43188c4b583e22772c486221: Status 404 returned error can't find the container with id 8e3856997f12c7b931803f2d93cf7b930281b42e43188c4b583e22772c486221 Oct 31 01:13:59.437924 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:13:59.437896 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-abe7bd-dff668477pzlvc"] Oct 31 01:13:59.771442 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:13:59.771410 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-abe7bd-dff668477pzlvc" event={"ID":"e7a9db10-c5b4-4a95-b67a-84a98fbbb553","Type":"ContainerStarted","Data":"5c26d853c675b26592f10adc2cabb9e318b5251cdf72a610554c2f10a1a583d1"} Oct 31 01:13:59.771870 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:13:59.771460 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-abe7bd-dff668477pzlvc" event={"ID":"e7a9db10-c5b4-4a95-b67a-84a98fbbb553","Type":"ContainerStarted","Data":"8e3856997f12c7b931803f2d93cf7b930281b42e43188c4b583e22772c486221"} Oct 31 01:13:59.771870 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:13:59.771557 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-abe7bd-dff668477pzlvc" Oct 31 01:13:59.891880 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:13:59.891829 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-abe7bd-dff668477pzlvc" podStartSLOduration=1.891813041 podStartE2EDuration="1.891813041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:13:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:13:59.890290975 +0000 UTC m=+3532.361460723" watchObservedRunningTime="2026-10-31 01:13:59.891813041 +0000 UTC m=+3532.362982790" Oct 31 01:14:05.888818 ip-10-0-137-56 kubenswrapper[5222]: E1031 01:14:05.888773 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 01:14:09.988799 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:14:09.988761 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-79tk8"] Oct 31 01:14:10.779290 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:14:10.779262 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-abe7bd-dff668477pzlvc" Oct 31 01:14:19.901450 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:14:19.901331 5222 generic.go:356] "Generic (PLEG): container finished" podID="8fb9a8e6-d32e-42d6-8348-5249d6a655f4" containerID="9325b01f25a16be5ee2f74e151437bf0b0144016582a188ecbf4d99266ef2a10" exitCode=0 Oct 31 01:14:19.901450 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:14:19.901416 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-abe7bd-openshift-default-7d97f8b7c75x2j7" event={"ID":"8fb9a8e6-d32e-42d6-8348-5249d6a655f4","Type":"ContainerDied","Data":"9325b01f25a16be5ee2f74e151437bf0b0144016582a188ecbf4d99266ef2a10"} Oct 31 01:14:19.901919 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:14:19.901827 5222 scope.go:122] "RemoveContainer" containerID="9325b01f25a16be5ee2f74e151437bf0b0144016582a188ecbf4d99266ef2a10" Oct 31 01:14:19.903116 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:14:19.903088 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 01:14:19.903194 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:14:19.903160 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 01:14:19.903233 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:14:19.903193 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 01:14:20.907767 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:14:20.907730 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-abe7bd-openshift-default-7d97f8b7c75x2j7" event={"ID":"8fb9a8e6-d32e-42d6-8348-5249d6a655f4","Type":"ContainerStarted","Data":"7c9f113c8b6b4bd35de75ecd26b967732ff488a63affd3f7e9f83d65bc81d12b"} Oct 31 01:14:21.011125 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:14:21.011086 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-abe7bd-openshift-default-7d97f8b7c75x2j7" Oct 31 01:14:21.012386 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:14:21.012344 5222 patch_prober.go:29] interesting pod/e2e-worker-single-main-abe7bd-openshift-default-7d97f8b7c75x2j7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.9:15021/healthz/ready\": dial tcp 10.134.1.9:15021: connect: connection refused" start-of-body= Oct 31 01:14:21.012523 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:14:21.012422 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-abe7bd-openshift-default-7d97f8b7c75x2j7" podUID="8fb9a8e6-d32e-42d6-8348-5249d6a655f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.9:15021/healthz/ready\": dial tcp 10.134.1.9:15021: connect: connection refused" Oct 31 01:14:22.010874 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:14:22.010840 5222 patch_prober.go:29] interesting pod/e2e-worker-single-main-abe7bd-openshift-default-7d97f8b7c75x2j7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.9:15021/healthz/ready\": dial tcp 10.134.1.9:15021: connect: connection refused" start-of-body= Oct 31 01:14:22.011278 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:14:22.010907 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-abe7bd-openshift-default-7d97f8b7c75x2j7" podUID="8fb9a8e6-d32e-42d6-8348-5249d6a655f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.9:15021/healthz/ready\": dial tcp 10.134.1.9:15021: connect: connection refused" Oct 31 01:14:22.737026 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:14:22.736983 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-abe7bd-openshift-default-7d97f8b7c75x2j7" Oct 31 01:14:23.011065 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:14:23.010983 5222 patch_prober.go:29] interesting pod/e2e-worker-single-main-abe7bd-openshift-default-7d97f8b7c75x2j7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.9:15021/healthz/ready\": dial tcp 10.134.1.9:15021: connect: connection refused" start-of-body= Oct 31 01:14:23.011065 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:14:23.011055 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-abe7bd-openshift-default-7d97f8b7c75x2j7" podUID="8fb9a8e6-d32e-42d6-8348-5249d6a655f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.9:15021/healthz/ready\": dial tcp 10.134.1.9:15021: connect: connection refused" Oct 31 01:14:24.011316 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:14:24.011273 5222 patch_prober.go:29] interesting pod/e2e-worker-single-main-abe7bd-openshift-default-7d97f8b7c75x2j7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.9:15021/healthz/ready\": dial tcp 10.134.1.9:15021: connect: connection refused" start-of-body= Oct 31 01:14:24.011768 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:14:24.011355 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-abe7bd-openshift-default-7d97f8b7c75x2j7" podUID="8fb9a8e6-d32e-42d6-8348-5249d6a655f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.9:15021/healthz/ready\": dial tcp 10.134.1.9:15021: connect: connection refused" Oct 31 01:14:25.011262 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:14:25.011229 5222 patch_prober.go:29] interesting pod/e2e-worker-single-main-abe7bd-openshift-default-7d97f8b7c75x2j7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.9:15021/healthz/ready\": dial tcp 10.134.1.9:15021: connect: connection refused" start-of-body= Oct 31 01:14:25.011474 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:14:25.011283 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-abe7bd-openshift-default-7d97f8b7c75x2j7" podUID="8fb9a8e6-d32e-42d6-8348-5249d6a655f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.9:15021/healthz/ready\": dial tcp 10.134.1.9:15021: connect: connection refused" Oct 31 01:14:26.048118 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:14:26.048063 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-abe7bd-openshift-default-7d97f8b7c75x2j7" Oct 31 01:14:26.050115 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:14:26.050090 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-abe7bd-openshift-default-7d97f8b7c75x2j7" Oct 31 01:14:28.129117 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:14:28.129069 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-79tk8"] Oct 31 01:14:35.889103 ip-10-0-137-56 kubenswrapper[5222]: E1031 01:14:35.889062 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 01:14:53.798563 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:14:53.798527 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-79tk8"] Oct 31 01:15:00.171316 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:15:00.171281 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-abe7bd-2989015fd6sb"] Oct 31 01:15:00.173966 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:15:00.173949 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-abe7bd-2989015fd6sb" Oct 31 01:15:00.235078 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:15:00.235038 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:15:00.235251 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:15:00.235080 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:15:00.283146 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:15:00.283115 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-abe7bd-2989015fd6sb"] Oct 31 01:15:00.298337 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:15:00.298303 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xgdl\" (UniqueName: \"kubernetes.io/projected/1eadd23c-6205-43d0-8483-690e92292e64-kube-api-access-7xgdl\") pod \"maas-api-key-cleanup-e2e-worker-single-main-abe7bd-2989015fd6sb\" (UID: \"1eadd23c-6205-43d0-8483-690e92292e64\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-abe7bd-2989015fd6sb" Oct 31 01:15:00.298568 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:15:00.298384 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1eadd23c-6205-43d0-8483-690e92292e64-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-single-main-abe7bd-2989015fd6sb\" (UID: \"1eadd23c-6205-43d0-8483-690e92292e64\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-abe7bd-2989015fd6sb" Oct 31 01:15:00.332799 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:15:00.332770 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfz9z\"" Oct 31 01:15:00.399643 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:15:00.399604 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xgdl\" (UniqueName: \"kubernetes.io/projected/1eadd23c-6205-43d0-8483-690e92292e64-kube-api-access-7xgdl\") pod \"maas-api-key-cleanup-e2e-worker-single-main-abe7bd-2989015fd6sb\" (UID: \"1eadd23c-6205-43d0-8483-690e92292e64\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-abe7bd-2989015fd6sb" Oct 31 01:15:00.399854 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:15:00.399661 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1eadd23c-6205-43d0-8483-690e92292e64-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-single-main-abe7bd-2989015fd6sb\" (UID: \"1eadd23c-6205-43d0-8483-690e92292e64\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-abe7bd-2989015fd6sb" Oct 31 01:15:00.400359 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:15:00.400336 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1eadd23c-6205-43d0-8483-690e92292e64-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-single-main-abe7bd-2989015fd6sb\" (UID: \"1eadd23c-6205-43d0-8483-690e92292e64\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-abe7bd-2989015fd6sb" Oct 31 01:15:00.462347 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:15:00.462269 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xgdl\" (UniqueName: \"kubernetes.io/projected/1eadd23c-6205-43d0-8483-690e92292e64-kube-api-access-7xgdl\") pod \"maas-api-key-cleanup-e2e-worker-single-main-abe7bd-2989015fd6sb\" (UID: \"1eadd23c-6205-43d0-8483-690e92292e64\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-abe7bd-2989015fd6sb" Oct 31 01:15:00.486274 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:15:00.486239 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-abe7bd-2989015fd6sb" Oct 31 01:15:00.859329 ip-10-0-137-56 kubenswrapper[5222]: W1031 01:15:00.859291 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1eadd23c_6205_43d0_8483_690e92292e64.slice/crio-5235cf307b53bbfa9b51e02bfebb50f76dd91ae151cc7363093f37233e68d33a WatchSource:0}: Error finding container 5235cf307b53bbfa9b51e02bfebb50f76dd91ae151cc7363093f37233e68d33a: Status 404 returned error can't find the container with id 5235cf307b53bbfa9b51e02bfebb50f76dd91ae151cc7363093f37233e68d33a Oct 31 01:15:00.860641 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:15:00.860612 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-abe7bd-2989015fd6sb"] Oct 31 01:15:01.097938 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:15:01.097900 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-abe7bd-2989015fd6sb" event={"ID":"1eadd23c-6205-43d0-8483-690e92292e64","Type":"ContainerStarted","Data":"5235cf307b53bbfa9b51e02bfebb50f76dd91ae151cc7363093f37233e68d33a"} Oct 31 01:15:02.104544 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:15:02.104505 5222 generic.go:356] "Generic (PLEG): container finished" podID="1eadd23c-6205-43d0-8483-690e92292e64" containerID="369971da1e4f415a8b21613fcac0c47527285942e9d32bc505a0113e19a5fade" exitCode=60 Oct 31 01:15:02.105013 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:15:02.104594 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-abe7bd-2989015fd6sb" event={"ID":"1eadd23c-6205-43d0-8483-690e92292e64","Type":"ContainerDied","Data":"369971da1e4f415a8b21613fcac0c47527285942e9d32bc505a0113e19a5fade"} Oct 31 01:15:02.105013 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:15:02.104808 5222 scope.go:122] "RemoveContainer" containerID="369971da1e4f415a8b21613fcac0c47527285942e9d32bc505a0113e19a5fade" Oct 31 01:15:03.111645 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:15:03.111608 5222 generic.go:356] "Generic (PLEG): container finished" podID="1eadd23c-6205-43d0-8483-690e92292e64" containerID="91f6a88131d5bc6edde04ab10db07e9beb3a8460d0e3e5e2270db869a7399ed1" exitCode=60 Oct 31 01:15:03.112121 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:15:03.111648 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-abe7bd-2989015fd6sb" event={"ID":"1eadd23c-6205-43d0-8483-690e92292e64","Type":"ContainerDied","Data":"91f6a88131d5bc6edde04ab10db07e9beb3a8460d0e3e5e2270db869a7399ed1"} Oct 31 01:15:03.112121 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:15:03.111715 5222 scope.go:122] "RemoveContainer" containerID="369971da1e4f415a8b21613fcac0c47527285942e9d32bc505a0113e19a5fade" Oct 31 01:15:03.112121 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:15:03.111960 5222 scope.go:122] "RemoveContainer" containerID="91f6a88131d5bc6edde04ab10db07e9beb3a8460d0e3e5e2270db869a7399ed1" Oct 31 01:15:03.112286 ip-10-0-137-56 kubenswrapper[5222]: E1031 01:15:03.112262 5222 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-single-main-abe7bd-2989015fd6sb_odh-ai-gateway-infra(1eadd23c-6205-43d0-8483-690e92292e64)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-abe7bd-2989015fd6sb" podUID="1eadd23c-6205-43d0-8483-690e92292e64" Oct 31 01:15:04.119307 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:15:04.119279 5222 scope.go:122] "RemoveContainer" containerID="91f6a88131d5bc6edde04ab10db07e9beb3a8460d0e3e5e2270db869a7399ed1" Oct 31 01:15:04.119730 ip-10-0-137-56 kubenswrapper[5222]: E1031 01:15:04.119550 5222 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-single-main-abe7bd-2989015fd6sb_odh-ai-gateway-infra(1eadd23c-6205-43d0-8483-690e92292e64)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-abe7bd-2989015fd6sb" podUID="1eadd23c-6205-43d0-8483-690e92292e64" Oct 31 01:15:05.888236 ip-10-0-137-56 kubenswrapper[5222]: E1031 01:15:05.888198 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 01:15:08.306775 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:15:08.306740 5222 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h9hsf_0f8c3c03-a1fc-4376-98c6-feeeefa08068/ovn-acl-logging/0.log" Oct 31 01:15:08.335614 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:15:08.335588 5222 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h9hsf_0f8c3c03-a1fc-4376-98c6-feeeefa08068/ovn-acl-logging/0.log" Oct 31 01:15:10.981698 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:15:10.981668 5222 scope.go:122] "RemoveContainer" containerID="5f0d1958915d1347e9d896f70b0c1e0a0f9c442c2f3cf3418a40c331e6c81397" Oct 31 01:15:10.990528 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:15:10.990507 5222 scope.go:122] "RemoveContainer" containerID="e99a4fb66f0fdf8660180e4ce2a0af279bbc66375c678ac9dbf052316400221d" Oct 31 01:15:13.649514 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:15:13.649479 5222 scope.go:122] "RemoveContainer" containerID="91f6a88131d5bc6edde04ab10db07e9beb3a8460d0e3e5e2270db869a7399ed1" Oct 31 01:15:14.168967 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:15:14.168932 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-abe7bd-2989015fd6sb" event={"ID":"1eadd23c-6205-43d0-8483-690e92292e64","Type":"ContainerStarted","Data":"19e53cdb79d5e524fa2786c8d4501d8ab1ace445c6e4760a8f474558a6a46006"} Oct 31 01:15:14.287941 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:15:14.287880 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-abe7bd-2989015fd6sb" podStartSLOduration=14.287865637 podStartE2EDuration="14.287865637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:15:14.287350146 +0000 UTC m=+3606.758519897" watchObservedRunningTime="2026-10-31 01:15:14.287865637 +0000 UTC m=+3606.759035395" Oct 31 01:15:15.175702 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:15:15.175660 5222 generic.go:356] "Generic (PLEG): container finished" podID="1eadd23c-6205-43d0-8483-690e92292e64" containerID="19e53cdb79d5e524fa2786c8d4501d8ab1ace445c6e4760a8f474558a6a46006" exitCode=60 Oct 31 01:15:15.176200 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:15:15.175733 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-abe7bd-2989015fd6sb" event={"ID":"1eadd23c-6205-43d0-8483-690e92292e64","Type":"ContainerDied","Data":"19e53cdb79d5e524fa2786c8d4501d8ab1ace445c6e4760a8f474558a6a46006"} Oct 31 01:15:15.176200 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:15:15.175797 5222 scope.go:122] "RemoveContainer" containerID="91f6a88131d5bc6edde04ab10db07e9beb3a8460d0e3e5e2270db869a7399ed1" Oct 31 01:15:15.176200 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:15:15.176117 5222 scope.go:122] "RemoveContainer" containerID="19e53cdb79d5e524fa2786c8d4501d8ab1ace445c6e4760a8f474558a6a46006" Oct 31 01:15:15.176466 ip-10-0-137-56 kubenswrapper[5222]: E1031 01:15:15.176431 5222 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-single-main-abe7bd-2989015fd6sb_odh-ai-gateway-infra(1eadd23c-6205-43d0-8483-690e92292e64)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-abe7bd-2989015fd6sb" podUID="1eadd23c-6205-43d0-8483-690e92292e64" Oct 31 01:15:15.303122 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:15:15.303091 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-abe7bd-2989015fd6sb"] Oct 31 01:15:16.321625 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:15:16.321596 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-abe7bd-2989015fd6sb" Oct 31 01:15:16.471220 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:15:16.471129 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1eadd23c-6205-43d0-8483-690e92292e64-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1eadd23c-6205-43d0-8483-690e92292e64-maas-api-service-ca\") pod \"1eadd23c-6205-43d0-8483-690e92292e64\" (UID: \"1eadd23c-6205-43d0-8483-690e92292e64\") " Oct 31 01:15:16.471220 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:15:16.471216 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1eadd23c-6205-43d0-8483-690e92292e64-kube-api-access-7xgdl\" (UniqueName: \"kubernetes.io/projected/1eadd23c-6205-43d0-8483-690e92292e64-kube-api-access-7xgdl\") pod \"1eadd23c-6205-43d0-8483-690e92292e64\" (UID: \"1eadd23c-6205-43d0-8483-690e92292e64\") " Oct 31 01:15:16.471569 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:15:16.471542 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1eadd23c-6205-43d0-8483-690e92292e64-maas-api-service-ca" pod "1eadd23c-6205-43d0-8483-690e92292e64" (UID: "1eadd23c-6205-43d0-8483-690e92292e64"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:15:16.473542 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:15:16.473516 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1eadd23c-6205-43d0-8483-690e92292e64-kube-api-access-7xgdl" pod "1eadd23c-6205-43d0-8483-690e92292e64" (UID: "1eadd23c-6205-43d0-8483-690e92292e64"). InnerVolumeSpecName "kube-api-access-7xgdl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:15:16.572120 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:15:16.572078 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1eadd23c-6205-43d0-8483-690e92292e64-maas-api-service-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:16.572120 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:15:16.572112 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xgdl\" (UniqueName: \"kubernetes.io/projected/1eadd23c-6205-43d0-8483-690e92292e64-kube-api-access-7xgdl\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:17.188568 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:15:17.188531 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-abe7bd-2989015fd6sb" Oct 31 01:15:17.188763 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:15:17.188567 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-abe7bd-2989015fd6sb" event={"ID":"1eadd23c-6205-43d0-8483-690e92292e64","Type":"ContainerDied","Data":"5235cf307b53bbfa9b51e02bfebb50f76dd91ae151cc7363093f37233e68d33a"} Oct 31 01:15:17.188763 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:15:17.188635 5222 scope.go:122] "RemoveContainer" containerID="19e53cdb79d5e524fa2786c8d4501d8ab1ace445c6e4760a8f474558a6a46006" Oct 31 01:15:17.363677 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:15:17.363638 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-abe7bd-2989015fd6sb"] Oct 31 01:15:17.414054 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:15:17.414014 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-abe7bd-2989015fd6sb"] Oct 31 01:15:17.655016 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:15:17.654982 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1eadd23c-6205-43d0-8483-690e92292e64" path="/var/lib/kubelet/pods/1eadd23c-6205-43d0-8483-690e92292e64/volumes" Oct 31 01:15:22.082355 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:15:22.082321 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-79tk8"] Oct 31 01:15:32.026246 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:15:32.026210 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-abe7bd-dff668477pzlvc"] Oct 31 01:15:35.890679 ip-10-0-137-56 kubenswrapper[5222]: E1031 01:15:35.890632 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 01:15:37.026931 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:15:37.026884 5222 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-abe7bd-dff668477pzlvc" podUID="e7a9db10-c5b4-4a95-b67a-84a98fbbb553" containerName="payload-processing" containerID="cri-o://5c26d853c675b26592f10adc2cabb9e318b5251cdf72a610554c2f10a1a583d1" gracePeriod=25 Oct 31 01:15:39.015205 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:15:39.015169 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-79tk8"] Oct 31 01:15:40.778047 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:15:40.778006 5222 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-abe7bd-dff668477pzlvc" podUID="e7a9db10-c5b4-4a95-b67a-84a98fbbb553" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:15:49.759523 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:15:49.759487 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-abe7bd-openshift-default-7d97f8b7c75x2j7"] Oct 31 01:15:49.760007 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:15:49.759708 5222 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-abe7bd-openshift-default-7d97f8b7c75x2j7" podUID="8fb9a8e6-d32e-42d6-8348-5249d6a655f4" containerName="istio-proxy" containerID="cri-o://7c9f113c8b6b4bd35de75ecd26b967732ff488a63affd3f7e9f83d65bc81d12b" gracePeriod=30 Oct 31 01:15:50.777943 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:15:50.777905 5222 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-abe7bd-dff668477pzlvc" podUID="e7a9db10-c5b4-4a95-b67a-84a98fbbb553" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:15:55.013765 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:15:55.013739 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-abe7bd-openshift-default-7d97f8b7c75x2j7" Oct 31 01:15:55.145177 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:15:55.145089 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fb9a8e6-d32e-42d6-8348-5249d6a655f4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fb9a8e6-d32e-42d6-8348-5249d6a655f4-istio-data\") pod \"8fb9a8e6-d32e-42d6-8348-5249d6a655f4\" (UID: \"8fb9a8e6-d32e-42d6-8348-5249d6a655f4\") " Oct 31 01:15:55.145177 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:15:55.145138 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fb9a8e6-d32e-42d6-8348-5249d6a655f4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fb9a8e6-d32e-42d6-8348-5249d6a655f4-workload-certs\") pod \"8fb9a8e6-d32e-42d6-8348-5249d6a655f4\" (UID: \"8fb9a8e6-d32e-42d6-8348-5249d6a655f4\") " Oct 31 01:15:55.145177 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:15:55.145180 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fb9a8e6-d32e-42d6-8348-5249d6a655f4-kube-api-access-kg6lh\" (UniqueName: \"kubernetes.io/projected/8fb9a8e6-d32e-42d6-8348-5249d6a655f4-kube-api-access-kg6lh\") pod \"8fb9a8e6-d32e-42d6-8348-5249d6a655f4\" (UID: \"8fb9a8e6-d32e-42d6-8348-5249d6a655f4\") " Oct 31 01:15:55.145536 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:15:55.145204 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fb9a8e6-d32e-42d6-8348-5249d6a655f4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fb9a8e6-d32e-42d6-8348-5249d6a655f4-workload-socket\") pod \"8fb9a8e6-d32e-42d6-8348-5249d6a655f4\" (UID: \"8fb9a8e6-d32e-42d6-8348-5249d6a655f4\") " Oct 31 01:15:55.145536 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:15:55.145222 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fb9a8e6-d32e-42d6-8348-5249d6a655f4-istio-token\" (UniqueName: \"kubernetes.io/projected/8fb9a8e6-d32e-42d6-8348-5249d6a655f4-istio-token\") pod \"8fb9a8e6-d32e-42d6-8348-5249d6a655f4\" (UID: \"8fb9a8e6-d32e-42d6-8348-5249d6a655f4\") " Oct 31 01:15:55.145536 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:15:55.145250 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fb9a8e6-d32e-42d6-8348-5249d6a655f4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fb9a8e6-d32e-42d6-8348-5249d6a655f4-istio-envoy\") pod \"8fb9a8e6-d32e-42d6-8348-5249d6a655f4\" (UID: \"8fb9a8e6-d32e-42d6-8348-5249d6a655f4\") " Oct 31 01:15:55.145536 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:15:55.145291 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8fb9a8e6-d32e-42d6-8348-5249d6a655f4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fb9a8e6-d32e-42d6-8348-5249d6a655f4-istio-podinfo\") pod \"8fb9a8e6-d32e-42d6-8348-5249d6a655f4\" (UID: \"8fb9a8e6-d32e-42d6-8348-5249d6a655f4\") " Oct 31 01:15:55.145536 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:15:55.145336 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fb9a8e6-d32e-42d6-8348-5249d6a655f4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fb9a8e6-d32e-42d6-8348-5249d6a655f4-credential-socket\") pod \"8fb9a8e6-d32e-42d6-8348-5249d6a655f4\" (UID: \"8fb9a8e6-d32e-42d6-8348-5249d6a655f4\") " Oct 31 01:15:55.145536 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:15:55.145431 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fb9a8e6-d32e-42d6-8348-5249d6a655f4-istio-data" pod "8fb9a8e6-d32e-42d6-8348-5249d6a655f4" (UID: "8fb9a8e6-d32e-42d6-8348-5249d6a655f4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:15:55.145536 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:15:55.145439 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fb9a8e6-d32e-42d6-8348-5249d6a655f4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fb9a8e6-d32e-42d6-8348-5249d6a655f4-istiod-ca-cert\") pod \"8fb9a8e6-d32e-42d6-8348-5249d6a655f4\" (UID: \"8fb9a8e6-d32e-42d6-8348-5249d6a655f4\") " Oct 31 01:15:55.145930 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:15:55.145602 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fb9a8e6-d32e-42d6-8348-5249d6a655f4-workload-socket" pod "8fb9a8e6-d32e-42d6-8348-5249d6a655f4" (UID: "8fb9a8e6-d32e-42d6-8348-5249d6a655f4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:15:55.145930 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:15:55.145677 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fb9a8e6-d32e-42d6-8348-5249d6a655f4-workload-certs" pod "8fb9a8e6-d32e-42d6-8348-5249d6a655f4" (UID: "8fb9a8e6-d32e-42d6-8348-5249d6a655f4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:15:55.145930 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:15:55.145819 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fb9a8e6-d32e-42d6-8348-5249d6a655f4-credential-socket" pod "8fb9a8e6-d32e-42d6-8348-5249d6a655f4" (UID: "8fb9a8e6-d32e-42d6-8348-5249d6a655f4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:15:55.146280 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:15:55.146078 5222 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fb9a8e6-d32e-42d6-8348-5249d6a655f4-istio-data\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:55.146280 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:15:55.146098 5222 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fb9a8e6-d32e-42d6-8348-5249d6a655f4-workload-certs\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:55.146280 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:15:55.146113 5222 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fb9a8e6-d32e-42d6-8348-5249d6a655f4-workload-socket\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:55.146280 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:15:55.146128 5222 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fb9a8e6-d32e-42d6-8348-5249d6a655f4-credential-socket\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:55.146280 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:15:55.146127 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fb9a8e6-d32e-42d6-8348-5249d6a655f4-istiod-ca-cert" pod "8fb9a8e6-d32e-42d6-8348-5249d6a655f4" (UID: "8fb9a8e6-d32e-42d6-8348-5249d6a655f4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:15:55.147749 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:15:55.147722 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fb9a8e6-d32e-42d6-8348-5249d6a655f4-kube-api-access-kg6lh" pod "8fb9a8e6-d32e-42d6-8348-5249d6a655f4" (UID: "8fb9a8e6-d32e-42d6-8348-5249d6a655f4"). InnerVolumeSpecName "kube-api-access-kg6lh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:15:55.147898 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:15:55.147817 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fb9a8e6-d32e-42d6-8348-5249d6a655f4-istio-envoy" pod "8fb9a8e6-d32e-42d6-8348-5249d6a655f4" (UID: "8fb9a8e6-d32e-42d6-8348-5249d6a655f4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:15:55.148135 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:15:55.148111 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fb9a8e6-d32e-42d6-8348-5249d6a655f4-istio-token" pod "8fb9a8e6-d32e-42d6-8348-5249d6a655f4" (UID: "8fb9a8e6-d32e-42d6-8348-5249d6a655f4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:15:55.148135 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:15:55.148116 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8fb9a8e6-d32e-42d6-8348-5249d6a655f4-istio-podinfo" pod "8fb9a8e6-d32e-42d6-8348-5249d6a655f4" (UID: "8fb9a8e6-d32e-42d6-8348-5249d6a655f4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:15:55.247394 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:15:55.247334 5222 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fb9a8e6-d32e-42d6-8348-5249d6a655f4-istiod-ca-cert\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:55.247394 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:15:55.247393 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kg6lh\" (UniqueName: \"kubernetes.io/projected/8fb9a8e6-d32e-42d6-8348-5249d6a655f4-kube-api-access-kg6lh\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:55.247675 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:15:55.247405 5222 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fb9a8e6-d32e-42d6-8348-5249d6a655f4-istio-token\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:55.247675 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:15:55.247415 5222 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fb9a8e6-d32e-42d6-8348-5249d6a655f4-istio-envoy\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:55.247675 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:15:55.247424 5222 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fb9a8e6-d32e-42d6-8348-5249d6a655f4-istio-podinfo\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:55.369436 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:15:55.369396 5222 generic.go:356] "Generic (PLEG): container finished" podID="8fb9a8e6-d32e-42d6-8348-5249d6a655f4" containerID="7c9f113c8b6b4bd35de75ecd26b967732ff488a63affd3f7e9f83d65bc81d12b" exitCode=0 Oct 31 01:15:55.369623 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:15:55.369467 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-abe7bd-openshift-default-7d97f8b7c75x2j7" Oct 31 01:15:55.369623 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:15:55.369478 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-abe7bd-openshift-default-7d97f8b7c75x2j7" event={"ID":"8fb9a8e6-d32e-42d6-8348-5249d6a655f4","Type":"ContainerDied","Data":"7c9f113c8b6b4bd35de75ecd26b967732ff488a63affd3f7e9f83d65bc81d12b"} Oct 31 01:15:55.369623 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:15:55.369571 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-abe7bd-openshift-default-7d97f8b7c75x2j7" event={"ID":"8fb9a8e6-d32e-42d6-8348-5249d6a655f4","Type":"ContainerDied","Data":"77bfd3603820c62c7ad046bfa27c80f068391be2db248340ba1081f7eda944a1"} Oct 31 01:15:55.369623 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:15:55.369594 5222 scope.go:122] "RemoveContainer" containerID="7c9f113c8b6b4bd35de75ecd26b967732ff488a63affd3f7e9f83d65bc81d12b" Oct 31 01:15:55.379216 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:15:55.379197 5222 scope.go:122] "RemoveContainer" containerID="9325b01f25a16be5ee2f74e151437bf0b0144016582a188ecbf4d99266ef2a10" Oct 31 01:15:55.387364 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:15:55.387338 5222 scope.go:122] "RemoveContainer" containerID="7c9f113c8b6b4bd35de75ecd26b967732ff488a63affd3f7e9f83d65bc81d12b" Oct 31 01:15:55.387640 ip-10-0-137-56 kubenswrapper[5222]: E1031 01:15:55.387617 5222 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c9f113c8b6b4bd35de75ecd26b967732ff488a63affd3f7e9f83d65bc81d12b\": container with ID starting with 7c9f113c8b6b4bd35de75ecd26b967732ff488a63affd3f7e9f83d65bc81d12b not found: ID does not exist" containerID="7c9f113c8b6b4bd35de75ecd26b967732ff488a63affd3f7e9f83d65bc81d12b" Oct 31 01:15:55.387696 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:15:55.387651 5222 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c9f113c8b6b4bd35de75ecd26b967732ff488a63affd3f7e9f83d65bc81d12b"} err="failed to get container status \"7c9f113c8b6b4bd35de75ecd26b967732ff488a63affd3f7e9f83d65bc81d12b\": rpc error: code = NotFound desc = could not find container \"7c9f113c8b6b4bd35de75ecd26b967732ff488a63affd3f7e9f83d65bc81d12b\": container with ID starting with 7c9f113c8b6b4bd35de75ecd26b967732ff488a63affd3f7e9f83d65bc81d12b not found: ID does not exist" Oct 31 01:15:55.387696 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:15:55.387670 5222 scope.go:122] "RemoveContainer" containerID="9325b01f25a16be5ee2f74e151437bf0b0144016582a188ecbf4d99266ef2a10" Oct 31 01:15:55.387940 ip-10-0-137-56 kubenswrapper[5222]: E1031 01:15:55.387916 5222 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9325b01f25a16be5ee2f74e151437bf0b0144016582a188ecbf4d99266ef2a10\": container with ID starting with 9325b01f25a16be5ee2f74e151437bf0b0144016582a188ecbf4d99266ef2a10 not found: ID does not exist" containerID="9325b01f25a16be5ee2f74e151437bf0b0144016582a188ecbf4d99266ef2a10" Oct 31 01:15:55.387988 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:15:55.387947 5222 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9325b01f25a16be5ee2f74e151437bf0b0144016582a188ecbf4d99266ef2a10"} err="failed to get container status \"9325b01f25a16be5ee2f74e151437bf0b0144016582a188ecbf4d99266ef2a10\": rpc error: code = NotFound desc = could not find container \"9325b01f25a16be5ee2f74e151437bf0b0144016582a188ecbf4d99266ef2a10\": container with ID starting with 9325b01f25a16be5ee2f74e151437bf0b0144016582a188ecbf4d99266ef2a10 not found: ID does not exist" Oct 31 01:15:55.538313 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:15:55.538276 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-abe7bd-openshift-default-7d97f8b7c75x2j7"] Oct 31 01:15:55.544403 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:15:55.544356 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-abe7bd-openshift-default-7d97f8b7c75x2j7"] Oct 31 01:15:55.654311 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:15:55.654274 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fb9a8e6-d32e-42d6-8348-5249d6a655f4" path="/var/lib/kubelet/pods/8fb9a8e6-d32e-42d6-8348-5249d6a655f4/volumes" Oct 31 01:15:57.279783 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:15:57.279758 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-abe7bd-dff668477pzlvc" Oct 31 01:15:57.372035 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:15:57.371944 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7a9db10-c5b4-4a95-b67a-84a98fbbb553-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7a9db10-c5b4-4a95-b67a-84a98fbbb553-plugins-config-volume\") pod \"e7a9db10-c5b4-4a95-b67a-84a98fbbb553\" (UID: \"e7a9db10-c5b4-4a95-b67a-84a98fbbb553\") " Oct 31 01:15:57.372035 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:15:57.371996 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7a9db10-c5b4-4a95-b67a-84a98fbbb553-kube-api-access-4flrn\" (UniqueName: \"kubernetes.io/projected/e7a9db10-c5b4-4a95-b67a-84a98fbbb553-kube-api-access-4flrn\") pod \"e7a9db10-c5b4-4a95-b67a-84a98fbbb553\" (UID: \"e7a9db10-c5b4-4a95-b67a-84a98fbbb553\") " Oct 31 01:15:57.372467 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:15:57.372424 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7a9db10-c5b4-4a95-b67a-84a98fbbb553-plugins-config-volume" pod "e7a9db10-c5b4-4a95-b67a-84a98fbbb553" (UID: "e7a9db10-c5b4-4a95-b67a-84a98fbbb553"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:15:57.374276 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:15:57.374247 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7a9db10-c5b4-4a95-b67a-84a98fbbb553-kube-api-access-4flrn" pod "e7a9db10-c5b4-4a95-b67a-84a98fbbb553" (UID: "e7a9db10-c5b4-4a95-b67a-84a98fbbb553"). InnerVolumeSpecName "kube-api-access-4flrn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:15:57.382394 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:15:57.382345 5222 generic.go:356] "Generic (PLEG): container finished" podID="e7a9db10-c5b4-4a95-b67a-84a98fbbb553" containerID="5c26d853c675b26592f10adc2cabb9e318b5251cdf72a610554c2f10a1a583d1" exitCode=0 Oct 31 01:15:57.382512 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:15:57.382429 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-abe7bd-dff668477pzlvc" Oct 31 01:15:57.382565 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:15:57.382426 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-abe7bd-dff668477pzlvc" event={"ID":"e7a9db10-c5b4-4a95-b67a-84a98fbbb553","Type":"ContainerDied","Data":"5c26d853c675b26592f10adc2cabb9e318b5251cdf72a610554c2f10a1a583d1"} Oct 31 01:15:57.382600 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:15:57.382585 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-abe7bd-dff668477pzlvc" event={"ID":"e7a9db10-c5b4-4a95-b67a-84a98fbbb553","Type":"ContainerDied","Data":"8e3856997f12c7b931803f2d93cf7b930281b42e43188c4b583e22772c486221"} Oct 31 01:15:57.382647 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:15:57.382614 5222 scope.go:122] "RemoveContainer" containerID="5c26d853c675b26592f10adc2cabb9e318b5251cdf72a610554c2f10a1a583d1" Oct 31 01:15:57.392294 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:15:57.392274 5222 scope.go:122] "RemoveContainer" containerID="5c26d853c675b26592f10adc2cabb9e318b5251cdf72a610554c2f10a1a583d1" Oct 31 01:15:57.392652 ip-10-0-137-56 kubenswrapper[5222]: E1031 01:15:57.392632 5222 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c26d853c675b26592f10adc2cabb9e318b5251cdf72a610554c2f10a1a583d1\": container with ID starting with 5c26d853c675b26592f10adc2cabb9e318b5251cdf72a610554c2f10a1a583d1 not found: ID does not exist" containerID="5c26d853c675b26592f10adc2cabb9e318b5251cdf72a610554c2f10a1a583d1" Oct 31 01:15:57.392726 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:15:57.392662 5222 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c26d853c675b26592f10adc2cabb9e318b5251cdf72a610554c2f10a1a583d1"} err="failed to get container status \"5c26d853c675b26592f10adc2cabb9e318b5251cdf72a610554c2f10a1a583d1\": rpc error: code = NotFound desc = could not find container \"5c26d853c675b26592f10adc2cabb9e318b5251cdf72a610554c2f10a1a583d1\": container with ID starting with 5c26d853c675b26592f10adc2cabb9e318b5251cdf72a610554c2f10a1a583d1 not found: ID does not exist" Oct 31 01:15:57.472767 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:15:57.472736 5222 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7a9db10-c5b4-4a95-b67a-84a98fbbb553-plugins-config-volume\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:57.472767 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:15:57.472764 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4flrn\" (UniqueName: \"kubernetes.io/projected/e7a9db10-c5b4-4a95-b67a-84a98fbbb553-kube-api-access-4flrn\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:57.511598 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:15:57.511558 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-abe7bd-dff668477pzlvc"] Oct 31 01:15:57.515484 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:15:57.515453 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-abe7bd-dff668477pzlvc"] Oct 31 01:15:57.654210 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:15:57.654132 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7a9db10-c5b4-4a95-b67a-84a98fbbb553" path="/var/lib/kubelet/pods/e7a9db10-c5b4-4a95-b67a-84a98fbbb553/volumes" Oct 31 01:16:01.211619 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:16:01.211575 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-79tk8"] Oct 31 01:16:05.889867 ip-10-0-137-56 kubenswrapper[5222]: E1031 01:16:05.889821 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 01:16:35.889685 ip-10-0-137-56 kubenswrapper[5222]: E1031 01:16:35.889645 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 01:16:37.828331 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:16:37.828292 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-df3d3673-kserve-b4ccb7bfc-cb96s"] Oct 31 01:16:37.828779 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:16:37.828682 5222 cpu_manager.go:414] "containerMap: removing container" podUID="e7a9db10-c5b4-4a95-b67a-84a98fbbb553" containerName="payload-processing" Oct 31 01:16:37.828779 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:16:37.828695 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7a9db10-c5b4-4a95-b67a-84a98fbbb553" containerName="payload-processing" Oct 31 01:16:37.828779 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:16:37.828708 5222 cpu_manager.go:414] "containerMap: removing container" podUID="1eadd23c-6205-43d0-8483-690e92292e64" containerName="cleanup" Oct 31 01:16:37.828779 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:16:37.828713 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1eadd23c-6205-43d0-8483-690e92292e64" containerName="cleanup" Oct 31 01:16:37.828779 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:16:37.828736 5222 cpu_manager.go:414] "containerMap: removing container" podUID="1eadd23c-6205-43d0-8483-690e92292e64" containerName="cleanup" Oct 31 01:16:37.828779 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:16:37.828741 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1eadd23c-6205-43d0-8483-690e92292e64" containerName="cleanup" Oct 31 01:16:37.828779 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:16:37.828753 5222 cpu_manager.go:414] "containerMap: removing container" podUID="8fb9a8e6-d32e-42d6-8348-5249d6a655f4" containerName="istio-proxy" Oct 31 01:16:37.828779 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:16:37.828757 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fb9a8e6-d32e-42d6-8348-5249d6a655f4" containerName="istio-proxy" Oct 31 01:16:37.828779 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:16:37.828764 5222 cpu_manager.go:414] "containerMap: removing container" podUID="8fb9a8e6-d32e-42d6-8348-5249d6a655f4" containerName="istio-proxy" Oct 31 01:16:37.828779 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:16:37.828769 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fb9a8e6-d32e-42d6-8348-5249d6a655f4" containerName="istio-proxy" Oct 31 01:16:37.829102 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:16:37.828822 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="1eadd23c-6205-43d0-8483-690e92292e64" containerName="cleanup" Oct 31 01:16:37.829102 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:16:37.828831 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fb9a8e6-d32e-42d6-8348-5249d6a655f4" containerName="istio-proxy" Oct 31 01:16:37.829102 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:16:37.828837 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fb9a8e6-d32e-42d6-8348-5249d6a655f4" containerName="istio-proxy" Oct 31 01:16:37.829102 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:16:37.828842 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="1eadd23c-6205-43d0-8483-690e92292e64" containerName="cleanup" Oct 31 01:16:37.829102 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:16:37.828848 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7a9db10-c5b4-4a95-b67a-84a98fbbb553" containerName="payload-processing" Oct 31 01:16:37.829102 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:16:37.828854 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="1eadd23c-6205-43d0-8483-690e92292e64" containerName="cleanup" Oct 31 01:16:37.831935 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:16:37.831917 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-df3d3673-kserve-b4ccb7bfc-cb96s" Oct 31 01:16:37.888223 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:16:37.888189 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-df3d3673-kserve-b4ccb7bfc-cb96s"] Oct 31 01:16:37.932526 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:16:37.932487 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a1400cf-93a5-4be4-9504-b04ec9d64c86-home\") pod \"e2e-same-id-b-df3d3673-kserve-b4ccb7bfc-cb96s\" (UID: \"8a1400cf-93a5-4be4-9504-b04ec9d64c86\") " pod="llm/e2e-same-id-b-df3d3673-kserve-b4ccb7bfc-cb96s" Oct 31 01:16:37.932708 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:16:37.932563 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcmhh\" (UniqueName: \"kubernetes.io/projected/8a1400cf-93a5-4be4-9504-b04ec9d64c86-kube-api-access-jcmhh\") pod \"e2e-same-id-b-df3d3673-kserve-b4ccb7bfc-cb96s\" (UID: \"8a1400cf-93a5-4be4-9504-b04ec9d64c86\") " pod="llm/e2e-same-id-b-df3d3673-kserve-b4ccb7bfc-cb96s" Oct 31 01:16:37.932708 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:16:37.932665 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a1400cf-93a5-4be4-9504-b04ec9d64c86-tmp-dir\") pod \"e2e-same-id-b-df3d3673-kserve-b4ccb7bfc-cb96s\" (UID: \"8a1400cf-93a5-4be4-9504-b04ec9d64c86\") " pod="llm/e2e-same-id-b-df3d3673-kserve-b4ccb7bfc-cb96s" Oct 31 01:16:37.932708 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:16:37.932698 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a1400cf-93a5-4be4-9504-b04ec9d64c86-tls-certs\") pod \"e2e-same-id-b-df3d3673-kserve-b4ccb7bfc-cb96s\" (UID: \"8a1400cf-93a5-4be4-9504-b04ec9d64c86\") " pod="llm/e2e-same-id-b-df3d3673-kserve-b4ccb7bfc-cb96s" Oct 31 01:16:37.932827 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:16:37.932722 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a1400cf-93a5-4be4-9504-b04ec9d64c86-dshm\") pod \"e2e-same-id-b-df3d3673-kserve-b4ccb7bfc-cb96s\" (UID: \"8a1400cf-93a5-4be4-9504-b04ec9d64c86\") " pod="llm/e2e-same-id-b-df3d3673-kserve-b4ccb7bfc-cb96s" Oct 31 01:16:37.932863 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:16:37.932822 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a1400cf-93a5-4be4-9504-b04ec9d64c86-model-cache\") pod \"e2e-same-id-b-df3d3673-kserve-b4ccb7bfc-cb96s\" (UID: \"8a1400cf-93a5-4be4-9504-b04ec9d64c86\") " pod="llm/e2e-same-id-b-df3d3673-kserve-b4ccb7bfc-cb96s" Oct 31 01:16:37.936935 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:16:37.936916 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-df3d3673-kserve-self-signed-certs\"" Oct 31 01:16:38.034097 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:16:38.034051 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcmhh\" (UniqueName: \"kubernetes.io/projected/8a1400cf-93a5-4be4-9504-b04ec9d64c86-kube-api-access-jcmhh\") pod \"e2e-same-id-b-df3d3673-kserve-b4ccb7bfc-cb96s\" (UID: \"8a1400cf-93a5-4be4-9504-b04ec9d64c86\") " pod="llm/e2e-same-id-b-df3d3673-kserve-b4ccb7bfc-cb96s" Oct 31 01:16:38.034310 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:16:38.034138 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a1400cf-93a5-4be4-9504-b04ec9d64c86-tmp-dir\") pod \"e2e-same-id-b-df3d3673-kserve-b4ccb7bfc-cb96s\" (UID: \"8a1400cf-93a5-4be4-9504-b04ec9d64c86\") " pod="llm/e2e-same-id-b-df3d3673-kserve-b4ccb7bfc-cb96s" Oct 31 01:16:38.034310 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:16:38.034179 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a1400cf-93a5-4be4-9504-b04ec9d64c86-tls-certs\") pod \"e2e-same-id-b-df3d3673-kserve-b4ccb7bfc-cb96s\" (UID: \"8a1400cf-93a5-4be4-9504-b04ec9d64c86\") " pod="llm/e2e-same-id-b-df3d3673-kserve-b4ccb7bfc-cb96s" Oct 31 01:16:38.034310 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:16:38.034231 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a1400cf-93a5-4be4-9504-b04ec9d64c86-dshm\") pod \"e2e-same-id-b-df3d3673-kserve-b4ccb7bfc-cb96s\" (UID: \"8a1400cf-93a5-4be4-9504-b04ec9d64c86\") " pod="llm/e2e-same-id-b-df3d3673-kserve-b4ccb7bfc-cb96s" Oct 31 01:16:38.034310 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:16:38.034289 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a1400cf-93a5-4be4-9504-b04ec9d64c86-model-cache\") pod \"e2e-same-id-b-df3d3673-kserve-b4ccb7bfc-cb96s\" (UID: \"8a1400cf-93a5-4be4-9504-b04ec9d64c86\") " pod="llm/e2e-same-id-b-df3d3673-kserve-b4ccb7bfc-cb96s" Oct 31 01:16:38.034570 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:16:38.034349 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a1400cf-93a5-4be4-9504-b04ec9d64c86-home\") pod \"e2e-same-id-b-df3d3673-kserve-b4ccb7bfc-cb96s\" (UID: \"8a1400cf-93a5-4be4-9504-b04ec9d64c86\") " pod="llm/e2e-same-id-b-df3d3673-kserve-b4ccb7bfc-cb96s" Oct 31 01:16:38.034658 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:16:38.034636 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a1400cf-93a5-4be4-9504-b04ec9d64c86-tmp-dir\") pod \"e2e-same-id-b-df3d3673-kserve-b4ccb7bfc-cb96s\" (UID: \"8a1400cf-93a5-4be4-9504-b04ec9d64c86\") " pod="llm/e2e-same-id-b-df3d3673-kserve-b4ccb7bfc-cb96s" Oct 31 01:16:38.034715 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:16:38.034690 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a1400cf-93a5-4be4-9504-b04ec9d64c86-model-cache\") pod \"e2e-same-id-b-df3d3673-kserve-b4ccb7bfc-cb96s\" (UID: \"8a1400cf-93a5-4be4-9504-b04ec9d64c86\") " pod="llm/e2e-same-id-b-df3d3673-kserve-b4ccb7bfc-cb96s" Oct 31 01:16:38.034757 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:16:38.034720 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a1400cf-93a5-4be4-9504-b04ec9d64c86-home\") pod \"e2e-same-id-b-df3d3673-kserve-b4ccb7bfc-cb96s\" (UID: \"8a1400cf-93a5-4be4-9504-b04ec9d64c86\") " pod="llm/e2e-same-id-b-df3d3673-kserve-b4ccb7bfc-cb96s" Oct 31 01:16:38.036927 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:16:38.036898 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a1400cf-93a5-4be4-9504-b04ec9d64c86-dshm\") pod \"e2e-same-id-b-df3d3673-kserve-b4ccb7bfc-cb96s\" (UID: \"8a1400cf-93a5-4be4-9504-b04ec9d64c86\") " pod="llm/e2e-same-id-b-df3d3673-kserve-b4ccb7bfc-cb96s" Oct 31 01:16:38.037048 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:16:38.037026 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a1400cf-93a5-4be4-9504-b04ec9d64c86-tls-certs\") pod \"e2e-same-id-b-df3d3673-kserve-b4ccb7bfc-cb96s\" (UID: \"8a1400cf-93a5-4be4-9504-b04ec9d64c86\") " pod="llm/e2e-same-id-b-df3d3673-kserve-b4ccb7bfc-cb96s" Oct 31 01:16:38.097147 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:16:38.097071 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcmhh\" (UniqueName: \"kubernetes.io/projected/8a1400cf-93a5-4be4-9504-b04ec9d64c86-kube-api-access-jcmhh\") pod \"e2e-same-id-b-df3d3673-kserve-b4ccb7bfc-cb96s\" (UID: \"8a1400cf-93a5-4be4-9504-b04ec9d64c86\") " pod="llm/e2e-same-id-b-df3d3673-kserve-b4ccb7bfc-cb96s" Oct 31 01:16:38.145782 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:16:38.145740 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-df3d3673-kserve-b4ccb7bfc-cb96s" Oct 31 01:16:38.544301 ip-10-0-137-56 kubenswrapper[5222]: W1031 01:16:38.544272 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8a1400cf_93a5_4be4_9504_b04ec9d64c86.slice/crio-f98596a8e58bafeee1a2e4713f17345a4cc7c8c95054c9a343a352fc3ae17539 WatchSource:0}: Error finding container f98596a8e58bafeee1a2e4713f17345a4cc7c8c95054c9a343a352fc3ae17539: Status 404 returned error can't find the container with id f98596a8e58bafeee1a2e4713f17345a4cc7c8c95054c9a343a352fc3ae17539 Oct 31 01:16:38.545240 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:16:38.545210 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-df3d3673-kserve-b4ccb7bfc-cb96s"] Oct 31 01:16:38.630196 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:16:38.630158 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-df3d3673-kserve-b4ccb7bfc-cb96s" event={"ID":"8a1400cf-93a5-4be4-9504-b04ec9d64c86","Type":"ContainerStarted","Data":"571c486d1cdda1c7d95ed02786a6a8862993bf1e8c6611065c53709110589445"} Oct 31 01:16:38.630411 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:16:38.630209 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-df3d3673-kserve-b4ccb7bfc-cb96s" event={"ID":"8a1400cf-93a5-4be4-9504-b04ec9d64c86","Type":"ContainerStarted","Data":"f98596a8e58bafeee1a2e4713f17345a4cc7c8c95054c9a343a352fc3ae17539"} Oct 31 01:16:38.750190 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:16:38.750136 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-df3d3673-kserve-b4ccb7bfc-cb96s" podStartSLOduration=1.75012181 podStartE2EDuration="1.75012181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:16:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:16:38.749706037 +0000 UTC m=+3691.220875786" watchObservedRunningTime="2026-10-31 01:16:38.75012181 +0000 UTC m=+3691.221291555" Oct 31 01:16:39.145903 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:16:39.145862 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-df3d3673-kserve-b4ccb7bfc-cb96s" Oct 31 01:16:40.917278 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:16:40.917241 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-79tk8"] Oct 31 01:16:48.146636 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:16:48.146541 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-df3d3673-kserve-b4ccb7bfc-cb96s" Oct 31 01:16:48.159085 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:16:48.159052 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-df3d3673-kserve-b4ccb7bfc-cb96s" Oct 31 01:16:48.758405 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:16:48.758357 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-df3d3673-kserve-b4ccb7bfc-cb96s" Oct 31 01:17:05.889685 ip-10-0-137-56 kubenswrapper[5222]: E1031 01:17:05.889636 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 01:17:09.074764 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:09.074728 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-spjbk"] Oct 31 01:17:09.075176 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:09.075131 5222 cpu_manager.go:414] "containerMap: removing container" podUID="1eadd23c-6205-43d0-8483-690e92292e64" containerName="cleanup" Oct 31 01:17:09.075176 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:09.075145 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1eadd23c-6205-43d0-8483-690e92292e64" containerName="cleanup" Oct 31 01:17:09.078321 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:09.078302 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-spjbk" Oct 31 01:17:09.089317 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:09.089284 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76cad17e-7333-45cb-96dd-6a06ad834a06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-spjbk\" (UID: \"76cad17e-7333-45cb-96dd-6a06ad834a06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-spjbk" Oct 31 01:17:09.089476 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:09.089427 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76cad17e-7333-45cb-96dd-6a06ad834a06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-spjbk\" (UID: \"76cad17e-7333-45cb-96dd-6a06ad834a06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-spjbk" Oct 31 01:17:09.089476 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:09.089467 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hj7tk\" (UniqueName: \"kubernetes.io/projected/76cad17e-7333-45cb-96dd-6a06ad834a06-kube-api-access-hj7tk\") pod \"maas-api-revoke-sub-452b6d7b-spjbk\" (UID: \"76cad17e-7333-45cb-96dd-6a06ad834a06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-spjbk" Oct 31 01:17:09.135415 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:09.135381 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-spjbk"] Oct 31 01:17:09.184214 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:09.184175 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:17:09.184214 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:09.184210 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:17:09.184512 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:09.184246 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfz9z\"" Oct 31 01:17:09.184512 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:09.184285 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:17:09.190795 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:09.190765 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76cad17e-7333-45cb-96dd-6a06ad834a06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-spjbk\" (UID: \"76cad17e-7333-45cb-96dd-6a06ad834a06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-spjbk" Oct 31 01:17:09.190997 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:09.190834 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76cad17e-7333-45cb-96dd-6a06ad834a06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-spjbk\" (UID: \"76cad17e-7333-45cb-96dd-6a06ad834a06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-spjbk" Oct 31 01:17:09.190997 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:09.190870 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hj7tk\" (UniqueName: \"kubernetes.io/projected/76cad17e-7333-45cb-96dd-6a06ad834a06-kube-api-access-hj7tk\") pod \"maas-api-revoke-sub-452b6d7b-spjbk\" (UID: \"76cad17e-7333-45cb-96dd-6a06ad834a06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-spjbk" Oct 31 01:17:09.191550 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:09.191532 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76cad17e-7333-45cb-96dd-6a06ad834a06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-spjbk\" (UID: \"76cad17e-7333-45cb-96dd-6a06ad834a06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-spjbk" Oct 31 01:17:09.193385 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:09.193337 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76cad17e-7333-45cb-96dd-6a06ad834a06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-spjbk\" (UID: \"76cad17e-7333-45cb-96dd-6a06ad834a06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-spjbk" Oct 31 01:17:09.254361 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:09.254328 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hj7tk\" (UniqueName: \"kubernetes.io/projected/76cad17e-7333-45cb-96dd-6a06ad834a06-kube-api-access-hj7tk\") pod \"maas-api-revoke-sub-452b6d7b-spjbk\" (UID: \"76cad17e-7333-45cb-96dd-6a06ad834a06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-spjbk" Oct 31 01:17:09.391585 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:09.391495 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-spjbk" Oct 31 01:17:09.764135 ip-10-0-137-56 kubenswrapper[5222]: W1031 01:17:09.764105 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76cad17e_7333_45cb_96dd_6a06ad834a06.slice/crio-b7bc5a3c26b9bfbee6ecd7bb7d70210f98017af2184bb614d531c407923d8416 WatchSource:0}: Error finding container b7bc5a3c26b9bfbee6ecd7bb7d70210f98017af2184bb614d531c407923d8416: Status 404 returned error can't find the container with id b7bc5a3c26b9bfbee6ecd7bb7d70210f98017af2184bb614d531c407923d8416 Oct 31 01:17:09.766064 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:09.766026 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-spjbk"] Oct 31 01:17:09.846310 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:09.846273 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-spjbk" event={"ID":"76cad17e-7333-45cb-96dd-6a06ad834a06","Type":"ContainerStarted","Data":"b7bc5a3c26b9bfbee6ecd7bb7d70210f98017af2184bb614d531c407923d8416"} Oct 31 01:17:10.637439 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:10.634759 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-79tk8"] Oct 31 01:17:10.853120 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:10.853021 5222 generic.go:356] "Generic (PLEG): container finished" podID="76cad17e-7333-45cb-96dd-6a06ad834a06" containerID="d03e6a4e6c17e79109e6686ba35e494b0de851a5b66d0a3afe147df3f366a2fb" exitCode=0 Oct 31 01:17:10.853120 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:10.853098 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-spjbk" event={"ID":"76cad17e-7333-45cb-96dd-6a06ad834a06","Type":"ContainerDied","Data":"d03e6a4e6c17e79109e6686ba35e494b0de851a5b66d0a3afe147df3f366a2fb"} Oct 31 01:17:12.215319 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:12.215273 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-spjbk" Oct 31 01:17:12.280412 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:12.279783 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76cad17e-7333-45cb-96dd-6a06ad834a06-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76cad17e-7333-45cb-96dd-6a06ad834a06-maas-api-xks-ca\") pod \"76cad17e-7333-45cb-96dd-6a06ad834a06\" (UID: \"76cad17e-7333-45cb-96dd-6a06ad834a06\") " Oct 31 01:17:12.280412 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:12.279853 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76cad17e-7333-45cb-96dd-6a06ad834a06-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76cad17e-7333-45cb-96dd-6a06ad834a06-maas-api-openshift-ca\") pod \"76cad17e-7333-45cb-96dd-6a06ad834a06\" (UID: \"76cad17e-7333-45cb-96dd-6a06ad834a06\") " Oct 31 01:17:12.280412 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:12.279892 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76cad17e-7333-45cb-96dd-6a06ad834a06-kube-api-access-hj7tk\" (UniqueName: \"kubernetes.io/projected/76cad17e-7333-45cb-96dd-6a06ad834a06-kube-api-access-hj7tk\") pod \"76cad17e-7333-45cb-96dd-6a06ad834a06\" (UID: \"76cad17e-7333-45cb-96dd-6a06ad834a06\") " Oct 31 01:17:12.282108 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:12.281936 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76cad17e-7333-45cb-96dd-6a06ad834a06-maas-api-openshift-ca" pod "76cad17e-7333-45cb-96dd-6a06ad834a06" (UID: "76cad17e-7333-45cb-96dd-6a06ad834a06"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:17:12.306934 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:12.306872 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76cad17e-7333-45cb-96dd-6a06ad834a06-kube-api-access-hj7tk" pod "76cad17e-7333-45cb-96dd-6a06ad834a06" (UID: "76cad17e-7333-45cb-96dd-6a06ad834a06"). InnerVolumeSpecName "kube-api-access-hj7tk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:17:12.313395 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:12.310866 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76cad17e-7333-45cb-96dd-6a06ad834a06-maas-api-xks-ca" pod "76cad17e-7333-45cb-96dd-6a06ad834a06" (UID: "76cad17e-7333-45cb-96dd-6a06ad834a06"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:17:12.388496 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:12.381521 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76cad17e-7333-45cb-96dd-6a06ad834a06-maas-api-xks-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:17:12.388854 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:12.388752 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76cad17e-7333-45cb-96dd-6a06ad834a06-maas-api-openshift-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:17:12.388854 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:12.388819 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hj7tk\" (UniqueName: \"kubernetes.io/projected/76cad17e-7333-45cb-96dd-6a06ad834a06-kube-api-access-hj7tk\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:17:12.913397 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:12.909868 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-spjbk" event={"ID":"76cad17e-7333-45cb-96dd-6a06ad834a06","Type":"ContainerDied","Data":"b7bc5a3c26b9bfbee6ecd7bb7d70210f98017af2184bb614d531c407923d8416"} Oct 31 01:17:12.913397 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:12.909942 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b7bc5a3c26b9bfbee6ecd7bb7d70210f98017af2184bb614d531c407923d8416" Oct 31 01:17:12.913397 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:12.910016 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-spjbk" Oct 31 01:17:13.123151 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:13.123101 5222 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-jd2w5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.250:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 01:17:13.123490 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:13.123456 5222 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jd2w5" podUID="d2ec2aa7-9ec8-41bd-b10f-e5ef7ecbd803" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.250:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 01:17:24.833965 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:24.833919 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-df3d3673-kserve-b4ccb7bfc-cb96s"] Oct 31 01:17:26.392945 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:26.392911 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-79tk8"] Oct 31 01:17:28.106535 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:28.106490 5222 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-jd2w5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.250:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 01:17:28.107184 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:28.107148 5222 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jd2w5" podUID="d2ec2aa7-9ec8-41bd-b10f-e5ef7ecbd803" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.250:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 01:17:29.305166 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:29.305134 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-92zr8"] Oct 31 01:17:29.305584 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:29.305531 5222 cpu_manager.go:414] "containerMap: removing container" podUID="76cad17e-7333-45cb-96dd-6a06ad834a06" containerName="revoke-keys" Oct 31 01:17:29.305584 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:29.305543 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76cad17e-7333-45cb-96dd-6a06ad834a06" containerName="revoke-keys" Oct 31 01:17:29.305676 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:29.305622 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="76cad17e-7333-45cb-96dd-6a06ad834a06" containerName="revoke-keys" Oct 31 01:17:29.308841 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:29.308819 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-92zr8" Oct 31 01:17:29.344907 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:29.344872 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsfh2\" (UniqueName: \"kubernetes.io/projected/653bf085-9f4e-4e91-adb7-44799c8f400f-kube-api-access-tsfh2\") pod \"maas-api-revoke-sub-3e31a506-92zr8\" (UID: \"653bf085-9f4e-4e91-adb7-44799c8f400f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-92zr8" Oct 31 01:17:29.344907 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:29.344913 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/653bf085-9f4e-4e91-adb7-44799c8f400f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-92zr8\" (UID: \"653bf085-9f4e-4e91-adb7-44799c8f400f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-92zr8" Oct 31 01:17:29.345162 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:29.345062 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/653bf085-9f4e-4e91-adb7-44799c8f400f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-92zr8\" (UID: \"653bf085-9f4e-4e91-adb7-44799c8f400f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-92zr8" Oct 31 01:17:29.363686 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:29.363642 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-92zr8"] Oct 31 01:17:29.411936 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:29.411892 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:17:29.411936 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:29.411911 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:17:29.412158 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:29.411957 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfz9z\"" Oct 31 01:17:29.413559 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:29.413542 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:17:29.445959 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:29.445917 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsfh2\" (UniqueName: \"kubernetes.io/projected/653bf085-9f4e-4e91-adb7-44799c8f400f-kube-api-access-tsfh2\") pod \"maas-api-revoke-sub-3e31a506-92zr8\" (UID: \"653bf085-9f4e-4e91-adb7-44799c8f400f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-92zr8" Oct 31 01:17:29.445959 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:29.445964 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/653bf085-9f4e-4e91-adb7-44799c8f400f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-92zr8\" (UID: \"653bf085-9f4e-4e91-adb7-44799c8f400f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-92zr8" Oct 31 01:17:29.446209 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:29.446033 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/653bf085-9f4e-4e91-adb7-44799c8f400f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-92zr8\" (UID: \"653bf085-9f4e-4e91-adb7-44799c8f400f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-92zr8" Oct 31 01:17:29.446723 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:29.446696 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/653bf085-9f4e-4e91-adb7-44799c8f400f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-92zr8\" (UID: \"653bf085-9f4e-4e91-adb7-44799c8f400f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-92zr8" Oct 31 01:17:29.448643 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:29.448626 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/653bf085-9f4e-4e91-adb7-44799c8f400f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-92zr8\" (UID: \"653bf085-9f4e-4e91-adb7-44799c8f400f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-92zr8" Oct 31 01:17:29.508939 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:29.508903 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsfh2\" (UniqueName: \"kubernetes.io/projected/653bf085-9f4e-4e91-adb7-44799c8f400f-kube-api-access-tsfh2\") pod \"maas-api-revoke-sub-3e31a506-92zr8\" (UID: \"653bf085-9f4e-4e91-adb7-44799c8f400f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-92zr8" Oct 31 01:17:29.639252 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:29.639158 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-92zr8" Oct 31 01:17:30.020494 ip-10-0-137-56 kubenswrapper[5222]: W1031 01:17:30.020461 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod653bf085_9f4e_4e91_adb7_44799c8f400f.slice/crio-4c2045218d9b059cfd06d35a43f560482f54f44235d75bfec7dafb1c73340ddb WatchSource:0}: Error finding container 4c2045218d9b059cfd06d35a43f560482f54f44235d75bfec7dafb1c73340ddb: Status 404 returned error can't find the container with id 4c2045218d9b059cfd06d35a43f560482f54f44235d75bfec7dafb1c73340ddb Oct 31 01:17:30.021715 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:30.021686 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-92zr8"] Oct 31 01:17:30.034848 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:30.034823 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-92zr8" event={"ID":"653bf085-9f4e-4e91-adb7-44799c8f400f","Type":"ContainerStarted","Data":"4c2045218d9b059cfd06d35a43f560482f54f44235d75bfec7dafb1c73340ddb"} Oct 31 01:17:31.041469 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:31.041430 5222 generic.go:356] "Generic (PLEG): container finished" podID="653bf085-9f4e-4e91-adb7-44799c8f400f" containerID="07c35875ca71145f6649993960092fc81fe82d67934a256a21db08fff88b15b6" exitCode=0 Oct 31 01:17:31.041883 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:31.041520 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-92zr8" event={"ID":"653bf085-9f4e-4e91-adb7-44799c8f400f","Type":"ContainerDied","Data":"07c35875ca71145f6649993960092fc81fe82d67934a256a21db08fff88b15b6"} Oct 31 01:17:32.194199 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:32.194171 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-92zr8" Oct 31 01:17:32.273626 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:32.273583 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/653bf085-9f4e-4e91-adb7-44799c8f400f-kube-api-access-tsfh2\" (UniqueName: \"kubernetes.io/projected/653bf085-9f4e-4e91-adb7-44799c8f400f-kube-api-access-tsfh2\") pod \"653bf085-9f4e-4e91-adb7-44799c8f400f\" (UID: \"653bf085-9f4e-4e91-adb7-44799c8f400f\") " Oct 31 01:17:32.273626 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:32.273631 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/653bf085-9f4e-4e91-adb7-44799c8f400f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/653bf085-9f4e-4e91-adb7-44799c8f400f-maas-api-openshift-ca\") pod \"653bf085-9f4e-4e91-adb7-44799c8f400f\" (UID: \"653bf085-9f4e-4e91-adb7-44799c8f400f\") " Oct 31 01:17:32.273914 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:32.273701 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/653bf085-9f4e-4e91-adb7-44799c8f400f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/653bf085-9f4e-4e91-adb7-44799c8f400f-maas-api-xks-ca\") pod \"653bf085-9f4e-4e91-adb7-44799c8f400f\" (UID: \"653bf085-9f4e-4e91-adb7-44799c8f400f\") " Oct 31 01:17:32.274095 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:32.274064 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/653bf085-9f4e-4e91-adb7-44799c8f400f-maas-api-openshift-ca" pod "653bf085-9f4e-4e91-adb7-44799c8f400f" (UID: "653bf085-9f4e-4e91-adb7-44799c8f400f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:17:32.275988 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:32.275954 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/653bf085-9f4e-4e91-adb7-44799c8f400f-kube-api-access-tsfh2" pod "653bf085-9f4e-4e91-adb7-44799c8f400f" (UID: "653bf085-9f4e-4e91-adb7-44799c8f400f"). InnerVolumeSpecName "kube-api-access-tsfh2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:17:32.276082 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:32.276037 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/653bf085-9f4e-4e91-adb7-44799c8f400f-maas-api-xks-ca" pod "653bf085-9f4e-4e91-adb7-44799c8f400f" (UID: "653bf085-9f4e-4e91-adb7-44799c8f400f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:17:32.375524 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:32.375434 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/653bf085-9f4e-4e91-adb7-44799c8f400f-maas-api-xks-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:17:32.375524 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:32.375468 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsfh2\" (UniqueName: \"kubernetes.io/projected/653bf085-9f4e-4e91-adb7-44799c8f400f-kube-api-access-tsfh2\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:17:32.375524 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:32.375481 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/653bf085-9f4e-4e91-adb7-44799c8f400f-maas-api-openshift-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:17:33.052751 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:33.052721 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-92zr8" Oct 31 01:17:33.052942 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:33.052749 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-92zr8" event={"ID":"653bf085-9f4e-4e91-adb7-44799c8f400f","Type":"ContainerDied","Data":"4c2045218d9b059cfd06d35a43f560482f54f44235d75bfec7dafb1c73340ddb"} Oct 31 01:17:33.052942 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:33.052794 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c2045218d9b059cfd06d35a43f560482f54f44235d75bfec7dafb1c73340ddb" Oct 31 01:17:36.021399 ip-10-0-137-56 kubenswrapper[5222]: E1031 01:17:36.018830 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 01:17:39.224858 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:39.224822 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-79tk8"] Oct 31 01:17:39.855108 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:39.855067 5222 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-df3d3673-kserve-b4ccb7bfc-cb96s" podUID="8a1400cf-93a5-4be4-9504-b04ec9d64c86" containerName="main" containerID="cri-o://571c486d1cdda1c7d95ed02786a6a8862993bf1e8c6611065c53709110589445" gracePeriod=45 Oct 31 01:17:40.111599 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:40.111527 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-df3d3673-kserve-b4ccb7bfc-cb96s" Oct 31 01:17:40.134518 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:40.134483 5222 generic.go:356] "Generic (PLEG): container finished" podID="8a1400cf-93a5-4be4-9504-b04ec9d64c86" containerID="571c486d1cdda1c7d95ed02786a6a8862993bf1e8c6611065c53709110589445" exitCode=0 Oct 31 01:17:40.134703 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:40.134533 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-df3d3673-kserve-b4ccb7bfc-cb96s" Oct 31 01:17:40.134703 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:40.134559 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-df3d3673-kserve-b4ccb7bfc-cb96s" event={"ID":"8a1400cf-93a5-4be4-9504-b04ec9d64c86","Type":"ContainerDied","Data":"571c486d1cdda1c7d95ed02786a6a8862993bf1e8c6611065c53709110589445"} Oct 31 01:17:40.134703 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:40.134598 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-df3d3673-kserve-b4ccb7bfc-cb96s" event={"ID":"8a1400cf-93a5-4be4-9504-b04ec9d64c86","Type":"ContainerDied","Data":"f98596a8e58bafeee1a2e4713f17345a4cc7c8c95054c9a343a352fc3ae17539"} Oct 31 01:17:40.134703 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:40.134622 5222 scope.go:122] "RemoveContainer" containerID="571c486d1cdda1c7d95ed02786a6a8862993bf1e8c6611065c53709110589445" Oct 31 01:17:40.143233 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:40.143201 5222 scope.go:122] "RemoveContainer" containerID="571c486d1cdda1c7d95ed02786a6a8862993bf1e8c6611065c53709110589445" Oct 31 01:17:40.143653 ip-10-0-137-56 kubenswrapper[5222]: E1031 01:17:40.143630 5222 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"571c486d1cdda1c7d95ed02786a6a8862993bf1e8c6611065c53709110589445\": container with ID starting with 571c486d1cdda1c7d95ed02786a6a8862993bf1e8c6611065c53709110589445 not found: ID does not exist" containerID="571c486d1cdda1c7d95ed02786a6a8862993bf1e8c6611065c53709110589445" Oct 31 01:17:40.143753 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:40.143668 5222 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"571c486d1cdda1c7d95ed02786a6a8862993bf1e8c6611065c53709110589445"} err="failed to get container status \"571c486d1cdda1c7d95ed02786a6a8862993bf1e8c6611065c53709110589445\": rpc error: code = NotFound desc = could not find container \"571c486d1cdda1c7d95ed02786a6a8862993bf1e8c6611065c53709110589445\": container with ID starting with 571c486d1cdda1c7d95ed02786a6a8862993bf1e8c6611065c53709110589445 not found: ID does not exist" Oct 31 01:17:40.241969 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:40.241940 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a1400cf-93a5-4be4-9504-b04ec9d64c86-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a1400cf-93a5-4be4-9504-b04ec9d64c86-tmp-dir\") pod \"8a1400cf-93a5-4be4-9504-b04ec9d64c86\" (UID: \"8a1400cf-93a5-4be4-9504-b04ec9d64c86\") " Oct 31 01:17:40.242460 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:40.241980 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a1400cf-93a5-4be4-9504-b04ec9d64c86-home\" (UniqueName: \"kubernetes.io/empty-dir/8a1400cf-93a5-4be4-9504-b04ec9d64c86-home\") pod \"8a1400cf-93a5-4be4-9504-b04ec9d64c86\" (UID: \"8a1400cf-93a5-4be4-9504-b04ec9d64c86\") " Oct 31 01:17:40.242460 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:40.241998 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a1400cf-93a5-4be4-9504-b04ec9d64c86-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a1400cf-93a5-4be4-9504-b04ec9d64c86-dshm\") pod \"8a1400cf-93a5-4be4-9504-b04ec9d64c86\" (UID: \"8a1400cf-93a5-4be4-9504-b04ec9d64c86\") " Oct 31 01:17:40.242460 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:40.242032 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a1400cf-93a5-4be4-9504-b04ec9d64c86-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a1400cf-93a5-4be4-9504-b04ec9d64c86-model-cache\") pod \"8a1400cf-93a5-4be4-9504-b04ec9d64c86\" (UID: \"8a1400cf-93a5-4be4-9504-b04ec9d64c86\") " Oct 31 01:17:40.242460 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:40.242148 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a1400cf-93a5-4be4-9504-b04ec9d64c86-tls-certs\" (UniqueName: \"kubernetes.io/secret/8a1400cf-93a5-4be4-9504-b04ec9d64c86-tls-certs\") pod \"8a1400cf-93a5-4be4-9504-b04ec9d64c86\" (UID: \"8a1400cf-93a5-4be4-9504-b04ec9d64c86\") " Oct 31 01:17:40.242460 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:40.242188 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a1400cf-93a5-4be4-9504-b04ec9d64c86-kube-api-access-jcmhh\" (UniqueName: \"kubernetes.io/projected/8a1400cf-93a5-4be4-9504-b04ec9d64c86-kube-api-access-jcmhh\") pod \"8a1400cf-93a5-4be4-9504-b04ec9d64c86\" (UID: \"8a1400cf-93a5-4be4-9504-b04ec9d64c86\") " Oct 31 01:17:40.242460 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:40.242302 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a1400cf-93a5-4be4-9504-b04ec9d64c86-home" pod "8a1400cf-93a5-4be4-9504-b04ec9d64c86" (UID: "8a1400cf-93a5-4be4-9504-b04ec9d64c86"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:17:40.242460 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:40.242295 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a1400cf-93a5-4be4-9504-b04ec9d64c86-tmp-dir" pod "8a1400cf-93a5-4be4-9504-b04ec9d64c86" (UID: "8a1400cf-93a5-4be4-9504-b04ec9d64c86"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:17:40.242460 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:40.242386 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a1400cf-93a5-4be4-9504-b04ec9d64c86-model-cache" pod "8a1400cf-93a5-4be4-9504-b04ec9d64c86" (UID: "8a1400cf-93a5-4be4-9504-b04ec9d64c86"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:17:40.243076 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:40.242680 5222 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a1400cf-93a5-4be4-9504-b04ec9d64c86-tmp-dir\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:17:40.243076 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:40.242705 5222 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a1400cf-93a5-4be4-9504-b04ec9d64c86-home\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:17:40.243076 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:40.242720 5222 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a1400cf-93a5-4be4-9504-b04ec9d64c86-model-cache\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:17:40.244688 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:40.244662 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a1400cf-93a5-4be4-9504-b04ec9d64c86-tls-certs" pod "8a1400cf-93a5-4be4-9504-b04ec9d64c86" (UID: "8a1400cf-93a5-4be4-9504-b04ec9d64c86"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:17:40.244688 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:40.244674 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a1400cf-93a5-4be4-9504-b04ec9d64c86-dshm" pod "8a1400cf-93a5-4be4-9504-b04ec9d64c86" (UID: "8a1400cf-93a5-4be4-9504-b04ec9d64c86"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:17:40.244907 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:40.244701 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a1400cf-93a5-4be4-9504-b04ec9d64c86-kube-api-access-jcmhh" pod "8a1400cf-93a5-4be4-9504-b04ec9d64c86" (UID: "8a1400cf-93a5-4be4-9504-b04ec9d64c86"). InnerVolumeSpecName "kube-api-access-jcmhh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:17:40.343785 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:40.343746 5222 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a1400cf-93a5-4be4-9504-b04ec9d64c86-tls-certs\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:17:40.343785 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:40.343779 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcmhh\" (UniqueName: \"kubernetes.io/projected/8a1400cf-93a5-4be4-9504-b04ec9d64c86-kube-api-access-jcmhh\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:17:40.343785 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:40.343789 5222 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a1400cf-93a5-4be4-9504-b04ec9d64c86-dshm\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:17:40.559050 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:40.559007 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-df3d3673-kserve-b4ccb7bfc-cb96s"] Oct 31 01:17:40.562199 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:40.562170 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-df3d3673-kserve-b4ccb7bfc-cb96s"] Oct 31 01:17:41.658842 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:41.655789 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a1400cf-93a5-4be4-9504-b04ec9d64c86" path="/var/lib/kubelet/pods/8a1400cf-93a5-4be4-9504-b04ec9d64c86/volumes" Oct 31 01:17:49.026839 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:49.026801 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-79tk8"] Oct 31 01:17:59.083065 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:59.083029 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9f6da73-kpj95"] Oct 31 01:17:59.083517 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:59.083440 5222 cpu_manager.go:414] "containerMap: removing container" podUID="653bf085-9f4e-4e91-adb7-44799c8f400f" containerName="revoke-keys" Oct 31 01:17:59.083517 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:59.083452 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="653bf085-9f4e-4e91-adb7-44799c8f400f" containerName="revoke-keys" Oct 31 01:17:59.083517 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:59.083469 5222 cpu_manager.go:414] "containerMap: removing container" podUID="8a1400cf-93a5-4be4-9504-b04ec9d64c86" containerName="main" Oct 31 01:17:59.083517 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:59.083474 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a1400cf-93a5-4be4-9504-b04ec9d64c86" containerName="main" Oct 31 01:17:59.083687 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:59.083531 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="653bf085-9f4e-4e91-adb7-44799c8f400f" containerName="revoke-keys" Oct 31 01:17:59.083687 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:59.083542 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a1400cf-93a5-4be4-9504-b04ec9d64c86" containerName="main" Oct 31 01:17:59.087930 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:59.087906 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9f6da73-kpj95" Oct 31 01:17:59.140754 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:59.140716 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9f6da73-kpj95"] Oct 31 01:17:59.188067 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:59.188030 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k978p\" (UniqueName: \"kubernetes.io/projected/3e361b6c-3b9a-4871-b138-28d6c7c5f47e-kube-api-access-k978p\") pod \"maas-api-revoke-sub-b9f6da73-kpj95\" (UID: \"3e361b6c-3b9a-4871-b138-28d6c7c5f47e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9f6da73-kpj95" Oct 31 01:17:59.188255 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:59.188123 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e361b6c-3b9a-4871-b138-28d6c7c5f47e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b9f6da73-kpj95\" (UID: \"3e361b6c-3b9a-4871-b138-28d6c7c5f47e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9f6da73-kpj95" Oct 31 01:17:59.188255 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:59.188200 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e361b6c-3b9a-4871-b138-28d6c7c5f47e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b9f6da73-kpj95\" (UID: \"3e361b6c-3b9a-4871-b138-28d6c7c5f47e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9f6da73-kpj95" Oct 31 01:17:59.190043 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:59.190003 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:17:59.190043 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:59.190005 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:17:59.190232 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:59.190008 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:17:59.192832 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:59.192811 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfz9z\"" Oct 31 01:17:59.289037 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:59.289005 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e361b6c-3b9a-4871-b138-28d6c7c5f47e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b9f6da73-kpj95\" (UID: \"3e361b6c-3b9a-4871-b138-28d6c7c5f47e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9f6da73-kpj95" Oct 31 01:17:59.289249 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:59.289099 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k978p\" (UniqueName: \"kubernetes.io/projected/3e361b6c-3b9a-4871-b138-28d6c7c5f47e-kube-api-access-k978p\") pod \"maas-api-revoke-sub-b9f6da73-kpj95\" (UID: \"3e361b6c-3b9a-4871-b138-28d6c7c5f47e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9f6da73-kpj95" Oct 31 01:17:59.289249 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:59.289124 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e361b6c-3b9a-4871-b138-28d6c7c5f47e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b9f6da73-kpj95\" (UID: \"3e361b6c-3b9a-4871-b138-28d6c7c5f47e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9f6da73-kpj95" Oct 31 01:17:59.289730 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:59.289696 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e361b6c-3b9a-4871-b138-28d6c7c5f47e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b9f6da73-kpj95\" (UID: \"3e361b6c-3b9a-4871-b138-28d6c7c5f47e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9f6da73-kpj95" Oct 31 01:17:59.291769 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:59.291741 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e361b6c-3b9a-4871-b138-28d6c7c5f47e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b9f6da73-kpj95\" (UID: \"3e361b6c-3b9a-4871-b138-28d6c7c5f47e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9f6da73-kpj95" Oct 31 01:17:59.352305 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:59.352230 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k978p\" (UniqueName: \"kubernetes.io/projected/3e361b6c-3b9a-4871-b138-28d6c7c5f47e-kube-api-access-k978p\") pod \"maas-api-revoke-sub-b9f6da73-kpj95\" (UID: \"3e361b6c-3b9a-4871-b138-28d6c7c5f47e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9f6da73-kpj95" Oct 31 01:17:59.401841 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:59.401801 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9f6da73-kpj95" Oct 31 01:17:59.773220 ip-10-0-137-56 kubenswrapper[5222]: W1031 01:17:59.773190 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e361b6c_3b9a_4871_b138_28d6c7c5f47e.slice/crio-83482605e62b4c12480ac848c7cbedb80be1e8684119a12612696ed6140c1a61 WatchSource:0}: Error finding container 83482605e62b4c12480ac848c7cbedb80be1e8684119a12612696ed6140c1a61: Status 404 returned error can't find the container with id 83482605e62b4c12480ac848c7cbedb80be1e8684119a12612696ed6140c1a61 Oct 31 01:17:59.775103 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:17:59.775081 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9f6da73-kpj95"] Oct 31 01:18:00.261052 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:00.261010 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9f6da73-kpj95" event={"ID":"3e361b6c-3b9a-4871-b138-28d6c7c5f47e","Type":"ContainerStarted","Data":"8f63015e21b711d05755ff16a9ffab6cae70684cd725d8e87e9ce1eea7c7fdd4"} Oct 31 01:18:00.261478 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:00.261076 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9f6da73-kpj95" event={"ID":"3e361b6c-3b9a-4871-b138-28d6c7c5f47e","Type":"ContainerStarted","Data":"83482605e62b4c12480ac848c7cbedb80be1e8684119a12612696ed6140c1a61"} Oct 31 01:18:00.379241 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:00.379139 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9f6da73-kpj95" podStartSLOduration=1.3791252090000001 podStartE2EDuration="1.379125209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:17:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:18:00.37863051 +0000 UTC m=+3772.849800260" watchObservedRunningTime="2026-10-31 01:18:00.379125209 +0000 UTC m=+3772.850294958" Oct 31 01:18:01.266816 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:01.266778 5222 generic.go:356] "Generic (PLEG): container finished" podID="3e361b6c-3b9a-4871-b138-28d6c7c5f47e" containerID="8f63015e21b711d05755ff16a9ffab6cae70684cd725d8e87e9ce1eea7c7fdd4" exitCode=0 Oct 31 01:18:01.267287 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:01.266866 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9f6da73-kpj95" event={"ID":"3e361b6c-3b9a-4871-b138-28d6c7c5f47e","Type":"ContainerDied","Data":"8f63015e21b711d05755ff16a9ffab6cae70684cd725d8e87e9ce1eea7c7fdd4"} Oct 31 01:18:02.414033 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:02.414006 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9f6da73-kpj95" Oct 31 01:18:02.521719 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:02.521689 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e361b6c-3b9a-4871-b138-28d6c7c5f47e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e361b6c-3b9a-4871-b138-28d6c7c5f47e-maas-api-xks-ca\") pod \"3e361b6c-3b9a-4871-b138-28d6c7c5f47e\" (UID: \"3e361b6c-3b9a-4871-b138-28d6c7c5f47e\") " Oct 31 01:18:02.521934 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:02.521739 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e361b6c-3b9a-4871-b138-28d6c7c5f47e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e361b6c-3b9a-4871-b138-28d6c7c5f47e-maas-api-openshift-ca\") pod \"3e361b6c-3b9a-4871-b138-28d6c7c5f47e\" (UID: \"3e361b6c-3b9a-4871-b138-28d6c7c5f47e\") " Oct 31 01:18:02.521934 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:02.521781 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e361b6c-3b9a-4871-b138-28d6c7c5f47e-kube-api-access-k978p\" (UniqueName: \"kubernetes.io/projected/3e361b6c-3b9a-4871-b138-28d6c7c5f47e-kube-api-access-k978p\") pod \"3e361b6c-3b9a-4871-b138-28d6c7c5f47e\" (UID: \"3e361b6c-3b9a-4871-b138-28d6c7c5f47e\") " Oct 31 01:18:02.522211 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:02.522156 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e361b6c-3b9a-4871-b138-28d6c7c5f47e-maas-api-openshift-ca" pod "3e361b6c-3b9a-4871-b138-28d6c7c5f47e" (UID: "3e361b6c-3b9a-4871-b138-28d6c7c5f47e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:18:02.524089 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:02.524045 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e361b6c-3b9a-4871-b138-28d6c7c5f47e-maas-api-xks-ca" pod "3e361b6c-3b9a-4871-b138-28d6c7c5f47e" (UID: "3e361b6c-3b9a-4871-b138-28d6c7c5f47e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:18:02.524198 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:02.524098 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e361b6c-3b9a-4871-b138-28d6c7c5f47e-kube-api-access-k978p" pod "3e361b6c-3b9a-4871-b138-28d6c7c5f47e" (UID: "3e361b6c-3b9a-4871-b138-28d6c7c5f47e"). InnerVolumeSpecName "kube-api-access-k978p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:18:02.622658 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:02.622614 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e361b6c-3b9a-4871-b138-28d6c7c5f47e-maas-api-xks-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:18:02.622658 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:02.622650 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e361b6c-3b9a-4871-b138-28d6c7c5f47e-maas-api-openshift-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:18:02.622658 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:02.622661 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k978p\" (UniqueName: \"kubernetes.io/projected/3e361b6c-3b9a-4871-b138-28d6c7c5f47e-kube-api-access-k978p\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:18:03.277821 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:03.277793 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9f6da73-kpj95" Oct 31 01:18:03.277984 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:03.277798 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9f6da73-kpj95" event={"ID":"3e361b6c-3b9a-4871-b138-28d6c7c5f47e","Type":"ContainerDied","Data":"83482605e62b4c12480ac848c7cbedb80be1e8684119a12612696ed6140c1a61"} Oct 31 01:18:03.277984 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:03.277912 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="83482605e62b4c12480ac848c7cbedb80be1e8684119a12612696ed6140c1a61" Oct 31 01:18:05.888689 ip-10-0-137-56 kubenswrapper[5222]: E1031 01:18:05.888650 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 01:18:11.996406 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:11.996354 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-79tk8"] Oct 31 01:18:19.952730 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:19.952636 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-576e86c8-openshift-default-98874d9b5-w9brf"] Oct 31 01:18:19.953236 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:19.953211 5222 cpu_manager.go:414] "containerMap: removing container" podUID="3e361b6c-3b9a-4871-b138-28d6c7c5f47e" containerName="revoke-keys" Oct 31 01:18:19.953236 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:19.953231 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e361b6c-3b9a-4871-b138-28d6c7c5f47e" containerName="revoke-keys" Oct 31 01:18:19.953402 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:19.953348 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e361b6c-3b9a-4871-b138-28d6c7c5f47e" containerName="revoke-keys" Oct 31 01:18:19.957940 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:19.957914 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-576e86c8-openshift-default-98874d9b5-w9brf" Oct 31 01:18:20.005180 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:20.005139 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2ee6d42-4edc-4633-ae68-519fb4ed9bae-istio-token\") pod \"e2e-ait-576e86c8-openshift-default-98874d9b5-w9brf\" (UID: \"d2ee6d42-4edc-4633-ae68-519fb4ed9bae\") " pod="openshift-ingress/e2e-ait-576e86c8-openshift-default-98874d9b5-w9brf" Oct 31 01:18:20.005389 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:20.005204 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2ee6d42-4edc-4633-ae68-519fb4ed9bae-workload-socket\") pod \"e2e-ait-576e86c8-openshift-default-98874d9b5-w9brf\" (UID: \"d2ee6d42-4edc-4633-ae68-519fb4ed9bae\") " pod="openshift-ingress/e2e-ait-576e86c8-openshift-default-98874d9b5-w9brf" Oct 31 01:18:20.005389 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:20.005241 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2ee6d42-4edc-4633-ae68-519fb4ed9bae-istio-podinfo\") pod \"e2e-ait-576e86c8-openshift-default-98874d9b5-w9brf\" (UID: \"d2ee6d42-4edc-4633-ae68-519fb4ed9bae\") " pod="openshift-ingress/e2e-ait-576e86c8-openshift-default-98874d9b5-w9brf" Oct 31 01:18:20.005389 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:20.005324 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2ee6d42-4edc-4633-ae68-519fb4ed9bae-istiod-ca-cert\") pod \"e2e-ait-576e86c8-openshift-default-98874d9b5-w9brf\" (UID: \"d2ee6d42-4edc-4633-ae68-519fb4ed9bae\") " pod="openshift-ingress/e2e-ait-576e86c8-openshift-default-98874d9b5-w9brf" Oct 31 01:18:20.005389 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:20.005355 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2ee6d42-4edc-4633-ae68-519fb4ed9bae-istio-envoy\") pod \"e2e-ait-576e86c8-openshift-default-98874d9b5-w9brf\" (UID: \"d2ee6d42-4edc-4633-ae68-519fb4ed9bae\") " pod="openshift-ingress/e2e-ait-576e86c8-openshift-default-98874d9b5-w9brf" Oct 31 01:18:20.005675 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:20.005429 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2ee6d42-4edc-4633-ae68-519fb4ed9bae-workload-certs\") pod \"e2e-ait-576e86c8-openshift-default-98874d9b5-w9brf\" (UID: \"d2ee6d42-4edc-4633-ae68-519fb4ed9bae\") " pod="openshift-ingress/e2e-ait-576e86c8-openshift-default-98874d9b5-w9brf" Oct 31 01:18:20.005675 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:20.005474 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2ee6d42-4edc-4633-ae68-519fb4ed9bae-istio-data\") pod \"e2e-ait-576e86c8-openshift-default-98874d9b5-w9brf\" (UID: \"d2ee6d42-4edc-4633-ae68-519fb4ed9bae\") " pod="openshift-ingress/e2e-ait-576e86c8-openshift-default-98874d9b5-w9brf" Oct 31 01:18:20.005675 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:20.005641 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2ee6d42-4edc-4633-ae68-519fb4ed9bae-credential-socket\") pod \"e2e-ait-576e86c8-openshift-default-98874d9b5-w9brf\" (UID: \"d2ee6d42-4edc-4633-ae68-519fb4ed9bae\") " pod="openshift-ingress/e2e-ait-576e86c8-openshift-default-98874d9b5-w9brf" Oct 31 01:18:20.005839 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:20.005678 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7p846\" (UniqueName: \"kubernetes.io/projected/d2ee6d42-4edc-4633-ae68-519fb4ed9bae-kube-api-access-7p846\") pod \"e2e-ait-576e86c8-openshift-default-98874d9b5-w9brf\" (UID: \"d2ee6d42-4edc-4633-ae68-519fb4ed9bae\") " pod="openshift-ingress/e2e-ait-576e86c8-openshift-default-98874d9b5-w9brf" Oct 31 01:18:20.035449 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:20.019776 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-576e86c8-openshift-default-dockercfg-p2kdm\"" Oct 31 01:18:20.073063 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:20.073031 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-576e86c8-openshift-default-98874d9b5-w9brf"] Oct 31 01:18:20.108672 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:20.108630 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2ee6d42-4edc-4633-ae68-519fb4ed9bae-istio-token\") pod \"e2e-ait-576e86c8-openshift-default-98874d9b5-w9brf\" (UID: \"d2ee6d42-4edc-4633-ae68-519fb4ed9bae\") " pod="openshift-ingress/e2e-ait-576e86c8-openshift-default-98874d9b5-w9brf" Oct 31 01:18:20.108895 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:20.108704 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2ee6d42-4edc-4633-ae68-519fb4ed9bae-workload-socket\") pod \"e2e-ait-576e86c8-openshift-default-98874d9b5-w9brf\" (UID: \"d2ee6d42-4edc-4633-ae68-519fb4ed9bae\") " pod="openshift-ingress/e2e-ait-576e86c8-openshift-default-98874d9b5-w9brf" Oct 31 01:18:20.108895 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:20.108734 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2ee6d42-4edc-4633-ae68-519fb4ed9bae-istio-podinfo\") pod \"e2e-ait-576e86c8-openshift-default-98874d9b5-w9brf\" (UID: \"d2ee6d42-4edc-4633-ae68-519fb4ed9bae\") " pod="openshift-ingress/e2e-ait-576e86c8-openshift-default-98874d9b5-w9brf" Oct 31 01:18:20.108895 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:20.108827 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2ee6d42-4edc-4633-ae68-519fb4ed9bae-istiod-ca-cert\") pod \"e2e-ait-576e86c8-openshift-default-98874d9b5-w9brf\" (UID: \"d2ee6d42-4edc-4633-ae68-519fb4ed9bae\") " pod="openshift-ingress/e2e-ait-576e86c8-openshift-default-98874d9b5-w9brf" Oct 31 01:18:20.108895 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:20.108860 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2ee6d42-4edc-4633-ae68-519fb4ed9bae-istio-envoy\") pod \"e2e-ait-576e86c8-openshift-default-98874d9b5-w9brf\" (UID: \"d2ee6d42-4edc-4633-ae68-519fb4ed9bae\") " pod="openshift-ingress/e2e-ait-576e86c8-openshift-default-98874d9b5-w9brf" Oct 31 01:18:20.108895 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:20.108913 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2ee6d42-4edc-4633-ae68-519fb4ed9bae-workload-certs\") pod \"e2e-ait-576e86c8-openshift-default-98874d9b5-w9brf\" (UID: \"d2ee6d42-4edc-4633-ae68-519fb4ed9bae\") " pod="openshift-ingress/e2e-ait-576e86c8-openshift-default-98874d9b5-w9brf" Oct 31 01:18:20.110793 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:20.108961 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2ee6d42-4edc-4633-ae68-519fb4ed9bae-istio-data\") pod \"e2e-ait-576e86c8-openshift-default-98874d9b5-w9brf\" (UID: \"d2ee6d42-4edc-4633-ae68-519fb4ed9bae\") " pod="openshift-ingress/e2e-ait-576e86c8-openshift-default-98874d9b5-w9brf" Oct 31 01:18:20.110793 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:20.108992 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2ee6d42-4edc-4633-ae68-519fb4ed9bae-credential-socket\") pod \"e2e-ait-576e86c8-openshift-default-98874d9b5-w9brf\" (UID: \"d2ee6d42-4edc-4633-ae68-519fb4ed9bae\") " pod="openshift-ingress/e2e-ait-576e86c8-openshift-default-98874d9b5-w9brf" Oct 31 01:18:20.110793 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:20.109038 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7p846\" (UniqueName: \"kubernetes.io/projected/d2ee6d42-4edc-4633-ae68-519fb4ed9bae-kube-api-access-7p846\") pod \"e2e-ait-576e86c8-openshift-default-98874d9b5-w9brf\" (UID: \"d2ee6d42-4edc-4633-ae68-519fb4ed9bae\") " pod="openshift-ingress/e2e-ait-576e86c8-openshift-default-98874d9b5-w9brf" Oct 31 01:18:20.112510 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:20.112478 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2ee6d42-4edc-4633-ae68-519fb4ed9bae-credential-socket\") pod \"e2e-ait-576e86c8-openshift-default-98874d9b5-w9brf\" (UID: \"d2ee6d42-4edc-4633-ae68-519fb4ed9bae\") " pod="openshift-ingress/e2e-ait-576e86c8-openshift-default-98874d9b5-w9brf" Oct 31 01:18:20.112945 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:20.112831 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2ee6d42-4edc-4633-ae68-519fb4ed9bae-istiod-ca-cert\") pod \"e2e-ait-576e86c8-openshift-default-98874d9b5-w9brf\" (UID: \"d2ee6d42-4edc-4633-ae68-519fb4ed9bae\") " pod="openshift-ingress/e2e-ait-576e86c8-openshift-default-98874d9b5-w9brf" Oct 31 01:18:20.116159 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:20.114956 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2ee6d42-4edc-4633-ae68-519fb4ed9bae-workload-certs\") pod \"e2e-ait-576e86c8-openshift-default-98874d9b5-w9brf\" (UID: \"d2ee6d42-4edc-4633-ae68-519fb4ed9bae\") " pod="openshift-ingress/e2e-ait-576e86c8-openshift-default-98874d9b5-w9brf" Oct 31 01:18:20.116419 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:20.116334 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2ee6d42-4edc-4633-ae68-519fb4ed9bae-workload-socket\") pod \"e2e-ait-576e86c8-openshift-default-98874d9b5-w9brf\" (UID: \"d2ee6d42-4edc-4633-ae68-519fb4ed9bae\") " pod="openshift-ingress/e2e-ait-576e86c8-openshift-default-98874d9b5-w9brf" Oct 31 01:18:20.116829 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:20.116808 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2ee6d42-4edc-4633-ae68-519fb4ed9bae-istio-data\") pod \"e2e-ait-576e86c8-openshift-default-98874d9b5-w9brf\" (UID: \"d2ee6d42-4edc-4633-ae68-519fb4ed9bae\") " pod="openshift-ingress/e2e-ait-576e86c8-openshift-default-98874d9b5-w9brf" Oct 31 01:18:20.120714 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:20.120680 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2ee6d42-4edc-4633-ae68-519fb4ed9bae-istio-envoy\") pod \"e2e-ait-576e86c8-openshift-default-98874d9b5-w9brf\" (UID: \"d2ee6d42-4edc-4633-ae68-519fb4ed9bae\") " pod="openshift-ingress/e2e-ait-576e86c8-openshift-default-98874d9b5-w9brf" Oct 31 01:18:20.134511 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:20.134460 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2ee6d42-4edc-4633-ae68-519fb4ed9bae-istio-podinfo\") pod \"e2e-ait-576e86c8-openshift-default-98874d9b5-w9brf\" (UID: \"d2ee6d42-4edc-4633-ae68-519fb4ed9bae\") " pod="openshift-ingress/e2e-ait-576e86c8-openshift-default-98874d9b5-w9brf" Oct 31 01:18:20.174478 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:20.174436 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7p846\" (UniqueName: \"kubernetes.io/projected/d2ee6d42-4edc-4633-ae68-519fb4ed9bae-kube-api-access-7p846\") pod \"e2e-ait-576e86c8-openshift-default-98874d9b5-w9brf\" (UID: \"d2ee6d42-4edc-4633-ae68-519fb4ed9bae\") " pod="openshift-ingress/e2e-ait-576e86c8-openshift-default-98874d9b5-w9brf" Oct 31 01:18:20.174727 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:20.174708 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2ee6d42-4edc-4633-ae68-519fb4ed9bae-istio-token\") pod \"e2e-ait-576e86c8-openshift-default-98874d9b5-w9brf\" (UID: \"d2ee6d42-4edc-4633-ae68-519fb4ed9bae\") " pod="openshift-ingress/e2e-ait-576e86c8-openshift-default-98874d9b5-w9brf" Oct 31 01:18:20.272390 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:20.272333 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-576e86c8-openshift-default-98874d9b5-w9brf" Oct 31 01:18:20.699409 ip-10-0-137-56 kubenswrapper[5222]: W1031 01:18:20.699353 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2ee6d42_4edc_4633_ae68_519fb4ed9bae.slice/crio-33464a94ff292fd09a7bd40e16e3f333bfc6a8457bf7c343c3de34a29c1cd97c WatchSource:0}: Error finding container 33464a94ff292fd09a7bd40e16e3f333bfc6a8457bf7c343c3de34a29c1cd97c: Status 404 returned error can't find the container with id 33464a94ff292fd09a7bd40e16e3f333bfc6a8457bf7c343c3de34a29c1cd97c Oct 31 01:18:20.700752 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:20.700722 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-576e86c8-openshift-default-98874d9b5-w9brf"] Oct 31 01:18:20.701753 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:20.701719 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 01:18:20.701816 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:20.701795 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 01:18:20.701858 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:20.701824 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 01:18:21.367340 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:21.367295 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-576e86c8-openshift-default-98874d9b5-w9brf" event={"ID":"d2ee6d42-4edc-4633-ae68-519fb4ed9bae","Type":"ContainerStarted","Data":"3270990a928452289d9c62b91efb3518104381c534054225fdee956083f46a09"} Oct 31 01:18:21.367825 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:21.367409 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-576e86c8-openshift-default-98874d9b5-w9brf" event={"ID":"d2ee6d42-4edc-4633-ae68-519fb4ed9bae","Type":"ContainerStarted","Data":"33464a94ff292fd09a7bd40e16e3f333bfc6a8457bf7c343c3de34a29c1cd97c"} Oct 31 01:18:21.490619 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:21.490557 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-576e86c8-openshift-default-98874d9b5-w9brf" podStartSLOduration=2.490544701 podStartE2EDuration="2.490544701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:18:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:18:21.489924049 +0000 UTC m=+3793.961093807" watchObservedRunningTime="2026-10-31 01:18:21.490544701 +0000 UTC m=+3793.961714449" Oct 31 01:18:22.273119 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:22.273079 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-576e86c8-openshift-default-98874d9b5-w9brf" Oct 31 01:18:22.278143 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:22.278112 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-576e86c8-openshift-default-98874d9b5-w9brf" Oct 31 01:18:22.372623 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:22.372588 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-576e86c8-openshift-default-98874d9b5-w9brf" Oct 31 01:18:22.373645 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:22.373626 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-576e86c8-openshift-default-98874d9b5-w9brf" Oct 31 01:18:27.356023 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:27.355991 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-576e86c8-6876945bcf-2shmj"] Oct 31 01:18:27.359656 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:27.359634 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-576e86c8-6876945bcf-2shmj" Oct 31 01:18:27.381133 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:27.381078 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5d1a0ce-aa51-4030-8d4d-0f4a0feab2b3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-576e86c8-6876945bcf-2shmj\" (UID: \"a5d1a0ce-aa51-4030-8d4d-0f4a0feab2b3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-576e86c8-6876945bcf-2shmj" Oct 31 01:18:27.381342 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:27.381244 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsk4m\" (UniqueName: \"kubernetes.io/projected/a5d1a0ce-aa51-4030-8d4d-0f4a0feab2b3-kube-api-access-dsk4m\") pod \"payload-pre-processing-e2e-ait-576e86c8-6876945bcf-2shmj\" (UID: \"a5d1a0ce-aa51-4030-8d4d-0f4a0feab2b3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-576e86c8-6876945bcf-2shmj" Oct 31 01:18:27.425325 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:27.425278 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-576e86c8-6876945bcf-2shmj"] Oct 31 01:18:27.470080 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:27.470048 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-576e86c8-dockercfg-bzv9q\"" Oct 31 01:18:27.470080 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:27.470064 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-576e86c8\"" Oct 31 01:18:27.481894 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:27.481858 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5d1a0ce-aa51-4030-8d4d-0f4a0feab2b3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-576e86c8-6876945bcf-2shmj\" (UID: \"a5d1a0ce-aa51-4030-8d4d-0f4a0feab2b3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-576e86c8-6876945bcf-2shmj" Oct 31 01:18:27.482281 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:27.481981 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsk4m\" (UniqueName: \"kubernetes.io/projected/a5d1a0ce-aa51-4030-8d4d-0f4a0feab2b3-kube-api-access-dsk4m\") pod \"payload-pre-processing-e2e-ait-576e86c8-6876945bcf-2shmj\" (UID: \"a5d1a0ce-aa51-4030-8d4d-0f4a0feab2b3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-576e86c8-6876945bcf-2shmj" Oct 31 01:18:27.482848 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:27.482821 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5d1a0ce-aa51-4030-8d4d-0f4a0feab2b3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-576e86c8-6876945bcf-2shmj\" (UID: \"a5d1a0ce-aa51-4030-8d4d-0f4a0feab2b3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-576e86c8-6876945bcf-2shmj" Oct 31 01:18:27.505053 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:27.505014 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-576e86c8-6f8546b5c9-txsdf"] Oct 31 01:18:27.508828 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:27.508802 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-576e86c8-6f8546b5c9-txsdf" Oct 31 01:18:27.544681 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:27.544643 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsk4m\" (UniqueName: \"kubernetes.io/projected/a5d1a0ce-aa51-4030-8d4d-0f4a0feab2b3-kube-api-access-dsk4m\") pod \"payload-pre-processing-e2e-ait-576e86c8-6876945bcf-2shmj\" (UID: \"a5d1a0ce-aa51-4030-8d4d-0f4a0feab2b3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-576e86c8-6876945bcf-2shmj" Oct 31 01:18:27.563965 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:27.563931 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-576e86c8-6f8546b5c9-txsdf"] Oct 31 01:18:27.582674 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:27.582633 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcf6a668-1b6d-41d1-89b0-5cbddd61d86f-plugins-config-volume\") pod \"payload-processing-e2e-ait-576e86c8-6f8546b5c9-txsdf\" (UID: \"dcf6a668-1b6d-41d1-89b0-5cbddd61d86f\") " pod="openshift-ingress/payload-processing-e2e-ait-576e86c8-6f8546b5c9-txsdf" Oct 31 01:18:27.582885 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:27.582820 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6v5px\" (UniqueName: \"kubernetes.io/projected/dcf6a668-1b6d-41d1-89b0-5cbddd61d86f-kube-api-access-6v5px\") pod \"payload-processing-e2e-ait-576e86c8-6f8546b5c9-txsdf\" (UID: \"dcf6a668-1b6d-41d1-89b0-5cbddd61d86f\") " pod="openshift-ingress/payload-processing-e2e-ait-576e86c8-6f8546b5c9-txsdf" Oct 31 01:18:27.675195 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:27.675101 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-576e86c8-6876945bcf-2shmj" Oct 31 01:18:27.683572 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:27.683537 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6v5px\" (UniqueName: \"kubernetes.io/projected/dcf6a668-1b6d-41d1-89b0-5cbddd61d86f-kube-api-access-6v5px\") pod \"payload-processing-e2e-ait-576e86c8-6f8546b5c9-txsdf\" (UID: \"dcf6a668-1b6d-41d1-89b0-5cbddd61d86f\") " pod="openshift-ingress/payload-processing-e2e-ait-576e86c8-6f8546b5c9-txsdf" Oct 31 01:18:27.683718 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:27.683633 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcf6a668-1b6d-41d1-89b0-5cbddd61d86f-plugins-config-volume\") pod \"payload-processing-e2e-ait-576e86c8-6f8546b5c9-txsdf\" (UID: \"dcf6a668-1b6d-41d1-89b0-5cbddd61d86f\") " pod="openshift-ingress/payload-processing-e2e-ait-576e86c8-6f8546b5c9-txsdf" Oct 31 01:18:27.684352 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:27.684292 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcf6a668-1b6d-41d1-89b0-5cbddd61d86f-plugins-config-volume\") pod \"payload-processing-e2e-ait-576e86c8-6f8546b5c9-txsdf\" (UID: \"dcf6a668-1b6d-41d1-89b0-5cbddd61d86f\") " pod="openshift-ingress/payload-processing-e2e-ait-576e86c8-6f8546b5c9-txsdf" Oct 31 01:18:27.746897 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:27.746856 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6v5px\" (UniqueName: \"kubernetes.io/projected/dcf6a668-1b6d-41d1-89b0-5cbddd61d86f-kube-api-access-6v5px\") pod \"payload-processing-e2e-ait-576e86c8-6f8546b5c9-txsdf\" (UID: \"dcf6a668-1b6d-41d1-89b0-5cbddd61d86f\") " pod="openshift-ingress/payload-processing-e2e-ait-576e86c8-6f8546b5c9-txsdf" Oct 31 01:18:27.826435 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:27.826402 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-576e86c8-6f8546b5c9-txsdf" Oct 31 01:18:28.046071 ip-10-0-137-56 kubenswrapper[5222]: W1031 01:18:28.046040 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5d1a0ce_aa51_4030_8d4d_0f4a0feab2b3.slice/crio-fafddef4d300b9264dbe2901314fc56008c464c15d815b486b8f0a2a276f19d2 WatchSource:0}: Error finding container fafddef4d300b9264dbe2901314fc56008c464c15d815b486b8f0a2a276f19d2: Status 404 returned error can't find the container with id fafddef4d300b9264dbe2901314fc56008c464c15d815b486b8f0a2a276f19d2 Oct 31 01:18:28.047439 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:28.047410 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-576e86c8-6876945bcf-2shmj"] Oct 31 01:18:28.200794 ip-10-0-137-56 kubenswrapper[5222]: W1031 01:18:28.200749 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcf6a668_1b6d_41d1_89b0_5cbddd61d86f.slice/crio-8ee46390647af82337cd7f5a2c0787eb438a3a48fd1a4959303a3b0ea7b4adca WatchSource:0}: Error finding container 8ee46390647af82337cd7f5a2c0787eb438a3a48fd1a4959303a3b0ea7b4adca: Status 404 returned error can't find the container with id 8ee46390647af82337cd7f5a2c0787eb438a3a48fd1a4959303a3b0ea7b4adca Oct 31 01:18:28.202516 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:28.202402 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-576e86c8-6f8546b5c9-txsdf"] Oct 31 01:18:28.401545 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:28.401431 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-576e86c8-6876945bcf-2shmj" event={"ID":"a5d1a0ce-aa51-4030-8d4d-0f4a0feab2b3","Type":"ContainerStarted","Data":"991b2ff696a1b508f467841de5649357effb154e0a12023bc9d0d30ed8c495dd"} Oct 31 01:18:28.401545 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:28.401520 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-576e86c8-6876945bcf-2shmj" event={"ID":"a5d1a0ce-aa51-4030-8d4d-0f4a0feab2b3","Type":"ContainerStarted","Data":"fafddef4d300b9264dbe2901314fc56008c464c15d815b486b8f0a2a276f19d2"} Oct 31 01:18:28.402073 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:28.401551 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-576e86c8-6876945bcf-2shmj" Oct 31 01:18:28.403018 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:28.402994 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-576e86c8-6f8546b5c9-txsdf" event={"ID":"dcf6a668-1b6d-41d1-89b0-5cbddd61d86f","Type":"ContainerStarted","Data":"f7b121c8bc5afd1319d9932be5a8b0a96683f7b33fb87c7e79b6b5d022df5533"} Oct 31 01:18:28.403150 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:28.403032 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-576e86c8-6f8546b5c9-txsdf" event={"ID":"dcf6a668-1b6d-41d1-89b0-5cbddd61d86f","Type":"ContainerStarted","Data":"8ee46390647af82337cd7f5a2c0787eb438a3a48fd1a4959303a3b0ea7b4adca"} Oct 31 01:18:28.403150 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:28.403090 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-576e86c8-6f8546b5c9-txsdf" Oct 31 01:18:28.522985 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:28.522930 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-576e86c8-6876945bcf-2shmj" podStartSLOduration=1.52290592 podStartE2EDuration="1.52290592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:18:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:18:28.520856999 +0000 UTC m=+3800.992026748" watchObservedRunningTime="2026-10-31 01:18:28.52290592 +0000 UTC m=+3800.994075671" Oct 31 01:18:28.663238 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:28.662226 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-576e86c8-6f8546b5c9-txsdf" podStartSLOduration=1.662206749 podStartE2EDuration="1.662206749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:18:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:18:28.645333931 +0000 UTC m=+3801.116503684" watchObservedRunningTime="2026-10-31 01:18:28.662206749 +0000 UTC m=+3801.133376500" Oct 31 01:18:35.890232 ip-10-0-137-56 kubenswrapper[5222]: E1031 01:18:35.890191 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 01:18:37.306284 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:37.306246 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-79tk8"] Oct 31 01:18:39.411079 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:39.411050 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-576e86c8-6876945bcf-2shmj" Oct 31 01:18:39.411569 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:39.411135 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-576e86c8-6f8546b5c9-txsdf" Oct 31 01:18:41.057395 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:41.055716 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nhmqk"] Oct 31 01:18:41.105668 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:41.105637 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nhmqk"] Oct 31 01:18:41.666869 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:41.666839 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd488448-cbbb-490e-a43b-ae78cb4f0f3b" path="/var/lib/kubelet/pods/cd488448-cbbb-490e-a43b-ae78cb4f0f3b/volumes" Oct 31 01:18:47.510898 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:47.510866 5222 generic.go:356] "Generic (PLEG): container finished" podID="d2ee6d42-4edc-4633-ae68-519fb4ed9bae" containerID="3270990a928452289d9c62b91efb3518104381c534054225fdee956083f46a09" exitCode=0 Oct 31 01:18:47.511338 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:47.510939 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-576e86c8-openshift-default-98874d9b5-w9brf" event={"ID":"d2ee6d42-4edc-4633-ae68-519fb4ed9bae","Type":"ContainerDied","Data":"3270990a928452289d9c62b91efb3518104381c534054225fdee956083f46a09"} Oct 31 01:18:47.511412 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:47.511353 5222 scope.go:122] "RemoveContainer" containerID="3270990a928452289d9c62b91efb3518104381c534054225fdee956083f46a09" Oct 31 01:18:47.512653 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:47.512626 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 01:18:47.512729 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:47.512693 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 01:18:47.512729 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:47.512725 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 01:18:48.517637 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:48.517597 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-576e86c8-openshift-default-98874d9b5-w9brf" event={"ID":"d2ee6d42-4edc-4633-ae68-519fb4ed9bae","Type":"ContainerStarted","Data":"bd49c93418cf7b1e8a5d6ab881aaca272e509cab931d40f8b99dfe2cdb47820a"} Oct 31 01:18:49.272718 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:49.272676 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-576e86c8-openshift-default-98874d9b5-w9brf" Oct 31 01:18:49.273999 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:49.273970 5222 patch_prober.go:29] interesting pod/e2e-ait-576e86c8-openshift-default-98874d9b5-w9brf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.16:15021/healthz/ready\": dial tcp 10.134.1.16:15021: connect: connection refused" start-of-body= Oct 31 01:18:49.274137 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:49.274033 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-576e86c8-openshift-default-98874d9b5-w9brf" podUID="d2ee6d42-4edc-4633-ae68-519fb4ed9bae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.16:15021/healthz/ready\": dial tcp 10.134.1.16:15021: connect: connection refused" Oct 31 01:18:50.273168 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:50.273127 5222 patch_prober.go:29] interesting pod/e2e-ait-576e86c8-openshift-default-98874d9b5-w9brf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.16:15021/healthz/ready\": dial tcp 10.134.1.16:15021: connect: connection refused" start-of-body= Oct 31 01:18:50.273616 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:50.273187 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-576e86c8-openshift-default-98874d9b5-w9brf" podUID="d2ee6d42-4edc-4633-ae68-519fb4ed9bae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.16:15021/healthz/ready\": dial tcp 10.134.1.16:15021: connect: connection refused" Oct 31 01:18:51.273278 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:51.273235 5222 patch_prober.go:29] interesting pod/e2e-ait-576e86c8-openshift-default-98874d9b5-w9brf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.16:15021/healthz/ready\": dial tcp 10.134.1.16:15021: connect: connection refused" start-of-body= Oct 31 01:18:51.273823 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:51.273312 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-576e86c8-openshift-default-98874d9b5-w9brf" podUID="d2ee6d42-4edc-4633-ae68-519fb4ed9bae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.16:15021/healthz/ready\": dial tcp 10.134.1.16:15021: connect: connection refused" Oct 31 01:18:52.273261 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:52.273220 5222 patch_prober.go:29] interesting pod/e2e-ait-576e86c8-openshift-default-98874d9b5-w9brf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.16:15021/healthz/ready\": dial tcp 10.134.1.16:15021: connect: connection refused" start-of-body= Oct 31 01:18:52.273486 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:52.273281 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-576e86c8-openshift-default-98874d9b5-w9brf" podUID="d2ee6d42-4edc-4633-ae68-519fb4ed9bae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.16:15021/healthz/ready\": dial tcp 10.134.1.16:15021: connect: connection refused" Oct 31 01:18:52.372681 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:52.372636 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-576e86c8-openshift-default-98874d9b5-w9brf" Oct 31 01:18:53.273197 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:53.273155 5222 patch_prober.go:29] interesting pod/e2e-ait-576e86c8-openshift-default-98874d9b5-w9brf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.16:15021/healthz/ready\": dial tcp 10.134.1.16:15021: connect: connection refused" start-of-body= Oct 31 01:18:53.273444 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:53.273213 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-576e86c8-openshift-default-98874d9b5-w9brf" podUID="d2ee6d42-4edc-4633-ae68-519fb4ed9bae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.16:15021/healthz/ready\": dial tcp 10.134.1.16:15021: connect: connection refused" Oct 31 01:18:54.277022 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:54.276987 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-576e86c8-openshift-default-98874d9b5-w9brf" Oct 31 01:18:54.277955 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:18:54.277938 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-576e86c8-openshift-default-98874d9b5-w9brf" Oct 31 01:19:00.324735 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:00.324698 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-576e86c8-ggldm"] Oct 31 01:19:00.329400 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:00.329357 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-576e86c8-ggldm" Oct 31 01:19:00.381919 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:00.381874 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1e87fb0-00ca-41d4-8a32-67df2f86d36e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-576e86c8-ggldm\" (UID: \"d1e87fb0-00ca-41d4-8a32-67df2f86d36e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-576e86c8-ggldm" Oct 31 01:19:00.382116 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:00.382009 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pt7zr\" (UniqueName: \"kubernetes.io/projected/d1e87fb0-00ca-41d4-8a32-67df2f86d36e-kube-api-access-pt7zr\") pod \"maas-api-revoke-keys-e2e-ait-576e86c8-ggldm\" (UID: \"d1e87fb0-00ca-41d4-8a32-67df2f86d36e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-576e86c8-ggldm" Oct 31 01:19:00.382116 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:00.382053 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1e87fb0-00ca-41d4-8a32-67df2f86d36e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-576e86c8-ggldm\" (UID: \"d1e87fb0-00ca-41d4-8a32-67df2f86d36e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-576e86c8-ggldm" Oct 31 01:19:00.382845 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:00.382819 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-576e86c8-ggldm"] Oct 31 01:19:00.431714 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:00.431685 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:19:00.431714 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:00.431683 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:19:00.431714 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:00.431716 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-29cjc\"" Oct 31 01:19:00.431985 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:00.431685 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:19:00.482845 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:00.482800 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1e87fb0-00ca-41d4-8a32-67df2f86d36e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-576e86c8-ggldm\" (UID: \"d1e87fb0-00ca-41d4-8a32-67df2f86d36e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-576e86c8-ggldm" Oct 31 01:19:00.483038 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:00.482885 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pt7zr\" (UniqueName: \"kubernetes.io/projected/d1e87fb0-00ca-41d4-8a32-67df2f86d36e-kube-api-access-pt7zr\") pod \"maas-api-revoke-keys-e2e-ait-576e86c8-ggldm\" (UID: \"d1e87fb0-00ca-41d4-8a32-67df2f86d36e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-576e86c8-ggldm" Oct 31 01:19:00.483038 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:00.482908 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1e87fb0-00ca-41d4-8a32-67df2f86d36e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-576e86c8-ggldm\" (UID: \"d1e87fb0-00ca-41d4-8a32-67df2f86d36e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-576e86c8-ggldm" Oct 31 01:19:00.483551 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:00.483514 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1e87fb0-00ca-41d4-8a32-67df2f86d36e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-576e86c8-ggldm\" (UID: \"d1e87fb0-00ca-41d4-8a32-67df2f86d36e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-576e86c8-ggldm" Oct 31 01:19:00.485542 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:00.485509 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1e87fb0-00ca-41d4-8a32-67df2f86d36e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-576e86c8-ggldm\" (UID: \"d1e87fb0-00ca-41d4-8a32-67df2f86d36e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-576e86c8-ggldm" Oct 31 01:19:00.546738 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:00.546700 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pt7zr\" (UniqueName: \"kubernetes.io/projected/d1e87fb0-00ca-41d4-8a32-67df2f86d36e-kube-api-access-pt7zr\") pod \"maas-api-revoke-keys-e2e-ait-576e86c8-ggldm\" (UID: \"d1e87fb0-00ca-41d4-8a32-67df2f86d36e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-576e86c8-ggldm" Oct 31 01:19:00.642490 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:00.642400 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-576e86c8-ggldm" Oct 31 01:19:01.013635 ip-10-0-137-56 kubenswrapper[5222]: W1031 01:19:01.013592 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1e87fb0_00ca_41d4_8a32_67df2f86d36e.slice/crio-54a4893c92e6d66887e76ed305dd81fcced5854c252389f84de4e22d4c70540f WatchSource:0}: Error finding container 54a4893c92e6d66887e76ed305dd81fcced5854c252389f84de4e22d4c70540f: Status 404 returned error can't find the container with id 54a4893c92e6d66887e76ed305dd81fcced5854c252389f84de4e22d4c70540f Oct 31 01:19:01.015205 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:01.015182 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-576e86c8-ggldm"] Oct 31 01:19:01.582803 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:01.582766 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-576e86c8-ggldm" event={"ID":"d1e87fb0-00ca-41d4-8a32-67df2f86d36e","Type":"ContainerStarted","Data":"415519ba45e228b6bda7f80eecb4795c4f3ef12669e6ea9338260a7148864d1c"} Oct 31 01:19:01.583219 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:01.582826 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-576e86c8-ggldm" event={"ID":"d1e87fb0-00ca-41d4-8a32-67df2f86d36e","Type":"ContainerStarted","Data":"54a4893c92e6d66887e76ed305dd81fcced5854c252389f84de4e22d4c70540f"} Oct 31 01:19:01.702864 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:01.702754 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-576e86c8-ggldm" podStartSLOduration=1.702740733 podStartE2EDuration="1.702740733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:19:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:19:01.70123583 +0000 UTC m=+3834.172405582" watchObservedRunningTime="2026-10-31 01:19:01.702740733 +0000 UTC m=+3834.173910482" Oct 31 01:19:02.590364 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:02.590327 5222 generic.go:356] "Generic (PLEG): container finished" podID="d1e87fb0-00ca-41d4-8a32-67df2f86d36e" containerID="415519ba45e228b6bda7f80eecb4795c4f3ef12669e6ea9338260a7148864d1c" exitCode=0 Oct 31 01:19:02.590808 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:02.590406 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-576e86c8-ggldm" event={"ID":"d1e87fb0-00ca-41d4-8a32-67df2f86d36e","Type":"ContainerDied","Data":"415519ba45e228b6bda7f80eecb4795c4f3ef12669e6ea9338260a7148864d1c"} Oct 31 01:19:03.732646 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:03.732617 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-576e86c8-ggldm" Oct 31 01:19:03.829939 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:03.829898 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1e87fb0-00ca-41d4-8a32-67df2f86d36e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1e87fb0-00ca-41d4-8a32-67df2f86d36e-maas-api-openshift-ca\") pod \"d1e87fb0-00ca-41d4-8a32-67df2f86d36e\" (UID: \"d1e87fb0-00ca-41d4-8a32-67df2f86d36e\") " Oct 31 01:19:03.830135 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:03.829974 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1e87fb0-00ca-41d4-8a32-67df2f86d36e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1e87fb0-00ca-41d4-8a32-67df2f86d36e-maas-api-xks-ca\") pod \"d1e87fb0-00ca-41d4-8a32-67df2f86d36e\" (UID: \"d1e87fb0-00ca-41d4-8a32-67df2f86d36e\") " Oct 31 01:19:03.830135 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:03.829998 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1e87fb0-00ca-41d4-8a32-67df2f86d36e-kube-api-access-pt7zr\" (UniqueName: \"kubernetes.io/projected/d1e87fb0-00ca-41d4-8a32-67df2f86d36e-kube-api-access-pt7zr\") pod \"d1e87fb0-00ca-41d4-8a32-67df2f86d36e\" (UID: \"d1e87fb0-00ca-41d4-8a32-67df2f86d36e\") " Oct 31 01:19:03.830365 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:03.830325 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1e87fb0-00ca-41d4-8a32-67df2f86d36e-maas-api-openshift-ca" pod "d1e87fb0-00ca-41d4-8a32-67df2f86d36e" (UID: "d1e87fb0-00ca-41d4-8a32-67df2f86d36e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:19:03.832257 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:03.832229 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1e87fb0-00ca-41d4-8a32-67df2f86d36e-maas-api-xks-ca" pod "d1e87fb0-00ca-41d4-8a32-67df2f86d36e" (UID: "d1e87fb0-00ca-41d4-8a32-67df2f86d36e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:19:03.832383 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:03.832306 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1e87fb0-00ca-41d4-8a32-67df2f86d36e-kube-api-access-pt7zr" pod "d1e87fb0-00ca-41d4-8a32-67df2f86d36e" (UID: "d1e87fb0-00ca-41d4-8a32-67df2f86d36e"). InnerVolumeSpecName "kube-api-access-pt7zr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:19:03.931928 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:03.931833 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1e87fb0-00ca-41d4-8a32-67df2f86d36e-maas-api-openshift-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:19:03.931928 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:03.931871 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1e87fb0-00ca-41d4-8a32-67df2f86d36e-maas-api-xks-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:19:03.931928 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:03.931881 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pt7zr\" (UniqueName: \"kubernetes.io/projected/d1e87fb0-00ca-41d4-8a32-67df2f86d36e-kube-api-access-pt7zr\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:19:04.603098 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:04.603062 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-576e86c8-ggldm" Oct 31 01:19:04.603290 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:04.603102 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-576e86c8-ggldm" event={"ID":"d1e87fb0-00ca-41d4-8a32-67df2f86d36e","Type":"ContainerDied","Data":"54a4893c92e6d66887e76ed305dd81fcced5854c252389f84de4e22d4c70540f"} Oct 31 01:19:04.603290 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:04.603147 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="54a4893c92e6d66887e76ed305dd81fcced5854c252389f84de4e22d4c70540f" Oct 31 01:19:05.893841 ip-10-0-137-56 kubenswrapper[5222]: E1031 01:19:05.893795 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 01:19:11.762625 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:11.762493 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-576e86c8-6f8546b5c9-txsdf"] Oct 31 01:19:11.871281 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:11.871235 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-576e86c8-6876945bcf-2shmj"] Oct 31 01:19:16.763489 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:16.763423 5222 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-576e86c8-6f8546b5c9-txsdf" podUID="dcf6a668-1b6d-41d1-89b0-5cbddd61d86f" containerName="payload-processing" containerID="cri-o://f7b121c8bc5afd1319d9932be5a8b0a96683f7b33fb87c7e79b6b5d022df5533" gracePeriod=25 Oct 31 01:19:16.872516 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:16.872449 5222 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-576e86c8-6876945bcf-2shmj" podUID="a5d1a0ce-aa51-4030-8d4d-0f4a0feab2b3" containerName="payload-pre-processing" containerID="cri-o://991b2ff696a1b508f467841de5649357effb154e0a12023bc9d0d30ed8c495dd" gracePeriod=25 Oct 31 01:19:17.018558 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:17.018287 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-576e86c8-6f8546b5c9-txsdf" Oct 31 01:19:17.122401 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:17.122258 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-576e86c8-6876945bcf-2shmj" Oct 31 01:19:17.183157 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:17.183118 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcf6a668-1b6d-41d1-89b0-5cbddd61d86f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcf6a668-1b6d-41d1-89b0-5cbddd61d86f-plugins-config-volume\") pod \"dcf6a668-1b6d-41d1-89b0-5cbddd61d86f\" (UID: \"dcf6a668-1b6d-41d1-89b0-5cbddd61d86f\") " Oct 31 01:19:17.183351 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:17.183178 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcf6a668-1b6d-41d1-89b0-5cbddd61d86f-kube-api-access-6v5px\" (UniqueName: \"kubernetes.io/projected/dcf6a668-1b6d-41d1-89b0-5cbddd61d86f-kube-api-access-6v5px\") pod \"dcf6a668-1b6d-41d1-89b0-5cbddd61d86f\" (UID: \"dcf6a668-1b6d-41d1-89b0-5cbddd61d86f\") " Oct 31 01:19:17.183532 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:17.183502 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcf6a668-1b6d-41d1-89b0-5cbddd61d86f-plugins-config-volume" pod "dcf6a668-1b6d-41d1-89b0-5cbddd61d86f" (UID: "dcf6a668-1b6d-41d1-89b0-5cbddd61d86f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:19:17.185496 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:17.185475 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcf6a668-1b6d-41d1-89b0-5cbddd61d86f-kube-api-access-6v5px" pod "dcf6a668-1b6d-41d1-89b0-5cbddd61d86f" (UID: "dcf6a668-1b6d-41d1-89b0-5cbddd61d86f"). InnerVolumeSpecName "kube-api-access-6v5px". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:19:17.283934 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:17.283836 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5d1a0ce-aa51-4030-8d4d-0f4a0feab2b3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5d1a0ce-aa51-4030-8d4d-0f4a0feab2b3-plugins-config-volume\") pod \"a5d1a0ce-aa51-4030-8d4d-0f4a0feab2b3\" (UID: \"a5d1a0ce-aa51-4030-8d4d-0f4a0feab2b3\") " Oct 31 01:19:17.284100 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:17.283934 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5d1a0ce-aa51-4030-8d4d-0f4a0feab2b3-kube-api-access-dsk4m\" (UniqueName: \"kubernetes.io/projected/a5d1a0ce-aa51-4030-8d4d-0f4a0feab2b3-kube-api-access-dsk4m\") pod \"a5d1a0ce-aa51-4030-8d4d-0f4a0feab2b3\" (UID: \"a5d1a0ce-aa51-4030-8d4d-0f4a0feab2b3\") " Oct 31 01:19:17.284239 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:17.284209 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5d1a0ce-aa51-4030-8d4d-0f4a0feab2b3-plugins-config-volume" pod "a5d1a0ce-aa51-4030-8d4d-0f4a0feab2b3" (UID: "a5d1a0ce-aa51-4030-8d4d-0f4a0feab2b3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:19:17.284420 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:17.284402 5222 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5d1a0ce-aa51-4030-8d4d-0f4a0feab2b3-plugins-config-volume\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:19:17.284501 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:17.284427 5222 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcf6a668-1b6d-41d1-89b0-5cbddd61d86f-plugins-config-volume\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:19:17.284501 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:17.284447 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6v5px\" (UniqueName: \"kubernetes.io/projected/dcf6a668-1b6d-41d1-89b0-5cbddd61d86f-kube-api-access-6v5px\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:19:17.286221 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:17.286193 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5d1a0ce-aa51-4030-8d4d-0f4a0feab2b3-kube-api-access-dsk4m" pod "a5d1a0ce-aa51-4030-8d4d-0f4a0feab2b3" (UID: "a5d1a0ce-aa51-4030-8d4d-0f4a0feab2b3"). InnerVolumeSpecName "kube-api-access-dsk4m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:19:17.385236 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:17.385197 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dsk4m\" (UniqueName: \"kubernetes.io/projected/a5d1a0ce-aa51-4030-8d4d-0f4a0feab2b3-kube-api-access-dsk4m\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:19:17.663407 ip-10-0-137-56 kubenswrapper[5222]: E1031 01:19:17.663356 5222 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcf6a668_1b6d_41d1_89b0_5cbddd61d86f.slice/crio-8ee46390647af82337cd7f5a2c0787eb438a3a48fd1a4959303a3b0ea7b4adca\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcf6a668_1b6d_41d1_89b0_5cbddd61d86f.slice\": RecentStats: unable to find data in memory cache]" Oct 31 01:19:17.667987 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:17.667955 5222 generic.go:356] "Generic (PLEG): container finished" podID="a5d1a0ce-aa51-4030-8d4d-0f4a0feab2b3" containerID="991b2ff696a1b508f467841de5649357effb154e0a12023bc9d0d30ed8c495dd" exitCode=0 Oct 31 01:19:17.668134 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:17.668003 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-576e86c8-6876945bcf-2shmj" Oct 31 01:19:17.668134 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:17.668030 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-576e86c8-6876945bcf-2shmj" event={"ID":"a5d1a0ce-aa51-4030-8d4d-0f4a0feab2b3","Type":"ContainerDied","Data":"991b2ff696a1b508f467841de5649357effb154e0a12023bc9d0d30ed8c495dd"} Oct 31 01:19:17.668134 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:17.668091 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-576e86c8-6876945bcf-2shmj" event={"ID":"a5d1a0ce-aa51-4030-8d4d-0f4a0feab2b3","Type":"ContainerDied","Data":"fafddef4d300b9264dbe2901314fc56008c464c15d815b486b8f0a2a276f19d2"} Oct 31 01:19:17.668134 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:17.668120 5222 scope.go:122] "RemoveContainer" containerID="991b2ff696a1b508f467841de5649357effb154e0a12023bc9d0d30ed8c495dd" Oct 31 01:19:17.669469 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:17.669442 5222 generic.go:356] "Generic (PLEG): container finished" podID="dcf6a668-1b6d-41d1-89b0-5cbddd61d86f" containerID="f7b121c8bc5afd1319d9932be5a8b0a96683f7b33fb87c7e79b6b5d022df5533" exitCode=0 Oct 31 01:19:17.669588 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:17.669482 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-576e86c8-6f8546b5c9-txsdf" Oct 31 01:19:17.669588 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:17.669534 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-576e86c8-6f8546b5c9-txsdf" event={"ID":"dcf6a668-1b6d-41d1-89b0-5cbddd61d86f","Type":"ContainerDied","Data":"f7b121c8bc5afd1319d9932be5a8b0a96683f7b33fb87c7e79b6b5d022df5533"} Oct 31 01:19:17.669717 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:17.669593 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-576e86c8-6f8546b5c9-txsdf" event={"ID":"dcf6a668-1b6d-41d1-89b0-5cbddd61d86f","Type":"ContainerDied","Data":"8ee46390647af82337cd7f5a2c0787eb438a3a48fd1a4959303a3b0ea7b4adca"} Oct 31 01:19:17.677472 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:17.677446 5222 scope.go:122] "RemoveContainer" containerID="991b2ff696a1b508f467841de5649357effb154e0a12023bc9d0d30ed8c495dd" Oct 31 01:19:17.677721 ip-10-0-137-56 kubenswrapper[5222]: E1031 01:19:17.677705 5222 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"991b2ff696a1b508f467841de5649357effb154e0a12023bc9d0d30ed8c495dd\": container with ID starting with 991b2ff696a1b508f467841de5649357effb154e0a12023bc9d0d30ed8c495dd not found: ID does not exist" containerID="991b2ff696a1b508f467841de5649357effb154e0a12023bc9d0d30ed8c495dd" Oct 31 01:19:17.677767 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:17.677734 5222 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"991b2ff696a1b508f467841de5649357effb154e0a12023bc9d0d30ed8c495dd"} err="failed to get container status \"991b2ff696a1b508f467841de5649357effb154e0a12023bc9d0d30ed8c495dd\": rpc error: code = NotFound desc = could not find container \"991b2ff696a1b508f467841de5649357effb154e0a12023bc9d0d30ed8c495dd\": container with ID starting with 991b2ff696a1b508f467841de5649357effb154e0a12023bc9d0d30ed8c495dd not found: ID does not exist" Oct 31 01:19:17.677767 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:17.677757 5222 scope.go:122] "RemoveContainer" containerID="f7b121c8bc5afd1319d9932be5a8b0a96683f7b33fb87c7e79b6b5d022df5533" Oct 31 01:19:17.685805 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:17.685784 5222 scope.go:122] "RemoveContainer" containerID="f7b121c8bc5afd1319d9932be5a8b0a96683f7b33fb87c7e79b6b5d022df5533" Oct 31 01:19:17.686126 ip-10-0-137-56 kubenswrapper[5222]: E1031 01:19:17.686103 5222 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7b121c8bc5afd1319d9932be5a8b0a96683f7b33fb87c7e79b6b5d022df5533\": container with ID starting with f7b121c8bc5afd1319d9932be5a8b0a96683f7b33fb87c7e79b6b5d022df5533 not found: ID does not exist" containerID="f7b121c8bc5afd1319d9932be5a8b0a96683f7b33fb87c7e79b6b5d022df5533" Oct 31 01:19:17.686212 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:17.686132 5222 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7b121c8bc5afd1319d9932be5a8b0a96683f7b33fb87c7e79b6b5d022df5533"} err="failed to get container status \"f7b121c8bc5afd1319d9932be5a8b0a96683f7b33fb87c7e79b6b5d022df5533\": rpc error: code = NotFound desc = could not find container \"f7b121c8bc5afd1319d9932be5a8b0a96683f7b33fb87c7e79b6b5d022df5533\": container with ID starting with f7b121c8bc5afd1319d9932be5a8b0a96683f7b33fb87c7e79b6b5d022df5533 not found: ID does not exist" Oct 31 01:19:17.836360 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:17.836324 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-576e86c8-6f8546b5c9-txsdf"] Oct 31 01:19:17.840808 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:17.840779 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-576e86c8-6f8546b5c9-txsdf"] Oct 31 01:19:17.952032 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:17.951952 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-576e86c8-6876945bcf-2shmj"] Oct 31 01:19:17.955820 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:17.955794 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-576e86c8-6876945bcf-2shmj"] Oct 31 01:19:19.654499 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:19.654465 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5d1a0ce-aa51-4030-8d4d-0f4a0feab2b3" path="/var/lib/kubelet/pods/a5d1a0ce-aa51-4030-8d4d-0f4a0feab2b3/volumes" Oct 31 01:19:19.654901 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:19.654889 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcf6a668-1b6d-41d1-89b0-5cbddd61d86f" path="/var/lib/kubelet/pods/dcf6a668-1b6d-41d1-89b0-5cbddd61d86f/volumes" Oct 31 01:19:22.574260 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:22.574220 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-576e86c8-ggldm"] Oct 31 01:19:22.625216 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:22.625181 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-576e86c8-ggldm"] Oct 31 01:19:23.654278 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:23.654239 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1e87fb0-00ca-41d4-8a32-67df2f86d36e" path="/var/lib/kubelet/pods/d1e87fb0-00ca-41d4-8a32-67df2f86d36e/volumes" Oct 31 01:19:25.128561 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:25.128529 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-79tk8"] Oct 31 01:19:26.552204 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:26.552167 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-576e86c8-openshift-default-98874d9b5-w9brf"] Oct 31 01:19:26.552746 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:26.552427 5222 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-576e86c8-openshift-default-98874d9b5-w9brf" podUID="d2ee6d42-4edc-4633-ae68-519fb4ed9bae" containerName="istio-proxy" containerID="cri-o://bd49c93418cf7b1e8a5d6ab881aaca272e509cab931d40f8b99dfe2cdb47820a" gracePeriod=30 Oct 31 01:19:28.417140 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:28.417101 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-authcfg-http-44d22c-openshift-default-6b997dc7fc-dgpzm"] Oct 31 01:19:28.417589 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:28.417504 5222 cpu_manager.go:414] "containerMap: removing container" podUID="a5d1a0ce-aa51-4030-8d4d-0f4a0feab2b3" containerName="payload-pre-processing" Oct 31 01:19:28.417589 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:28.417516 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5d1a0ce-aa51-4030-8d4d-0f4a0feab2b3" containerName="payload-pre-processing" Oct 31 01:19:28.417589 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:28.417551 5222 cpu_manager.go:414] "containerMap: removing container" podUID="d1e87fb0-00ca-41d4-8a32-67df2f86d36e" containerName="revoke-keys" Oct 31 01:19:28.417589 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:28.417556 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1e87fb0-00ca-41d4-8a32-67df2f86d36e" containerName="revoke-keys" Oct 31 01:19:28.417589 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:28.417565 5222 cpu_manager.go:414] "containerMap: removing container" podUID="dcf6a668-1b6d-41d1-89b0-5cbddd61d86f" containerName="payload-processing" Oct 31 01:19:28.417589 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:28.417569 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcf6a668-1b6d-41d1-89b0-5cbddd61d86f" containerName="payload-processing" Oct 31 01:19:28.417795 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:28.417652 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1e87fb0-00ca-41d4-8a32-67df2f86d36e" containerName="revoke-keys" Oct 31 01:19:28.417795 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:28.417660 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcf6a668-1b6d-41d1-89b0-5cbddd61d86f" containerName="payload-processing" Oct 31 01:19:28.417795 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:28.417668 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5d1a0ce-aa51-4030-8d4d-0f4a0feab2b3" containerName="payload-pre-processing" Oct 31 01:19:28.422512 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:28.422490 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-44d22c-openshift-default-6b997dc7fc-dgpzm" Oct 31 01:19:28.482830 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:28.482795 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-authcfg-http-44d22c-openshift-default-dockercfg-nlc2q\"" Oct 31 01:19:28.512029 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:28.511993 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b01d020-5db4-48ea-945d-af5f2e4a5347-istio-data\") pod \"e2e-authcfg-http-44d22c-openshift-default-6b997dc7fc-dgpzm\" (UID: \"9b01d020-5db4-48ea-945d-af5f2e4a5347\") " pod="openshift-ingress/e2e-authcfg-http-44d22c-openshift-default-6b997dc7fc-dgpzm" Oct 31 01:19:28.512029 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:28.512030 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b01d020-5db4-48ea-945d-af5f2e4a5347-istio-podinfo\") pod \"e2e-authcfg-http-44d22c-openshift-default-6b997dc7fc-dgpzm\" (UID: \"9b01d020-5db4-48ea-945d-af5f2e4a5347\") " pod="openshift-ingress/e2e-authcfg-http-44d22c-openshift-default-6b997dc7fc-dgpzm" Oct 31 01:19:28.512248 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:28.512049 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b01d020-5db4-48ea-945d-af5f2e4a5347-istio-envoy\") pod \"e2e-authcfg-http-44d22c-openshift-default-6b997dc7fc-dgpzm\" (UID: \"9b01d020-5db4-48ea-945d-af5f2e4a5347\") " pod="openshift-ingress/e2e-authcfg-http-44d22c-openshift-default-6b997dc7fc-dgpzm" Oct 31 01:19:28.512248 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:28.512139 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdbwv\" (UniqueName: \"kubernetes.io/projected/9b01d020-5db4-48ea-945d-af5f2e4a5347-kube-api-access-gdbwv\") pod \"e2e-authcfg-http-44d22c-openshift-default-6b997dc7fc-dgpzm\" (UID: \"9b01d020-5db4-48ea-945d-af5f2e4a5347\") " pod="openshift-ingress/e2e-authcfg-http-44d22c-openshift-default-6b997dc7fc-dgpzm" Oct 31 01:19:28.512248 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:28.512161 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b01d020-5db4-48ea-945d-af5f2e4a5347-workload-socket\") pod \"e2e-authcfg-http-44d22c-openshift-default-6b997dc7fc-dgpzm\" (UID: \"9b01d020-5db4-48ea-945d-af5f2e4a5347\") " pod="openshift-ingress/e2e-authcfg-http-44d22c-openshift-default-6b997dc7fc-dgpzm" Oct 31 01:19:28.512248 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:28.512184 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b01d020-5db4-48ea-945d-af5f2e4a5347-workload-certs\") pod \"e2e-authcfg-http-44d22c-openshift-default-6b997dc7fc-dgpzm\" (UID: \"9b01d020-5db4-48ea-945d-af5f2e4a5347\") " pod="openshift-ingress/e2e-authcfg-http-44d22c-openshift-default-6b997dc7fc-dgpzm" Oct 31 01:19:28.512412 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:28.512330 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b01d020-5db4-48ea-945d-af5f2e4a5347-credential-socket\") pod \"e2e-authcfg-http-44d22c-openshift-default-6b997dc7fc-dgpzm\" (UID: \"9b01d020-5db4-48ea-945d-af5f2e4a5347\") " pod="openshift-ingress/e2e-authcfg-http-44d22c-openshift-default-6b997dc7fc-dgpzm" Oct 31 01:19:28.512412 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:28.512361 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b01d020-5db4-48ea-945d-af5f2e4a5347-istiod-ca-cert\") pod \"e2e-authcfg-http-44d22c-openshift-default-6b997dc7fc-dgpzm\" (UID: \"9b01d020-5db4-48ea-945d-af5f2e4a5347\") " pod="openshift-ingress/e2e-authcfg-http-44d22c-openshift-default-6b997dc7fc-dgpzm" Oct 31 01:19:28.512504 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:28.512455 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b01d020-5db4-48ea-945d-af5f2e4a5347-istio-token\") pod \"e2e-authcfg-http-44d22c-openshift-default-6b997dc7fc-dgpzm\" (UID: \"9b01d020-5db4-48ea-945d-af5f2e4a5347\") " pod="openshift-ingress/e2e-authcfg-http-44d22c-openshift-default-6b997dc7fc-dgpzm" Oct 31 01:19:28.535123 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:28.535092 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-44d22c-openshift-default-6b997dc7fc-dgpzm"] Oct 31 01:19:28.613655 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:28.613613 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b01d020-5db4-48ea-945d-af5f2e4a5347-istio-data\") pod \"e2e-authcfg-http-44d22c-openshift-default-6b997dc7fc-dgpzm\" (UID: \"9b01d020-5db4-48ea-945d-af5f2e4a5347\") " pod="openshift-ingress/e2e-authcfg-http-44d22c-openshift-default-6b997dc7fc-dgpzm" Oct 31 01:19:28.613655 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:28.613656 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b01d020-5db4-48ea-945d-af5f2e4a5347-istio-podinfo\") pod \"e2e-authcfg-http-44d22c-openshift-default-6b997dc7fc-dgpzm\" (UID: \"9b01d020-5db4-48ea-945d-af5f2e4a5347\") " pod="openshift-ingress/e2e-authcfg-http-44d22c-openshift-default-6b997dc7fc-dgpzm" Oct 31 01:19:28.613902 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:28.613681 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b01d020-5db4-48ea-945d-af5f2e4a5347-istio-envoy\") pod \"e2e-authcfg-http-44d22c-openshift-default-6b997dc7fc-dgpzm\" (UID: \"9b01d020-5db4-48ea-945d-af5f2e4a5347\") " pod="openshift-ingress/e2e-authcfg-http-44d22c-openshift-default-6b997dc7fc-dgpzm" Oct 31 01:19:28.613902 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:28.613728 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdbwv\" (UniqueName: \"kubernetes.io/projected/9b01d020-5db4-48ea-945d-af5f2e4a5347-kube-api-access-gdbwv\") pod \"e2e-authcfg-http-44d22c-openshift-default-6b997dc7fc-dgpzm\" (UID: \"9b01d020-5db4-48ea-945d-af5f2e4a5347\") " pod="openshift-ingress/e2e-authcfg-http-44d22c-openshift-default-6b997dc7fc-dgpzm" Oct 31 01:19:28.613902 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:28.613758 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b01d020-5db4-48ea-945d-af5f2e4a5347-workload-socket\") pod \"e2e-authcfg-http-44d22c-openshift-default-6b997dc7fc-dgpzm\" (UID: \"9b01d020-5db4-48ea-945d-af5f2e4a5347\") " pod="openshift-ingress/e2e-authcfg-http-44d22c-openshift-default-6b997dc7fc-dgpzm" Oct 31 01:19:28.613902 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:28.613792 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b01d020-5db4-48ea-945d-af5f2e4a5347-workload-certs\") pod \"e2e-authcfg-http-44d22c-openshift-default-6b997dc7fc-dgpzm\" (UID: \"9b01d020-5db4-48ea-945d-af5f2e4a5347\") " pod="openshift-ingress/e2e-authcfg-http-44d22c-openshift-default-6b997dc7fc-dgpzm" Oct 31 01:19:28.613902 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:28.613862 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b01d020-5db4-48ea-945d-af5f2e4a5347-credential-socket\") pod \"e2e-authcfg-http-44d22c-openshift-default-6b997dc7fc-dgpzm\" (UID: \"9b01d020-5db4-48ea-945d-af5f2e4a5347\") " pod="openshift-ingress/e2e-authcfg-http-44d22c-openshift-default-6b997dc7fc-dgpzm" Oct 31 01:19:28.613902 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:28.613892 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b01d020-5db4-48ea-945d-af5f2e4a5347-istiod-ca-cert\") pod \"e2e-authcfg-http-44d22c-openshift-default-6b997dc7fc-dgpzm\" (UID: \"9b01d020-5db4-48ea-945d-af5f2e4a5347\") " pod="openshift-ingress/e2e-authcfg-http-44d22c-openshift-default-6b997dc7fc-dgpzm" Oct 31 01:19:28.614243 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:28.613942 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b01d020-5db4-48ea-945d-af5f2e4a5347-istio-token\") pod \"e2e-authcfg-http-44d22c-openshift-default-6b997dc7fc-dgpzm\" (UID: \"9b01d020-5db4-48ea-945d-af5f2e4a5347\") " pod="openshift-ingress/e2e-authcfg-http-44d22c-openshift-default-6b997dc7fc-dgpzm" Oct 31 01:19:28.614243 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:28.614090 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b01d020-5db4-48ea-945d-af5f2e4a5347-istio-data\") pod \"e2e-authcfg-http-44d22c-openshift-default-6b997dc7fc-dgpzm\" (UID: \"9b01d020-5db4-48ea-945d-af5f2e4a5347\") " pod="openshift-ingress/e2e-authcfg-http-44d22c-openshift-default-6b997dc7fc-dgpzm" Oct 31 01:19:28.614243 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:28.614170 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b01d020-5db4-48ea-945d-af5f2e4a5347-workload-socket\") pod \"e2e-authcfg-http-44d22c-openshift-default-6b997dc7fc-dgpzm\" (UID: \"9b01d020-5db4-48ea-945d-af5f2e4a5347\") " pod="openshift-ingress/e2e-authcfg-http-44d22c-openshift-default-6b997dc7fc-dgpzm" Oct 31 01:19:28.614477 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:28.614442 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b01d020-5db4-48ea-945d-af5f2e4a5347-workload-certs\") pod \"e2e-authcfg-http-44d22c-openshift-default-6b997dc7fc-dgpzm\" (UID: \"9b01d020-5db4-48ea-945d-af5f2e4a5347\") " pod="openshift-ingress/e2e-authcfg-http-44d22c-openshift-default-6b997dc7fc-dgpzm" Oct 31 01:19:28.614560 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:28.614519 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b01d020-5db4-48ea-945d-af5f2e4a5347-credential-socket\") pod \"e2e-authcfg-http-44d22c-openshift-default-6b997dc7fc-dgpzm\" (UID: \"9b01d020-5db4-48ea-945d-af5f2e4a5347\") " pod="openshift-ingress/e2e-authcfg-http-44d22c-openshift-default-6b997dc7fc-dgpzm" Oct 31 01:19:28.614802 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:28.614776 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b01d020-5db4-48ea-945d-af5f2e4a5347-istiod-ca-cert\") pod \"e2e-authcfg-http-44d22c-openshift-default-6b997dc7fc-dgpzm\" (UID: \"9b01d020-5db4-48ea-945d-af5f2e4a5347\") " pod="openshift-ingress/e2e-authcfg-http-44d22c-openshift-default-6b997dc7fc-dgpzm" Oct 31 01:19:28.616923 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:28.616899 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b01d020-5db4-48ea-945d-af5f2e4a5347-istio-envoy\") pod \"e2e-authcfg-http-44d22c-openshift-default-6b997dc7fc-dgpzm\" (UID: \"9b01d020-5db4-48ea-945d-af5f2e4a5347\") " pod="openshift-ingress/e2e-authcfg-http-44d22c-openshift-default-6b997dc7fc-dgpzm" Oct 31 01:19:28.617134 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:28.617116 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b01d020-5db4-48ea-945d-af5f2e4a5347-istio-podinfo\") pod \"e2e-authcfg-http-44d22c-openshift-default-6b997dc7fc-dgpzm\" (UID: \"9b01d020-5db4-48ea-945d-af5f2e4a5347\") " pod="openshift-ingress/e2e-authcfg-http-44d22c-openshift-default-6b997dc7fc-dgpzm" Oct 31 01:19:28.676837 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:28.676764 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b01d020-5db4-48ea-945d-af5f2e4a5347-istio-token\") pod \"e2e-authcfg-http-44d22c-openshift-default-6b997dc7fc-dgpzm\" (UID: \"9b01d020-5db4-48ea-945d-af5f2e4a5347\") " pod="openshift-ingress/e2e-authcfg-http-44d22c-openshift-default-6b997dc7fc-dgpzm" Oct 31 01:19:28.676988 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:28.676945 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdbwv\" (UniqueName: \"kubernetes.io/projected/9b01d020-5db4-48ea-945d-af5f2e4a5347-kube-api-access-gdbwv\") pod \"e2e-authcfg-http-44d22c-openshift-default-6b997dc7fc-dgpzm\" (UID: \"9b01d020-5db4-48ea-945d-af5f2e4a5347\") " pod="openshift-ingress/e2e-authcfg-http-44d22c-openshift-default-6b997dc7fc-dgpzm" Oct 31 01:19:28.736626 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:28.736591 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-44d22c-openshift-default-6b997dc7fc-dgpzm" Oct 31 01:19:29.160426 ip-10-0-137-56 kubenswrapper[5222]: W1031 01:19:29.160396 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b01d020_5db4_48ea_945d_af5f2e4a5347.slice/crio-59d5b75bf268a77b3bdbd9937894832c5f9d73559727195ac3d6f09d2eee64ba WatchSource:0}: Error finding container 59d5b75bf268a77b3bdbd9937894832c5f9d73559727195ac3d6f09d2eee64ba: Status 404 returned error can't find the container with id 59d5b75bf268a77b3bdbd9937894832c5f9d73559727195ac3d6f09d2eee64ba Oct 31 01:19:29.162788 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:29.162749 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 01:19:29.162916 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:29.162835 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 01:19:29.162916 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:29.162880 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 01:19:29.163070 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:29.163050 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-44d22c-openshift-default-6b997dc7fc-dgpzm"] Oct 31 01:19:29.732245 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:29.732198 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-44d22c-openshift-default-6b997dc7fc-dgpzm" event={"ID":"9b01d020-5db4-48ea-945d-af5f2e4a5347","Type":"ContainerStarted","Data":"9ef7909f6e9ee77e937cb9a42b7a13a0e9a8651e89117b143e44d7120a387909"} Oct 31 01:19:29.732245 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:29.732248 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-44d22c-openshift-default-6b997dc7fc-dgpzm" event={"ID":"9b01d020-5db4-48ea-945d-af5f2e4a5347","Type":"ContainerStarted","Data":"59d5b75bf268a77b3bdbd9937894832c5f9d73559727195ac3d6f09d2eee64ba"} Oct 31 01:19:29.737683 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:29.737623 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-44d22c-openshift-default-6b997dc7fc-dgpzm" Oct 31 01:19:29.856651 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:29.856597 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-authcfg-http-44d22c-openshift-default-6b997dc7fc-dgpzm" podStartSLOduration=1.85658305 podStartE2EDuration="1.85658305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:19:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:19:29.855338973 +0000 UTC m=+3862.326508726" watchObservedRunningTime="2026-10-31 01:19:29.85658305 +0000 UTC m=+3862.327752790" Oct 31 01:19:30.742180 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:30.742150 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-44d22c-openshift-default-6b997dc7fc-dgpzm" Oct 31 01:19:31.745563 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:31.745523 5222 generic.go:356] "Generic (PLEG): container finished" podID="d2ee6d42-4edc-4633-ae68-519fb4ed9bae" containerID="bd49c93418cf7b1e8a5d6ab881aaca272e509cab931d40f8b99dfe2cdb47820a" exitCode=0 Oct 31 01:19:31.746706 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:31.746673 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-576e86c8-openshift-default-98874d9b5-w9brf" event={"ID":"d2ee6d42-4edc-4633-ae68-519fb4ed9bae","Type":"ContainerDied","Data":"bd49c93418cf7b1e8a5d6ab881aaca272e509cab931d40f8b99dfe2cdb47820a"} Oct 31 01:19:31.746838 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:31.746747 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-44d22c-openshift-default-6b997dc7fc-dgpzm" Oct 31 01:19:31.746838 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:31.746783 5222 scope.go:122] "RemoveContainer" containerID="3270990a928452289d9c62b91efb3518104381c534054225fdee956083f46a09" Oct 31 01:19:31.749012 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:31.747972 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-44d22c-openshift-default-6b997dc7fc-dgpzm" Oct 31 01:19:31.813812 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:31.813784 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-576e86c8-openshift-default-98874d9b5-w9brf" Oct 31 01:19:31.950762 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:31.950683 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d2ee6d42-4edc-4633-ae68-519fb4ed9bae-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2ee6d42-4edc-4633-ae68-519fb4ed9bae-istio-podinfo\") pod \"d2ee6d42-4edc-4633-ae68-519fb4ed9bae\" (UID: \"d2ee6d42-4edc-4633-ae68-519fb4ed9bae\") " Oct 31 01:19:31.950762 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:31.950748 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2ee6d42-4edc-4633-ae68-519fb4ed9bae-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2ee6d42-4edc-4633-ae68-519fb4ed9bae-workload-certs\") pod \"d2ee6d42-4edc-4633-ae68-519fb4ed9bae\" (UID: \"d2ee6d42-4edc-4633-ae68-519fb4ed9bae\") " Oct 31 01:19:31.951017 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:31.950769 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2ee6d42-4edc-4633-ae68-519fb4ed9bae-istio-token\" (UniqueName: \"kubernetes.io/projected/d2ee6d42-4edc-4633-ae68-519fb4ed9bae-istio-token\") pod \"d2ee6d42-4edc-4633-ae68-519fb4ed9bae\" (UID: \"d2ee6d42-4edc-4633-ae68-519fb4ed9bae\") " Oct 31 01:19:31.951017 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:31.950830 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2ee6d42-4edc-4633-ae68-519fb4ed9bae-kube-api-access-7p846\" (UniqueName: \"kubernetes.io/projected/d2ee6d42-4edc-4633-ae68-519fb4ed9bae-kube-api-access-7p846\") pod \"d2ee6d42-4edc-4633-ae68-519fb4ed9bae\" (UID: \"d2ee6d42-4edc-4633-ae68-519fb4ed9bae\") " Oct 31 01:19:31.951017 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:31.950851 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2ee6d42-4edc-4633-ae68-519fb4ed9bae-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2ee6d42-4edc-4633-ae68-519fb4ed9bae-workload-socket\") pod \"d2ee6d42-4edc-4633-ae68-519fb4ed9bae\" (UID: \"d2ee6d42-4edc-4633-ae68-519fb4ed9bae\") " Oct 31 01:19:31.951017 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:31.950889 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2ee6d42-4edc-4633-ae68-519fb4ed9bae-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2ee6d42-4edc-4633-ae68-519fb4ed9bae-istiod-ca-cert\") pod \"d2ee6d42-4edc-4633-ae68-519fb4ed9bae\" (UID: \"d2ee6d42-4edc-4633-ae68-519fb4ed9bae\") " Oct 31 01:19:31.951017 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:31.950922 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2ee6d42-4edc-4633-ae68-519fb4ed9bae-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2ee6d42-4edc-4633-ae68-519fb4ed9bae-istio-envoy\") pod \"d2ee6d42-4edc-4633-ae68-519fb4ed9bae\" (UID: \"d2ee6d42-4edc-4633-ae68-519fb4ed9bae\") " Oct 31 01:19:31.951017 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:31.950960 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2ee6d42-4edc-4633-ae68-519fb4ed9bae-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2ee6d42-4edc-4633-ae68-519fb4ed9bae-istio-data\") pod \"d2ee6d42-4edc-4633-ae68-519fb4ed9bae\" (UID: \"d2ee6d42-4edc-4633-ae68-519fb4ed9bae\") " Oct 31 01:19:31.951017 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:31.951012 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2ee6d42-4edc-4633-ae68-519fb4ed9bae-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2ee6d42-4edc-4633-ae68-519fb4ed9bae-credential-socket\") pod \"d2ee6d42-4edc-4633-ae68-519fb4ed9bae\" (UID: \"d2ee6d42-4edc-4633-ae68-519fb4ed9bae\") " Oct 31 01:19:31.951438 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:31.951107 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2ee6d42-4edc-4633-ae68-519fb4ed9bae-workload-certs" pod "d2ee6d42-4edc-4633-ae68-519fb4ed9bae" (UID: "d2ee6d42-4edc-4633-ae68-519fb4ed9bae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:19:31.951438 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:31.951198 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2ee6d42-4edc-4633-ae68-519fb4ed9bae-workload-socket" pod "d2ee6d42-4edc-4633-ae68-519fb4ed9bae" (UID: "d2ee6d42-4edc-4633-ae68-519fb4ed9bae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:19:31.951438 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:31.951327 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2ee6d42-4edc-4633-ae68-519fb4ed9bae-istiod-ca-cert" pod "d2ee6d42-4edc-4633-ae68-519fb4ed9bae" (UID: "d2ee6d42-4edc-4633-ae68-519fb4ed9bae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:19:31.951438 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:31.951347 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2ee6d42-4edc-4633-ae68-519fb4ed9bae-istio-data" pod "d2ee6d42-4edc-4633-ae68-519fb4ed9bae" (UID: "d2ee6d42-4edc-4633-ae68-519fb4ed9bae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:19:31.951641 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:31.951455 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2ee6d42-4edc-4633-ae68-519fb4ed9bae-credential-socket" pod "d2ee6d42-4edc-4633-ae68-519fb4ed9bae" (UID: "d2ee6d42-4edc-4633-ae68-519fb4ed9bae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:19:31.951839 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:31.951815 5222 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2ee6d42-4edc-4633-ae68-519fb4ed9bae-workload-certs\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:19:31.951895 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:31.951846 5222 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2ee6d42-4edc-4633-ae68-519fb4ed9bae-workload-socket\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:19:31.951895 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:31.951863 5222 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2ee6d42-4edc-4633-ae68-519fb4ed9bae-istiod-ca-cert\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:19:31.951895 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:31.951877 5222 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2ee6d42-4edc-4633-ae68-519fb4ed9bae-istio-data\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:19:31.951992 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:31.951893 5222 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2ee6d42-4edc-4633-ae68-519fb4ed9bae-credential-socket\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:19:31.953181 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:31.953155 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d2ee6d42-4edc-4633-ae68-519fb4ed9bae-istio-podinfo" pod "d2ee6d42-4edc-4633-ae68-519fb4ed9bae" (UID: "d2ee6d42-4edc-4633-ae68-519fb4ed9bae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:19:31.953181 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:31.953173 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2ee6d42-4edc-4633-ae68-519fb4ed9bae-kube-api-access-7p846" pod "d2ee6d42-4edc-4633-ae68-519fb4ed9bae" (UID: "d2ee6d42-4edc-4633-ae68-519fb4ed9bae"). InnerVolumeSpecName "kube-api-access-7p846". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:19:31.953786 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:31.953764 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2ee6d42-4edc-4633-ae68-519fb4ed9bae-istio-envoy" pod "d2ee6d42-4edc-4633-ae68-519fb4ed9bae" (UID: "d2ee6d42-4edc-4633-ae68-519fb4ed9bae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:19:31.953786 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:31.953766 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2ee6d42-4edc-4633-ae68-519fb4ed9bae-istio-token" pod "d2ee6d42-4edc-4633-ae68-519fb4ed9bae" (UID: "d2ee6d42-4edc-4633-ae68-519fb4ed9bae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:19:32.053512 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:32.053474 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7p846\" (UniqueName: \"kubernetes.io/projected/d2ee6d42-4edc-4633-ae68-519fb4ed9bae-kube-api-access-7p846\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:19:32.053512 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:32.053505 5222 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2ee6d42-4edc-4633-ae68-519fb4ed9bae-istio-envoy\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:19:32.053512 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:32.053515 5222 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2ee6d42-4edc-4633-ae68-519fb4ed9bae-istio-podinfo\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:19:32.053758 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:32.053524 5222 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2ee6d42-4edc-4633-ae68-519fb4ed9bae-istio-token\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:19:32.752496 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:32.752463 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-576e86c8-openshift-default-98874d9b5-w9brf" Oct 31 01:19:32.752983 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:32.752493 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-576e86c8-openshift-default-98874d9b5-w9brf" event={"ID":"d2ee6d42-4edc-4633-ae68-519fb4ed9bae","Type":"ContainerDied","Data":"33464a94ff292fd09a7bd40e16e3f333bfc6a8457bf7c343c3de34a29c1cd97c"} Oct 31 01:19:32.752983 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:32.752553 5222 scope.go:122] "RemoveContainer" containerID="bd49c93418cf7b1e8a5d6ab881aaca272e509cab931d40f8b99dfe2cdb47820a" Oct 31 01:19:32.973538 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:32.973498 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-576e86c8-openshift-default-98874d9b5-w9brf"] Oct 31 01:19:33.024298 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:33.024258 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-576e86c8-openshift-default-98874d9b5-w9brf"] Oct 31 01:19:33.654791 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:33.654757 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2ee6d42-4edc-4633-ae68-519fb4ed9bae" path="/var/lib/kubelet/pods/d2ee6d42-4edc-4633-ae68-519fb4ed9bae/volumes" Oct 31 01:19:35.889607 ip-10-0-137-56 kubenswrapper[5222]: E1031 01:19:35.889570 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 01:19:36.263510 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:36.263427 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-44d22c-649bc5675f-zvc4z"] Oct 31 01:19:36.263829 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:36.263817 5222 cpu_manager.go:414] "containerMap: removing container" podUID="d2ee6d42-4edc-4633-ae68-519fb4ed9bae" containerName="istio-proxy" Oct 31 01:19:36.263875 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:36.263831 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2ee6d42-4edc-4633-ae68-519fb4ed9bae" containerName="istio-proxy" Oct 31 01:19:36.263875 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:36.263861 5222 cpu_manager.go:414] "containerMap: removing container" podUID="d2ee6d42-4edc-4633-ae68-519fb4ed9bae" containerName="istio-proxy" Oct 31 01:19:36.263875 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:36.263866 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2ee6d42-4edc-4633-ae68-519fb4ed9bae" containerName="istio-proxy" Oct 31 01:19:36.263979 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:36.263924 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2ee6d42-4edc-4633-ae68-519fb4ed9bae" containerName="istio-proxy" Oct 31 01:19:36.268443 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:36.268411 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-44d22c-649bc5675f-zvc4z" Oct 31 01:19:36.322423 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:36.322391 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-44d22c-649bc5675f-zvc4z"] Oct 31 01:19:36.345685 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:36.345607 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-authcfg-http-44d22c\"" Oct 31 01:19:36.371807 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:36.371776 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-authcfg-http-44d22c-dockercfg-jxx99\"" Oct 31 01:19:36.400337 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:36.400301 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qqkb\" (UniqueName: \"kubernetes.io/projected/0adf21e2-d5cd-4c6e-9b6a-567bc7880f36-kube-api-access-2qqkb\") pod \"payload-pre-processing-e2e-authcfg-http-44d22c-649bc5675f-zvc4z\" (UID: \"0adf21e2-d5cd-4c6e-9b6a-567bc7880f36\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-44d22c-649bc5675f-zvc4z" Oct 31 01:19:36.400337 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:36.400350 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0adf21e2-d5cd-4c6e-9b6a-567bc7880f36-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-44d22c-649bc5675f-zvc4z\" (UID: \"0adf21e2-d5cd-4c6e-9b6a-567bc7880f36\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-44d22c-649bc5675f-zvc4z" Oct 31 01:19:36.438449 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:36.438414 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-44d22c-9c679cbbf-rmpq6"] Oct 31 01:19:36.438903 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:36.438890 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2ee6d42-4edc-4633-ae68-519fb4ed9bae" containerName="istio-proxy" Oct 31 01:19:36.441903 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:36.441886 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-44d22c-9c679cbbf-rmpq6" Oct 31 01:19:36.496386 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:36.496343 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-44d22c-9c679cbbf-rmpq6"] Oct 31 01:19:36.501281 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:36.501249 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qqkb\" (UniqueName: \"kubernetes.io/projected/0adf21e2-d5cd-4c6e-9b6a-567bc7880f36-kube-api-access-2qqkb\") pod \"payload-pre-processing-e2e-authcfg-http-44d22c-649bc5675f-zvc4z\" (UID: \"0adf21e2-d5cd-4c6e-9b6a-567bc7880f36\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-44d22c-649bc5675f-zvc4z" Oct 31 01:19:36.501490 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:36.501298 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2cbe681-d873-40f2-b768-32bfff6d2acf-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-44d22c-9c679cbbf-rmpq6\" (UID: \"a2cbe681-d873-40f2-b768-32bfff6d2acf\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-44d22c-9c679cbbf-rmpq6" Oct 31 01:19:36.501490 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:36.501334 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0adf21e2-d5cd-4c6e-9b6a-567bc7880f36-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-44d22c-649bc5675f-zvc4z\" (UID: \"0adf21e2-d5cd-4c6e-9b6a-567bc7880f36\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-44d22c-649bc5675f-zvc4z" Oct 31 01:19:36.501490 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:36.501396 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vm79z\" (UniqueName: \"kubernetes.io/projected/a2cbe681-d873-40f2-b768-32bfff6d2acf-kube-api-access-vm79z\") pod \"payload-processing-e2e-authcfg-http-44d22c-9c679cbbf-rmpq6\" (UID: \"a2cbe681-d873-40f2-b768-32bfff6d2acf\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-44d22c-9c679cbbf-rmpq6" Oct 31 01:19:36.502300 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:36.502272 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0adf21e2-d5cd-4c6e-9b6a-567bc7880f36-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-44d22c-649bc5675f-zvc4z\" (UID: \"0adf21e2-d5cd-4c6e-9b6a-567bc7880f36\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-44d22c-649bc5675f-zvc4z" Oct 31 01:19:36.565998 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:36.565968 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qqkb\" (UniqueName: \"kubernetes.io/projected/0adf21e2-d5cd-4c6e-9b6a-567bc7880f36-kube-api-access-2qqkb\") pod \"payload-pre-processing-e2e-authcfg-http-44d22c-649bc5675f-zvc4z\" (UID: \"0adf21e2-d5cd-4c6e-9b6a-567bc7880f36\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-44d22c-649bc5675f-zvc4z" Oct 31 01:19:36.584877 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:36.584843 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-44d22c-649bc5675f-zvc4z" Oct 31 01:19:36.602654 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:36.602622 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2cbe681-d873-40f2-b768-32bfff6d2acf-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-44d22c-9c679cbbf-rmpq6\" (UID: \"a2cbe681-d873-40f2-b768-32bfff6d2acf\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-44d22c-9c679cbbf-rmpq6" Oct 31 01:19:36.602769 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:36.602678 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vm79z\" (UniqueName: \"kubernetes.io/projected/a2cbe681-d873-40f2-b768-32bfff6d2acf-kube-api-access-vm79z\") pod \"payload-processing-e2e-authcfg-http-44d22c-9c679cbbf-rmpq6\" (UID: \"a2cbe681-d873-40f2-b768-32bfff6d2acf\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-44d22c-9c679cbbf-rmpq6" Oct 31 01:19:36.603613 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:36.603585 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2cbe681-d873-40f2-b768-32bfff6d2acf-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-44d22c-9c679cbbf-rmpq6\" (UID: \"a2cbe681-d873-40f2-b768-32bfff6d2acf\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-44d22c-9c679cbbf-rmpq6" Oct 31 01:19:36.668690 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:36.668619 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vm79z\" (UniqueName: \"kubernetes.io/projected/a2cbe681-d873-40f2-b768-32bfff6d2acf-kube-api-access-vm79z\") pod \"payload-processing-e2e-authcfg-http-44d22c-9c679cbbf-rmpq6\" (UID: \"a2cbe681-d873-40f2-b768-32bfff6d2acf\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-44d22c-9c679cbbf-rmpq6" Oct 31 01:19:36.755854 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:36.755814 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-44d22c-9c679cbbf-rmpq6" Oct 31 01:19:36.959257 ip-10-0-137-56 kubenswrapper[5222]: W1031 01:19:36.959222 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0adf21e2_d5cd_4c6e_9b6a_567bc7880f36.slice/crio-69214b883949a4e2204ca759b0257148d0a75be0b2ef868afa39f9e903ebc8f1 WatchSource:0}: Error finding container 69214b883949a4e2204ca759b0257148d0a75be0b2ef868afa39f9e903ebc8f1: Status 404 returned error can't find the container with id 69214b883949a4e2204ca759b0257148d0a75be0b2ef868afa39f9e903ebc8f1 Oct 31 01:19:36.961058 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:36.961037 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-44d22c-649bc5675f-zvc4z"] Oct 31 01:19:37.129754 ip-10-0-137-56 kubenswrapper[5222]: W1031 01:19:37.129723 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2cbe681_d873_40f2_b768_32bfff6d2acf.slice/crio-92777ae52bc89c40be463f0e38855d15ec005de1dcd9f8f809bd5d1f7423c524 WatchSource:0}: Error finding container 92777ae52bc89c40be463f0e38855d15ec005de1dcd9f8f809bd5d1f7423c524: Status 404 returned error can't find the container with id 92777ae52bc89c40be463f0e38855d15ec005de1dcd9f8f809bd5d1f7423c524 Oct 31 01:19:37.131364 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:37.131338 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-44d22c-9c679cbbf-rmpq6"] Oct 31 01:19:37.779825 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:37.779785 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-44d22c-649bc5675f-zvc4z" event={"ID":"0adf21e2-d5cd-4c6e-9b6a-567bc7880f36","Type":"ContainerStarted","Data":"a1d156b2fdf3ef3b1d2598264a577b5adf8bdfa85f946dac8549633e866b20fc"} Oct 31 01:19:37.780045 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:37.779847 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-44d22c-649bc5675f-zvc4z" event={"ID":"0adf21e2-d5cd-4c6e-9b6a-567bc7880f36","Type":"ContainerStarted","Data":"69214b883949a4e2204ca759b0257148d0a75be0b2ef868afa39f9e903ebc8f1"} Oct 31 01:19:37.780045 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:37.779898 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-44d22c-649bc5675f-zvc4z" Oct 31 01:19:37.781641 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:37.781617 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-44d22c-9c679cbbf-rmpq6" event={"ID":"a2cbe681-d873-40f2-b768-32bfff6d2acf","Type":"ContainerStarted","Data":"18f1029bf135bc31a25b4fddca48c3ce6f6774a9c6da6b877c7d3c0b80e407d2"} Oct 31 01:19:37.781759 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:37.781659 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-44d22c-9c679cbbf-rmpq6" event={"ID":"a2cbe681-d873-40f2-b768-32bfff6d2acf","Type":"ContainerStarted","Data":"92777ae52bc89c40be463f0e38855d15ec005de1dcd9f8f809bd5d1f7423c524"} Oct 31 01:19:37.781759 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:37.781695 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-44d22c-9c679cbbf-rmpq6" Oct 31 01:19:37.899882 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:37.899827 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-44d22c-649bc5675f-zvc4z" podStartSLOduration=1.89981123 podStartE2EDuration="1.89981123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:19:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:19:37.898129719 +0000 UTC m=+3870.369299468" watchObservedRunningTime="2026-10-31 01:19:37.89981123 +0000 UTC m=+3870.370980979" Oct 31 01:19:38.025793 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:38.025725 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-authcfg-http-44d22c-9c679cbbf-rmpq6" podStartSLOduration=2.025703643 podStartE2EDuration="2.025703643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:19:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:19:38.02479158 +0000 UTC m=+3870.495961332" watchObservedRunningTime="2026-10-31 01:19:38.025703643 +0000 UTC m=+3870.496873394" Oct 31 01:19:47.599597 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:47.599521 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-79tk8"] Oct 31 01:19:48.789535 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:48.789506 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-44d22c-9c679cbbf-rmpq6" Oct 31 01:19:48.789926 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:48.789605 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-44d22c-649bc5675f-zvc4z" Oct 31 01:19:56.907025 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:56.906987 5222 generic.go:356] "Generic (PLEG): container finished" podID="9b01d020-5db4-48ea-945d-af5f2e4a5347" containerID="9ef7909f6e9ee77e937cb9a42b7a13a0e9a8651e89117b143e44d7120a387909" exitCode=0 Oct 31 01:19:56.907508 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:56.907075 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-44d22c-openshift-default-6b997dc7fc-dgpzm" event={"ID":"9b01d020-5db4-48ea-945d-af5f2e4a5347","Type":"ContainerDied","Data":"9ef7909f6e9ee77e937cb9a42b7a13a0e9a8651e89117b143e44d7120a387909"} Oct 31 01:19:56.907602 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:56.907572 5222 scope.go:122] "RemoveContainer" containerID="9ef7909f6e9ee77e937cb9a42b7a13a0e9a8651e89117b143e44d7120a387909" Oct 31 01:19:56.909077 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:56.909042 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 01:19:56.909169 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:56.909135 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 01:19:56.909236 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:56.909179 5222 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 01:19:57.913623 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:57.913588 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-44d22c-openshift-default-6b997dc7fc-dgpzm" event={"ID":"9b01d020-5db4-48ea-945d-af5f2e4a5347","Type":"ContainerStarted","Data":"e6185012b2e8d95355a3814f10da5f5c07a101ae688735a22d97715b4d3918b2"} Oct 31 01:19:58.737004 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:58.736970 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-44d22c-openshift-default-6b997dc7fc-dgpzm" Oct 31 01:19:58.743397 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:58.738398 5222 patch_prober.go:29] interesting pod/e2e-authcfg-http-44d22c-openshift-default-6b997dc7fc-dgpzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.20:15021/healthz/ready\": dial tcp 10.134.1.20:15021: connect: connection refused" start-of-body= Oct 31 01:19:58.743397 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:58.738458 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-44d22c-openshift-default-6b997dc7fc-dgpzm" podUID="9b01d020-5db4-48ea-945d-af5f2e4a5347" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.20:15021/healthz/ready\": dial tcp 10.134.1.20:15021: connect: connection refused" Oct 31 01:19:59.737091 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:59.737050 5222 patch_prober.go:29] interesting pod/e2e-authcfg-http-44d22c-openshift-default-6b997dc7fc-dgpzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.20:15021/healthz/ready\": dial tcp 10.134.1.20:15021: connect: connection refused" start-of-body= Oct 31 01:19:59.737591 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:19:59.737110 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-44d22c-openshift-default-6b997dc7fc-dgpzm" podUID="9b01d020-5db4-48ea-945d-af5f2e4a5347" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.20:15021/healthz/ready\": dial tcp 10.134.1.20:15021: connect: connection refused" Oct 31 01:20:00.742405 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:20:00.737478 5222 patch_prober.go:29] interesting pod/e2e-authcfg-http-44d22c-openshift-default-6b997dc7fc-dgpzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.20:15021/healthz/ready\": dial tcp 10.134.1.20:15021: connect: connection refused" start-of-body= Oct 31 01:20:00.742405 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:20:00.737565 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-44d22c-openshift-default-6b997dc7fc-dgpzm" podUID="9b01d020-5db4-48ea-945d-af5f2e4a5347" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.20:15021/healthz/ready\": dial tcp 10.134.1.20:15021: connect: connection refused" Oct 31 01:20:01.758397 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:20:01.758334 5222 patch_prober.go:29] interesting pod/e2e-authcfg-http-44d22c-openshift-default-6b997dc7fc-dgpzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.20:15021/healthz/ready\": dial tcp 10.134.1.20:15021: connect: connection refused" start-of-body= Oct 31 01:20:01.759110 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:20:01.759079 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-44d22c-openshift-default-6b997dc7fc-dgpzm" podUID="9b01d020-5db4-48ea-945d-af5f2e4a5347" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.20:15021/healthz/ready\": dial tcp 10.134.1.20:15021: connect: connection refused" Oct 31 01:20:01.762506 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:20:01.759004 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-44d22c-openshift-default-6b997dc7fc-dgpzm" Oct 31 01:20:02.737475 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:20:02.737437 5222 patch_prober.go:29] interesting pod/e2e-authcfg-http-44d22c-openshift-default-6b997dc7fc-dgpzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.20:15021/healthz/ready\": dial tcp 10.134.1.20:15021: connect: connection refused" start-of-body= Oct 31 01:20:02.737796 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:20:02.737766 5222 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-44d22c-openshift-default-6b997dc7fc-dgpzm" podUID="9b01d020-5db4-48ea-945d-af5f2e4a5347" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.20:15021/healthz/ready\": dial tcp 10.134.1.20:15021: connect: connection refused" Oct 31 01:20:03.741221 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:20:03.741191 5222 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-44d22c-openshift-default-6b997dc7fc-dgpzm" Oct 31 01:20:03.742363 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:20:03.742343 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-44d22c-openshift-default-6b997dc7fc-dgpzm" Oct 31 01:20:05.889854 ip-10-0-137-56 kubenswrapper[5222]: E1031 01:20:05.889815 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 01:20:08.340008 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:20:08.339981 5222 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h9hsf_0f8c3c03-a1fc-4376-98c6-feeeefa08068/ovn-acl-logging/0.log" Oct 31 01:20:08.369335 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:20:08.369312 5222 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h9hsf_0f8c3c03-a1fc-4376-98c6-feeeefa08068/ovn-acl-logging/0.log" Oct 31 01:20:11.077780 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:20:11.077747 5222 scope.go:122] "RemoveContainer" containerID="f18f9c653b78fe5a3e7c7ed97b783c260e4986510fceb0f2fcdc09d1e78cada1" Oct 31 01:20:35.889884 ip-10-0-137-56 kubenswrapper[5222]: E1031 01:20:35.889842 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 01:21:05.890459 ip-10-0-137-56 kubenswrapper[5222]: E1031 01:21:05.890412 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 01:21:18.352399 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:21:18.352295 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-44d22c-6r2lr"] Oct 31 01:21:18.355773 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:21:18.355754 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-44d22c-6r2lr" Oct 31 01:21:18.399636 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:21:18.399592 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba160820-dd33-4fb9-bb20-0986a552f450-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-44d22c-6r2lr\" (UID: \"ba160820-dd33-4fb9-bb20-0986a552f450\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-44d22c-6r2lr" Oct 31 01:21:18.399853 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:21:18.399752 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba160820-dd33-4fb9-bb20-0986a552f450-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-44d22c-6r2lr\" (UID: \"ba160820-dd33-4fb9-bb20-0986a552f450\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-44d22c-6r2lr" Oct 31 01:21:18.399853 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:21:18.399791 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvnvh\" (UniqueName: \"kubernetes.io/projected/ba160820-dd33-4fb9-bb20-0986a552f450-kube-api-access-pvnvh\") pod \"maas-api-revoke-keys-e2e-authcfg-http-44d22c-6r2lr\" (UID: \"ba160820-dd33-4fb9-bb20-0986a552f450\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-44d22c-6r2lr" Oct 31 01:21:18.418067 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:21:18.418034 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-44d22c-6r2lr"] Oct 31 01:21:18.465667 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:21:18.465630 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:21:18.465667 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:21:18.465651 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-29cjc\"" Oct 31 01:21:18.465667 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:21:18.465629 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:21:18.465667 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:21:18.465639 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:21:18.500956 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:21:18.500912 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba160820-dd33-4fb9-bb20-0986a552f450-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-44d22c-6r2lr\" (UID: \"ba160820-dd33-4fb9-bb20-0986a552f450\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-44d22c-6r2lr" Oct 31 01:21:18.501181 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:21:18.500969 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba160820-dd33-4fb9-bb20-0986a552f450-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-44d22c-6r2lr\" (UID: \"ba160820-dd33-4fb9-bb20-0986a552f450\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-44d22c-6r2lr" Oct 31 01:21:18.501181 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:21:18.501084 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvnvh\" (UniqueName: \"kubernetes.io/projected/ba160820-dd33-4fb9-bb20-0986a552f450-kube-api-access-pvnvh\") pod \"maas-api-revoke-keys-e2e-authcfg-http-44d22c-6r2lr\" (UID: \"ba160820-dd33-4fb9-bb20-0986a552f450\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-44d22c-6r2lr" Oct 31 01:21:18.501686 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:21:18.501663 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba160820-dd33-4fb9-bb20-0986a552f450-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-44d22c-6r2lr\" (UID: \"ba160820-dd33-4fb9-bb20-0986a552f450\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-44d22c-6r2lr" Oct 31 01:21:18.503623 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:21:18.503600 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba160820-dd33-4fb9-bb20-0986a552f450-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-44d22c-6r2lr\" (UID: \"ba160820-dd33-4fb9-bb20-0986a552f450\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-44d22c-6r2lr" Oct 31 01:21:18.563991 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:21:18.563949 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvnvh\" (UniqueName: \"kubernetes.io/projected/ba160820-dd33-4fb9-bb20-0986a552f450-kube-api-access-pvnvh\") pod \"maas-api-revoke-keys-e2e-authcfg-http-44d22c-6r2lr\" (UID: \"ba160820-dd33-4fb9-bb20-0986a552f450\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-44d22c-6r2lr" Oct 31 01:21:18.668633 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:21:18.668532 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-44d22c-6r2lr" Oct 31 01:21:19.039436 ip-10-0-137-56 kubenswrapper[5222]: W1031 01:21:19.039391 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba160820_dd33_4fb9_bb20_0986a552f450.slice/crio-ca5d66a06b951f8011824960f4a40f21eae50f576fe023108fa3221a8c927e20 WatchSource:0}: Error finding container ca5d66a06b951f8011824960f4a40f21eae50f576fe023108fa3221a8c927e20: Status 404 returned error can't find the container with id ca5d66a06b951f8011824960f4a40f21eae50f576fe023108fa3221a8c927e20 Oct 31 01:21:19.041252 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:21:19.041228 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-44d22c-6r2lr"] Oct 31 01:21:19.312779 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:21:19.312689 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-44d22c-6r2lr" event={"ID":"ba160820-dd33-4fb9-bb20-0986a552f450","Type":"ContainerStarted","Data":"ca5d66a06b951f8011824960f4a40f21eae50f576fe023108fa3221a8c927e20"} Oct 31 01:21:20.319737 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:21:20.319699 5222 generic.go:356] "Generic (PLEG): container finished" podID="ba160820-dd33-4fb9-bb20-0986a552f450" containerID="bdd671610c1435d9db0bf70be29c4a6ed5144a261dba8e480286085f74d69484" exitCode=0 Oct 31 01:21:20.319737 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:21:20.319740 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-44d22c-6r2lr" event={"ID":"ba160820-dd33-4fb9-bb20-0986a552f450","Type":"ContainerDied","Data":"bdd671610c1435d9db0bf70be29c4a6ed5144a261dba8e480286085f74d69484"} Oct 31 01:21:21.463847 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:21:21.463819 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-44d22c-6r2lr" Oct 31 01:21:21.528107 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:21:21.528067 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba160820-dd33-4fb9-bb20-0986a552f450-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba160820-dd33-4fb9-bb20-0986a552f450-maas-api-openshift-ca\") pod \"ba160820-dd33-4fb9-bb20-0986a552f450\" (UID: \"ba160820-dd33-4fb9-bb20-0986a552f450\") " Oct 31 01:21:21.528107 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:21:21.528113 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba160820-dd33-4fb9-bb20-0986a552f450-kube-api-access-pvnvh\" (UniqueName: \"kubernetes.io/projected/ba160820-dd33-4fb9-bb20-0986a552f450-kube-api-access-pvnvh\") pod \"ba160820-dd33-4fb9-bb20-0986a552f450\" (UID: \"ba160820-dd33-4fb9-bb20-0986a552f450\") " Oct 31 01:21:21.528401 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:21:21.528252 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba160820-dd33-4fb9-bb20-0986a552f450-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba160820-dd33-4fb9-bb20-0986a552f450-maas-api-xks-ca\") pod \"ba160820-dd33-4fb9-bb20-0986a552f450\" (UID: \"ba160820-dd33-4fb9-bb20-0986a552f450\") " Oct 31 01:21:21.528501 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:21:21.528479 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba160820-dd33-4fb9-bb20-0986a552f450-maas-api-openshift-ca" pod "ba160820-dd33-4fb9-bb20-0986a552f450" (UID: "ba160820-dd33-4fb9-bb20-0986a552f450"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:21:21.530557 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:21:21.530530 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba160820-dd33-4fb9-bb20-0986a552f450-maas-api-xks-ca" pod "ba160820-dd33-4fb9-bb20-0986a552f450" (UID: "ba160820-dd33-4fb9-bb20-0986a552f450"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:21:21.530666 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:21:21.530621 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba160820-dd33-4fb9-bb20-0986a552f450-kube-api-access-pvnvh" pod "ba160820-dd33-4fb9-bb20-0986a552f450" (UID: "ba160820-dd33-4fb9-bb20-0986a552f450"). InnerVolumeSpecName "kube-api-access-pvnvh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:21:21.629357 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:21:21.629271 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba160820-dd33-4fb9-bb20-0986a552f450-maas-api-xks-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:21:21.629357 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:21:21.629306 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba160820-dd33-4fb9-bb20-0986a552f450-maas-api-openshift-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:21:21.629357 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:21:21.629316 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvnvh\" (UniqueName: \"kubernetes.io/projected/ba160820-dd33-4fb9-bb20-0986a552f450-kube-api-access-pvnvh\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:21:22.332584 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:21:22.332551 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-44d22c-6r2lr" Oct 31 01:21:22.332798 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:21:22.332588 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-44d22c-6r2lr" event={"ID":"ba160820-dd33-4fb9-bb20-0986a552f450","Type":"ContainerDied","Data":"ca5d66a06b951f8011824960f4a40f21eae50f576fe023108fa3221a8c927e20"} Oct 31 01:21:22.332798 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:21:22.332638 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca5d66a06b951f8011824960f4a40f21eae50f576fe023108fa3221a8c927e20" Oct 31 01:21:29.823687 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:21:29.823653 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-44d22c-9c679cbbf-rmpq6"] Oct 31 01:21:29.932236 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:21:29.932200 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-44d22c-649bc5675f-zvc4z"] Oct 31 01:21:34.824712 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:21:34.824640 5222 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-authcfg-http-44d22c-9c679cbbf-rmpq6" podUID="a2cbe681-d873-40f2-b768-32bfff6d2acf" containerName="payload-processing" containerID="cri-o://18f1029bf135bc31a25b4fddca48c3ce6f6774a9c6da6b877c7d3c0b80e407d2" gracePeriod=25 Oct 31 01:21:34.933145 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:21:34.933097 5222 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-44d22c-649bc5675f-zvc4z" podUID="0adf21e2-d5cd-4c6e-9b6a-567bc7880f36" containerName="payload-pre-processing" containerID="cri-o://a1d156b2fdf3ef3b1d2598264a577b5adf8bdfa85f946dac8549633e866b20fc" gracePeriod=25 Oct 31 01:21:35.085634 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:21:35.085564 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-44d22c-9c679cbbf-rmpq6" Oct 31 01:21:35.177853 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:21:35.177818 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2cbe681-d873-40f2-b768-32bfff6d2acf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2cbe681-d873-40f2-b768-32bfff6d2acf-plugins-config-volume\") pod \"a2cbe681-d873-40f2-b768-32bfff6d2acf\" (UID: \"a2cbe681-d873-40f2-b768-32bfff6d2acf\") " Oct 31 01:21:35.178045 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:21:35.177910 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2cbe681-d873-40f2-b768-32bfff6d2acf-kube-api-access-vm79z\" (UniqueName: \"kubernetes.io/projected/a2cbe681-d873-40f2-b768-32bfff6d2acf-kube-api-access-vm79z\") pod \"a2cbe681-d873-40f2-b768-32bfff6d2acf\" (UID: \"a2cbe681-d873-40f2-b768-32bfff6d2acf\") " Oct 31 01:21:35.178243 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:21:35.178212 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2cbe681-d873-40f2-b768-32bfff6d2acf-plugins-config-volume" pod "a2cbe681-d873-40f2-b768-32bfff6d2acf" (UID: "a2cbe681-d873-40f2-b768-32bfff6d2acf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:21:35.180738 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:21:35.180712 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2cbe681-d873-40f2-b768-32bfff6d2acf-kube-api-access-vm79z" pod "a2cbe681-d873-40f2-b768-32bfff6d2acf" (UID: "a2cbe681-d873-40f2-b768-32bfff6d2acf"). InnerVolumeSpecName "kube-api-access-vm79z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:21:35.197199 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:21:35.197169 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-44d22c-649bc5675f-zvc4z" Oct 31 01:21:35.279278 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:21:35.279246 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0adf21e2-d5cd-4c6e-9b6a-567bc7880f36-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0adf21e2-d5cd-4c6e-9b6a-567bc7880f36-plugins-config-volume\") pod \"0adf21e2-d5cd-4c6e-9b6a-567bc7880f36\" (UID: \"0adf21e2-d5cd-4c6e-9b6a-567bc7880f36\") " Oct 31 01:21:35.279503 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:21:35.279400 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0adf21e2-d5cd-4c6e-9b6a-567bc7880f36-kube-api-access-2qqkb\" (UniqueName: \"kubernetes.io/projected/0adf21e2-d5cd-4c6e-9b6a-567bc7880f36-kube-api-access-2qqkb\") pod \"0adf21e2-d5cd-4c6e-9b6a-567bc7880f36\" (UID: \"0adf21e2-d5cd-4c6e-9b6a-567bc7880f36\") " Oct 31 01:21:35.279633 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:21:35.279621 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vm79z\" (UniqueName: \"kubernetes.io/projected/a2cbe681-d873-40f2-b768-32bfff6d2acf-kube-api-access-vm79z\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:21:35.279680 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:21:35.279635 5222 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2cbe681-d873-40f2-b768-32bfff6d2acf-plugins-config-volume\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:21:35.279680 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:21:35.279660 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0adf21e2-d5cd-4c6e-9b6a-567bc7880f36-plugins-config-volume" pod "0adf21e2-d5cd-4c6e-9b6a-567bc7880f36" (UID: "0adf21e2-d5cd-4c6e-9b6a-567bc7880f36"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:21:35.281659 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:21:35.281638 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0adf21e2-d5cd-4c6e-9b6a-567bc7880f36-kube-api-access-2qqkb" pod "0adf21e2-d5cd-4c6e-9b6a-567bc7880f36" (UID: "0adf21e2-d5cd-4c6e-9b6a-567bc7880f36"). InnerVolumeSpecName "kube-api-access-2qqkb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:21:35.380496 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:21:35.380352 5222 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0adf21e2-d5cd-4c6e-9b6a-567bc7880f36-plugins-config-volume\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:21:35.380496 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:21:35.380439 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qqkb\" (UniqueName: \"kubernetes.io/projected/0adf21e2-d5cd-4c6e-9b6a-567bc7880f36-kube-api-access-2qqkb\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:21:35.421517 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:21:35.421486 5222 generic.go:356] "Generic (PLEG): container finished" podID="0adf21e2-d5cd-4c6e-9b6a-567bc7880f36" containerID="a1d156b2fdf3ef3b1d2598264a577b5adf8bdfa85f946dac8549633e866b20fc" exitCode=0 Oct 31 01:21:35.421705 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:21:35.421536 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-44d22c-649bc5675f-zvc4z" Oct 31 01:21:35.421705 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:21:35.421576 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-44d22c-649bc5675f-zvc4z" event={"ID":"0adf21e2-d5cd-4c6e-9b6a-567bc7880f36","Type":"ContainerDied","Data":"a1d156b2fdf3ef3b1d2598264a577b5adf8bdfa85f946dac8549633e866b20fc"} Oct 31 01:21:35.421705 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:21:35.421634 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-44d22c-649bc5675f-zvc4z" event={"ID":"0adf21e2-d5cd-4c6e-9b6a-567bc7880f36","Type":"ContainerDied","Data":"69214b883949a4e2204ca759b0257148d0a75be0b2ef868afa39f9e903ebc8f1"} Oct 31 01:21:35.421705 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:21:35.421669 5222 scope.go:122] "RemoveContainer" containerID="a1d156b2fdf3ef3b1d2598264a577b5adf8bdfa85f946dac8549633e866b20fc" Oct 31 01:21:35.423011 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:21:35.422978 5222 generic.go:356] "Generic (PLEG): container finished" podID="a2cbe681-d873-40f2-b768-32bfff6d2acf" containerID="18f1029bf135bc31a25b4fddca48c3ce6f6774a9c6da6b877c7d3c0b80e407d2" exitCode=0 Oct 31 01:21:35.423107 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:21:35.423018 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-44d22c-9c679cbbf-rmpq6" Oct 31 01:21:35.423156 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:21:35.423030 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-44d22c-9c679cbbf-rmpq6" event={"ID":"a2cbe681-d873-40f2-b768-32bfff6d2acf","Type":"ContainerDied","Data":"18f1029bf135bc31a25b4fddca48c3ce6f6774a9c6da6b877c7d3c0b80e407d2"} Oct 31 01:21:35.423211 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:21:35.423159 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-44d22c-9c679cbbf-rmpq6" event={"ID":"a2cbe681-d873-40f2-b768-32bfff6d2acf","Type":"ContainerDied","Data":"92777ae52bc89c40be463f0e38855d15ec005de1dcd9f8f809bd5d1f7423c524"} Oct 31 01:21:35.431557 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:21:35.431534 5222 scope.go:122] "RemoveContainer" containerID="a1d156b2fdf3ef3b1d2598264a577b5adf8bdfa85f946dac8549633e866b20fc" Oct 31 01:21:35.431899 ip-10-0-137-56 kubenswrapper[5222]: E1031 01:21:35.431872 5222 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1d156b2fdf3ef3b1d2598264a577b5adf8bdfa85f946dac8549633e866b20fc\": container with ID starting with a1d156b2fdf3ef3b1d2598264a577b5adf8bdfa85f946dac8549633e866b20fc not found: ID does not exist" containerID="a1d156b2fdf3ef3b1d2598264a577b5adf8bdfa85f946dac8549633e866b20fc" Oct 31 01:21:35.432009 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:21:35.431912 5222 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1d156b2fdf3ef3b1d2598264a577b5adf8bdfa85f946dac8549633e866b20fc"} err="failed to get container status \"a1d156b2fdf3ef3b1d2598264a577b5adf8bdfa85f946dac8549633e866b20fc\": rpc error: code = NotFound desc = could not find container \"a1d156b2fdf3ef3b1d2598264a577b5adf8bdfa85f946dac8549633e866b20fc\": container with ID starting with a1d156b2fdf3ef3b1d2598264a577b5adf8bdfa85f946dac8549633e866b20fc not found: ID does not exist" Oct 31 01:21:35.432009 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:21:35.431939 5222 scope.go:122] "RemoveContainer" containerID="18f1029bf135bc31a25b4fddca48c3ce6f6774a9c6da6b877c7d3c0b80e407d2" Oct 31 01:21:35.440925 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:21:35.440904 5222 scope.go:122] "RemoveContainer" containerID="18f1029bf135bc31a25b4fddca48c3ce6f6774a9c6da6b877c7d3c0b80e407d2" Oct 31 01:21:35.441207 ip-10-0-137-56 kubenswrapper[5222]: E1031 01:21:35.441184 5222 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18f1029bf135bc31a25b4fddca48c3ce6f6774a9c6da6b877c7d3c0b80e407d2\": container with ID starting with 18f1029bf135bc31a25b4fddca48c3ce6f6774a9c6da6b877c7d3c0b80e407d2 not found: ID does not exist" containerID="18f1029bf135bc31a25b4fddca48c3ce6f6774a9c6da6b877c7d3c0b80e407d2" Oct 31 01:21:35.441297 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:21:35.441214 5222 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18f1029bf135bc31a25b4fddca48c3ce6f6774a9c6da6b877c7d3c0b80e407d2"} err="failed to get container status \"18f1029bf135bc31a25b4fddca48c3ce6f6774a9c6da6b877c7d3c0b80e407d2\": rpc error: code = NotFound desc = could not find container \"18f1029bf135bc31a25b4fddca48c3ce6f6774a9c6da6b877c7d3c0b80e407d2\": container with ID starting with 18f1029bf135bc31a25b4fddca48c3ce6f6774a9c6da6b877c7d3c0b80e407d2 not found: ID does not exist" Oct 31 01:21:35.589954 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:21:35.589914 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-44d22c-9c679cbbf-rmpq6"] Oct 31 01:21:35.598896 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:21:35.598862 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-44d22c-9c679cbbf-rmpq6"] Oct 31 01:21:35.654286 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:21:35.654208 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2cbe681-d873-40f2-b768-32bfff6d2acf" path="/var/lib/kubelet/pods/a2cbe681-d873-40f2-b768-32bfff6d2acf/volumes" Oct 31 01:21:35.711499 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:21:35.711465 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-44d22c-649bc5675f-zvc4z"] Oct 31 01:21:35.715205 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:21:35.715174 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-44d22c-649bc5675f-zvc4z"] Oct 31 01:21:35.889788 ip-10-0-137-56 kubenswrapper[5222]: E1031 01:21:35.889754 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 01:21:37.655031 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:21:37.654989 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0adf21e2-d5cd-4c6e-9b6a-567bc7880f36" path="/var/lib/kubelet/pods/0adf21e2-d5cd-4c6e-9b6a-567bc7880f36/volumes" Oct 31 01:21:39.420483 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:21:39.420446 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-44d22c-6r2lr"] Oct 31 01:21:39.471676 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:21:39.471636 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-44d22c-6r2lr"] Oct 31 01:21:39.654275 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:21:39.654242 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba160820-dd33-4fb9-bb20-0986a552f450" path="/var/lib/kubelet/pods/ba160820-dd33-4fb9-bb20-0986a552f450/volumes" Oct 31 01:21:42.792416 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:21:42.792364 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-79tk8"] Oct 31 01:21:46.220535 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:21:46.220503 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-44d22c-openshift-default-6b997dc7fc-dgpzm"] Oct 31 01:21:46.220945 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:21:46.220721 5222 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-authcfg-http-44d22c-openshift-default-6b997dc7fc-dgpzm" podUID="9b01d020-5db4-48ea-945d-af5f2e4a5347" containerName="istio-proxy" containerID="cri-o://e6185012b2e8d95355a3814f10da5f5c07a101ae688735a22d97715b4d3918b2" gracePeriod=30 Oct 31 01:21:48.744728 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:21:48.744686 5222 patch_prober.go:29] interesting pod/e2e-authcfg-http-44d22c-openshift-default-6b997dc7fc-dgpzm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 01:21:48.745136 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:21:48.744763 5222 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-authcfg-http-44d22c-openshift-default-6b997dc7fc-dgpzm" podUID="9b01d020-5db4-48ea-945d-af5f2e4a5347" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 01:21:51.478019 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:21:51.477989 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-44d22c-openshift-default-6b997dc7fc-dgpzm" Oct 31 01:21:51.503198 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:21:51.503108 5222 generic.go:356] "Generic (PLEG): container finished" podID="9b01d020-5db4-48ea-945d-af5f2e4a5347" containerID="e6185012b2e8d95355a3814f10da5f5c07a101ae688735a22d97715b4d3918b2" exitCode=0 Oct 31 01:21:51.503198 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:21:51.503157 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-44d22c-openshift-default-6b997dc7fc-dgpzm" Oct 31 01:21:51.503198 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:21:51.503168 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-44d22c-openshift-default-6b997dc7fc-dgpzm" event={"ID":"9b01d020-5db4-48ea-945d-af5f2e4a5347","Type":"ContainerDied","Data":"e6185012b2e8d95355a3814f10da5f5c07a101ae688735a22d97715b4d3918b2"} Oct 31 01:21:51.503512 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:21:51.503245 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-44d22c-openshift-default-6b997dc7fc-dgpzm" event={"ID":"9b01d020-5db4-48ea-945d-af5f2e4a5347","Type":"ContainerDied","Data":"59d5b75bf268a77b3bdbd9937894832c5f9d73559727195ac3d6f09d2eee64ba"} Oct 31 01:21:51.503512 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:21:51.503267 5222 scope.go:122] "RemoveContainer" containerID="e6185012b2e8d95355a3814f10da5f5c07a101ae688735a22d97715b4d3918b2" Oct 31 01:21:51.511999 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:21:51.511976 5222 scope.go:122] "RemoveContainer" containerID="9ef7909f6e9ee77e937cb9a42b7a13a0e9a8651e89117b143e44d7120a387909" Oct 31 01:21:51.520274 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:21:51.520253 5222 scope.go:122] "RemoveContainer" containerID="e6185012b2e8d95355a3814f10da5f5c07a101ae688735a22d97715b4d3918b2" Oct 31 01:21:51.520603 ip-10-0-137-56 kubenswrapper[5222]: E1031 01:21:51.520581 5222 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6185012b2e8d95355a3814f10da5f5c07a101ae688735a22d97715b4d3918b2\": container with ID starting with e6185012b2e8d95355a3814f10da5f5c07a101ae688735a22d97715b4d3918b2 not found: ID does not exist" containerID="e6185012b2e8d95355a3814f10da5f5c07a101ae688735a22d97715b4d3918b2" Oct 31 01:21:51.520673 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:21:51.520618 5222 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6185012b2e8d95355a3814f10da5f5c07a101ae688735a22d97715b4d3918b2"} err="failed to get container status \"e6185012b2e8d95355a3814f10da5f5c07a101ae688735a22d97715b4d3918b2\": rpc error: code = NotFound desc = could not find container \"e6185012b2e8d95355a3814f10da5f5c07a101ae688735a22d97715b4d3918b2\": container with ID starting with e6185012b2e8d95355a3814f10da5f5c07a101ae688735a22d97715b4d3918b2 not found: ID does not exist" Oct 31 01:21:51.520673 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:21:51.520637 5222 scope.go:122] "RemoveContainer" containerID="9ef7909f6e9ee77e937cb9a42b7a13a0e9a8651e89117b143e44d7120a387909" Oct 31 01:21:51.520932 ip-10-0-137-56 kubenswrapper[5222]: E1031 01:21:51.520909 5222 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ef7909f6e9ee77e937cb9a42b7a13a0e9a8651e89117b143e44d7120a387909\": container with ID starting with 9ef7909f6e9ee77e937cb9a42b7a13a0e9a8651e89117b143e44d7120a387909 not found: ID does not exist" containerID="9ef7909f6e9ee77e937cb9a42b7a13a0e9a8651e89117b143e44d7120a387909" Oct 31 01:21:51.520980 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:21:51.520938 5222 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ef7909f6e9ee77e937cb9a42b7a13a0e9a8651e89117b143e44d7120a387909"} err="failed to get container status \"9ef7909f6e9ee77e937cb9a42b7a13a0e9a8651e89117b143e44d7120a387909\": rpc error: code = NotFound desc = could not find container \"9ef7909f6e9ee77e937cb9a42b7a13a0e9a8651e89117b143e44d7120a387909\": container with ID starting with 9ef7909f6e9ee77e937cb9a42b7a13a0e9a8651e89117b143e44d7120a387909 not found: ID does not exist" Oct 31 01:21:51.545824 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:21:51.545780 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b01d020-5db4-48ea-945d-af5f2e4a5347-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b01d020-5db4-48ea-945d-af5f2e4a5347-istio-data\") pod \"9b01d020-5db4-48ea-945d-af5f2e4a5347\" (UID: \"9b01d020-5db4-48ea-945d-af5f2e4a5347\") " Oct 31 01:21:51.546022 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:21:51.545842 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b01d020-5db4-48ea-945d-af5f2e4a5347-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b01d020-5db4-48ea-945d-af5f2e4a5347-workload-certs\") pod \"9b01d020-5db4-48ea-945d-af5f2e4a5347\" (UID: \"9b01d020-5db4-48ea-945d-af5f2e4a5347\") " Oct 31 01:21:51.546022 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:21:51.545867 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b01d020-5db4-48ea-945d-af5f2e4a5347-istio-token\" (UniqueName: \"kubernetes.io/projected/9b01d020-5db4-48ea-945d-af5f2e4a5347-istio-token\") pod \"9b01d020-5db4-48ea-945d-af5f2e4a5347\" (UID: \"9b01d020-5db4-48ea-945d-af5f2e4a5347\") " Oct 31 01:21:51.546022 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:21:51.545915 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b01d020-5db4-48ea-945d-af5f2e4a5347-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b01d020-5db4-48ea-945d-af5f2e4a5347-istio-envoy\") pod \"9b01d020-5db4-48ea-945d-af5f2e4a5347\" (UID: \"9b01d020-5db4-48ea-945d-af5f2e4a5347\") " Oct 31 01:21:51.546022 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:21:51.545980 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b01d020-5db4-48ea-945d-af5f2e4a5347-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b01d020-5db4-48ea-945d-af5f2e4a5347-istiod-ca-cert\") pod \"9b01d020-5db4-48ea-945d-af5f2e4a5347\" (UID: \"9b01d020-5db4-48ea-945d-af5f2e4a5347\") " Oct 31 01:21:51.546022 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:21:51.546006 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b01d020-5db4-48ea-945d-af5f2e4a5347-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b01d020-5db4-48ea-945d-af5f2e4a5347-credential-socket\") pod \"9b01d020-5db4-48ea-945d-af5f2e4a5347\" (UID: \"9b01d020-5db4-48ea-945d-af5f2e4a5347\") " Oct 31 01:21:51.546305 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:21:51.546086 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b01d020-5db4-48ea-945d-af5f2e4a5347-workload-certs" pod "9b01d020-5db4-48ea-945d-af5f2e4a5347" (UID: "9b01d020-5db4-48ea-945d-af5f2e4a5347"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:21:51.546305 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:21:51.546092 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9b01d020-5db4-48ea-945d-af5f2e4a5347-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b01d020-5db4-48ea-945d-af5f2e4a5347-istio-podinfo\") pod \"9b01d020-5db4-48ea-945d-af5f2e4a5347\" (UID: \"9b01d020-5db4-48ea-945d-af5f2e4a5347\") " Oct 31 01:21:51.546305 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:21:51.546102 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b01d020-5db4-48ea-945d-af5f2e4a5347-istio-data" pod "9b01d020-5db4-48ea-945d-af5f2e4a5347" (UID: "9b01d020-5db4-48ea-945d-af5f2e4a5347"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:21:51.546305 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:21:51.546166 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b01d020-5db4-48ea-945d-af5f2e4a5347-kube-api-access-gdbwv\" (UniqueName: \"kubernetes.io/projected/9b01d020-5db4-48ea-945d-af5f2e4a5347-kube-api-access-gdbwv\") pod \"9b01d020-5db4-48ea-945d-af5f2e4a5347\" (UID: \"9b01d020-5db4-48ea-945d-af5f2e4a5347\") " Oct 31 01:21:51.546305 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:21:51.546256 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b01d020-5db4-48ea-945d-af5f2e4a5347-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b01d020-5db4-48ea-945d-af5f2e4a5347-workload-socket\") pod \"9b01d020-5db4-48ea-945d-af5f2e4a5347\" (UID: \"9b01d020-5db4-48ea-945d-af5f2e4a5347\") " Oct 31 01:21:51.546650 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:21:51.546359 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b01d020-5db4-48ea-945d-af5f2e4a5347-credential-socket" pod "9b01d020-5db4-48ea-945d-af5f2e4a5347" (UID: "9b01d020-5db4-48ea-945d-af5f2e4a5347"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:21:51.546650 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:21:51.546416 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b01d020-5db4-48ea-945d-af5f2e4a5347-istiod-ca-cert" pod "9b01d020-5db4-48ea-945d-af5f2e4a5347" (UID: "9b01d020-5db4-48ea-945d-af5f2e4a5347"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:21:51.546650 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:21:51.546571 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b01d020-5db4-48ea-945d-af5f2e4a5347-workload-socket" pod "9b01d020-5db4-48ea-945d-af5f2e4a5347" (UID: "9b01d020-5db4-48ea-945d-af5f2e4a5347"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:21:51.546997 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:21:51.546789 5222 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b01d020-5db4-48ea-945d-af5f2e4a5347-workload-certs\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:21:51.546997 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:21:51.546873 5222 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b01d020-5db4-48ea-945d-af5f2e4a5347-istiod-ca-cert\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:21:51.546997 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:21:51.546932 5222 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b01d020-5db4-48ea-945d-af5f2e4a5347-credential-socket\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:21:51.546997 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:21:51.546946 5222 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b01d020-5db4-48ea-945d-af5f2e4a5347-workload-socket\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:21:51.546997 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:21:51.546960 5222 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b01d020-5db4-48ea-945d-af5f2e4a5347-istio-data\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:21:51.548475 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:21:51.548445 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9b01d020-5db4-48ea-945d-af5f2e4a5347-istio-podinfo" pod "9b01d020-5db4-48ea-945d-af5f2e4a5347" (UID: "9b01d020-5db4-48ea-945d-af5f2e4a5347"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:21:51.548600 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:21:51.548473 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b01d020-5db4-48ea-945d-af5f2e4a5347-istio-token" pod "9b01d020-5db4-48ea-945d-af5f2e4a5347" (UID: "9b01d020-5db4-48ea-945d-af5f2e4a5347"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:21:51.548600 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:21:51.548495 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b01d020-5db4-48ea-945d-af5f2e4a5347-istio-envoy" pod "9b01d020-5db4-48ea-945d-af5f2e4a5347" (UID: "9b01d020-5db4-48ea-945d-af5f2e4a5347"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:21:51.548600 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:21:51.548574 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b01d020-5db4-48ea-945d-af5f2e4a5347-kube-api-access-gdbwv" pod "9b01d020-5db4-48ea-945d-af5f2e4a5347" (UID: "9b01d020-5db4-48ea-945d-af5f2e4a5347"). InnerVolumeSpecName "kube-api-access-gdbwv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:21:51.647718 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:21:51.647679 5222 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b01d020-5db4-48ea-945d-af5f2e4a5347-istio-envoy\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:21:51.647718 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:21:51.647710 5222 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b01d020-5db4-48ea-945d-af5f2e4a5347-istio-podinfo\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:21:51.647718 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:21:51.647720 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdbwv\" (UniqueName: \"kubernetes.io/projected/9b01d020-5db4-48ea-945d-af5f2e4a5347-kube-api-access-gdbwv\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:21:51.648067 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:21:51.647730 5222 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b01d020-5db4-48ea-945d-af5f2e4a5347-istio-token\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:21:51.920580 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:21:51.920543 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-44d22c-openshift-default-6b997dc7fc-dgpzm"] Oct 31 01:21:51.925238 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:21:51.925213 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-authcfg-http-44d22c-openshift-default-6b997dc7fc-dgpzm"] Oct 31 01:21:53.655351 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:21:53.655315 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b01d020-5db4-48ea-945d-af5f2e4a5347" path="/var/lib/kubelet/pods/9b01d020-5db4-48ea-945d-af5f2e4a5347/volumes" Oct 31 01:21:53.919897 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:21:53.919813 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-79tk8"] Oct 31 01:22:05.889474 ip-10-0-137-56 kubenswrapper[5222]: E1031 01:22:05.889436 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 01:22:05.912259 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:05.912223 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-79tk8"] Oct 31 01:22:13.050289 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:13.050250 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-spjbk"] Oct 31 01:22:13.101615 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:13.101576 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-spjbk"] Oct 31 01:22:13.654215 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:13.654175 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76cad17e-7333-45cb-96dd-6a06ad834a06" path="/var/lib/kubelet/pods/76cad17e-7333-45cb-96dd-6a06ad834a06/volumes" Oct 31 01:22:14.203540 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:14.203503 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wrnxz"] Oct 31 01:22:14.203944 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:14.203876 5222 cpu_manager.go:414] "containerMap: removing container" podUID="9b01d020-5db4-48ea-945d-af5f2e4a5347" containerName="istio-proxy" Oct 31 01:22:14.203944 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:14.203888 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b01d020-5db4-48ea-945d-af5f2e4a5347" containerName="istio-proxy" Oct 31 01:22:14.203944 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:14.203906 5222 cpu_manager.go:414] "containerMap: removing container" podUID="a2cbe681-d873-40f2-b768-32bfff6d2acf" containerName="payload-processing" Oct 31 01:22:14.203944 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:14.203911 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2cbe681-d873-40f2-b768-32bfff6d2acf" containerName="payload-processing" Oct 31 01:22:14.203944 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:14.203920 5222 cpu_manager.go:414] "containerMap: removing container" podUID="0adf21e2-d5cd-4c6e-9b6a-567bc7880f36" containerName="payload-pre-processing" Oct 31 01:22:14.203944 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:14.203925 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0adf21e2-d5cd-4c6e-9b6a-567bc7880f36" containerName="payload-pre-processing" Oct 31 01:22:14.203944 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:14.203936 5222 cpu_manager.go:414] "containerMap: removing container" podUID="ba160820-dd33-4fb9-bb20-0986a552f450" containerName="revoke-keys" Oct 31 01:22:14.203944 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:14.203940 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba160820-dd33-4fb9-bb20-0986a552f450" containerName="revoke-keys" Oct 31 01:22:14.204212 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:14.204000 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b01d020-5db4-48ea-945d-af5f2e4a5347" containerName="istio-proxy" Oct 31 01:22:14.204212 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:14.204008 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2cbe681-d873-40f2-b768-32bfff6d2acf" containerName="payload-processing" Oct 31 01:22:14.204212 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:14.204016 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="0adf21e2-d5cd-4c6e-9b6a-567bc7880f36" containerName="payload-pre-processing" Oct 31 01:22:14.204212 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:14.204023 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba160820-dd33-4fb9-bb20-0986a552f450" containerName="revoke-keys" Oct 31 01:22:14.208478 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:14.208458 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wrnxz" Oct 31 01:22:14.261714 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:14.261682 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wrnxz"] Oct 31 01:22:14.311082 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:14.311053 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:22:14.311285 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:14.311226 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:22:14.311285 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:14.311226 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:22:14.311411 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:14.311308 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-29cjc\"" Oct 31 01:22:14.356418 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:14.356355 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99c2a864-cb2f-4169-8852-d443a7d102cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-wrnxz\" (UID: \"99c2a864-cb2f-4169-8852-d443a7d102cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wrnxz" Oct 31 01:22:14.356631 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:14.356553 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qn7gg\" (UniqueName: \"kubernetes.io/projected/99c2a864-cb2f-4169-8852-d443a7d102cf-kube-api-access-qn7gg\") pod \"maas-api-revoke-sub-f03629db-wrnxz\" (UID: \"99c2a864-cb2f-4169-8852-d443a7d102cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wrnxz" Oct 31 01:22:14.356631 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:14.356628 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99c2a864-cb2f-4169-8852-d443a7d102cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-wrnxz\" (UID: \"99c2a864-cb2f-4169-8852-d443a7d102cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wrnxz" Oct 31 01:22:14.457998 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:14.457905 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn7gg\" (UniqueName: \"kubernetes.io/projected/99c2a864-cb2f-4169-8852-d443a7d102cf-kube-api-access-qn7gg\") pod \"maas-api-revoke-sub-f03629db-wrnxz\" (UID: \"99c2a864-cb2f-4169-8852-d443a7d102cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wrnxz" Oct 31 01:22:14.457998 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:14.457990 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99c2a864-cb2f-4169-8852-d443a7d102cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-wrnxz\" (UID: \"99c2a864-cb2f-4169-8852-d443a7d102cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wrnxz" Oct 31 01:22:14.458218 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:14.458039 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99c2a864-cb2f-4169-8852-d443a7d102cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-wrnxz\" (UID: \"99c2a864-cb2f-4169-8852-d443a7d102cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wrnxz" Oct 31 01:22:14.458689 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:14.458666 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99c2a864-cb2f-4169-8852-d443a7d102cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-wrnxz\" (UID: \"99c2a864-cb2f-4169-8852-d443a7d102cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wrnxz" Oct 31 01:22:14.460745 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:14.460711 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99c2a864-cb2f-4169-8852-d443a7d102cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-wrnxz\" (UID: \"99c2a864-cb2f-4169-8852-d443a7d102cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wrnxz" Oct 31 01:22:14.520701 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:14.520667 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qn7gg\" (UniqueName: \"kubernetes.io/projected/99c2a864-cb2f-4169-8852-d443a7d102cf-kube-api-access-qn7gg\") pod \"maas-api-revoke-sub-f03629db-wrnxz\" (UID: \"99c2a864-cb2f-4169-8852-d443a7d102cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wrnxz" Oct 31 01:22:14.521447 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:14.521425 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wrnxz" Oct 31 01:22:14.894753 ip-10-0-137-56 kubenswrapper[5222]: W1031 01:22:14.894721 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99c2a864_cb2f_4169_8852_d443a7d102cf.slice/crio-8683203ddfbb693a2d0ed6cd99b497278b794e12f958fd7f9d05e0a9fbfd8384 WatchSource:0}: Error finding container 8683203ddfbb693a2d0ed6cd99b497278b794e12f958fd7f9d05e0a9fbfd8384: Status 404 returned error can't find the container with id 8683203ddfbb693a2d0ed6cd99b497278b794e12f958fd7f9d05e0a9fbfd8384 Oct 31 01:22:14.895748 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:14.895719 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wrnxz"] Oct 31 01:22:15.685875 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:15.685835 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wrnxz" event={"ID":"99c2a864-cb2f-4169-8852-d443a7d102cf","Type":"ContainerStarted","Data":"7a2b51aab2c6c95b49b773ae515e1df9403dcfdda3b776b6b98c73427c4d9872"} Oct 31 01:22:15.686361 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:15.685897 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wrnxz" event={"ID":"99c2a864-cb2f-4169-8852-d443a7d102cf","Type":"ContainerStarted","Data":"8683203ddfbb693a2d0ed6cd99b497278b794e12f958fd7f9d05e0a9fbfd8384"} Oct 31 01:22:15.806349 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:15.806287 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wrnxz" podStartSLOduration=1.806272402 podStartE2EDuration="1.806272402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:22:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:22:15.80547895 +0000 UTC m=+4028.276648699" watchObservedRunningTime="2026-10-31 01:22:15.806272402 +0000 UTC m=+4028.277442150" Oct 31 01:22:16.692562 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:16.692531 5222 generic.go:356] "Generic (PLEG): container finished" podID="99c2a864-cb2f-4169-8852-d443a7d102cf" containerID="7a2b51aab2c6c95b49b773ae515e1df9403dcfdda3b776b6b98c73427c4d9872" exitCode=0 Oct 31 01:22:16.692955 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:16.692612 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wrnxz" event={"ID":"99c2a864-cb2f-4169-8852-d443a7d102cf","Type":"ContainerDied","Data":"7a2b51aab2c6c95b49b773ae515e1df9403dcfdda3b776b6b98c73427c4d9872"} Oct 31 01:22:17.827499 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:17.827470 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wrnxz" Oct 31 01:22:17.896711 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:17.896679 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99c2a864-cb2f-4169-8852-d443a7d102cf-kube-api-access-qn7gg\" (UniqueName: \"kubernetes.io/projected/99c2a864-cb2f-4169-8852-d443a7d102cf-kube-api-access-qn7gg\") pod \"99c2a864-cb2f-4169-8852-d443a7d102cf\" (UID: \"99c2a864-cb2f-4169-8852-d443a7d102cf\") " Oct 31 01:22:17.896892 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:17.896733 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99c2a864-cb2f-4169-8852-d443a7d102cf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99c2a864-cb2f-4169-8852-d443a7d102cf-maas-api-openshift-ca\") pod \"99c2a864-cb2f-4169-8852-d443a7d102cf\" (UID: \"99c2a864-cb2f-4169-8852-d443a7d102cf\") " Oct 31 01:22:17.896892 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:17.896858 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99c2a864-cb2f-4169-8852-d443a7d102cf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99c2a864-cb2f-4169-8852-d443a7d102cf-maas-api-xks-ca\") pod \"99c2a864-cb2f-4169-8852-d443a7d102cf\" (UID: \"99c2a864-cb2f-4169-8852-d443a7d102cf\") " Oct 31 01:22:17.897104 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:17.897076 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99c2a864-cb2f-4169-8852-d443a7d102cf-maas-api-openshift-ca" pod "99c2a864-cb2f-4169-8852-d443a7d102cf" (UID: "99c2a864-cb2f-4169-8852-d443a7d102cf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:22:17.899006 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:17.898972 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99c2a864-cb2f-4169-8852-d443a7d102cf-maas-api-xks-ca" pod "99c2a864-cb2f-4169-8852-d443a7d102cf" (UID: "99c2a864-cb2f-4169-8852-d443a7d102cf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:22:17.899123 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:17.899031 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99c2a864-cb2f-4169-8852-d443a7d102cf-kube-api-access-qn7gg" pod "99c2a864-cb2f-4169-8852-d443a7d102cf" (UID: "99c2a864-cb2f-4169-8852-d443a7d102cf"). InnerVolumeSpecName "kube-api-access-qn7gg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:22:17.997758 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:17.997669 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99c2a864-cb2f-4169-8852-d443a7d102cf-maas-api-xks-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:22:17.997758 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:17.997700 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qn7gg\" (UniqueName: \"kubernetes.io/projected/99c2a864-cb2f-4169-8852-d443a7d102cf-kube-api-access-qn7gg\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:22:17.997758 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:17.997712 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99c2a864-cb2f-4169-8852-d443a7d102cf-maas-api-openshift-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:22:18.709816 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:18.709777 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wrnxz" event={"ID":"99c2a864-cb2f-4169-8852-d443a7d102cf","Type":"ContainerDied","Data":"8683203ddfbb693a2d0ed6cd99b497278b794e12f958fd7f9d05e0a9fbfd8384"} Oct 31 01:22:18.709816 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:18.709805 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wrnxz" Oct 31 01:22:18.710054 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:18.709856 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8683203ddfbb693a2d0ed6cd99b497278b794e12f958fd7f9d05e0a9fbfd8384" Oct 31 01:22:27.195940 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:27.195907 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-79tk8"] Oct 31 01:22:33.054399 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:33.054313 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-92zr8"] Oct 31 01:22:33.110276 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:33.110238 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-92zr8"] Oct 31 01:22:33.666397 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:33.665578 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="653bf085-9f4e-4e91-adb7-44799c8f400f" path="/var/lib/kubelet/pods/653bf085-9f4e-4e91-adb7-44799c8f400f/volumes" Oct 31 01:22:35.225024 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:35.224988 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7c48k"] Oct 31 01:22:35.225482 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:35.225357 5222 cpu_manager.go:414] "containerMap: removing container" podUID="9b01d020-5db4-48ea-945d-af5f2e4a5347" containerName="istio-proxy" Oct 31 01:22:35.225482 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:35.225382 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b01d020-5db4-48ea-945d-af5f2e4a5347" containerName="istio-proxy" Oct 31 01:22:35.225482 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:35.225404 5222 cpu_manager.go:414] "containerMap: removing container" podUID="99c2a864-cb2f-4169-8852-d443a7d102cf" containerName="revoke-keys" Oct 31 01:22:35.225482 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:35.225410 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99c2a864-cb2f-4169-8852-d443a7d102cf" containerName="revoke-keys" Oct 31 01:22:35.225636 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:35.225487 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b01d020-5db4-48ea-945d-af5f2e4a5347" containerName="istio-proxy" Oct 31 01:22:35.225636 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:35.225500 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="99c2a864-cb2f-4169-8852-d443a7d102cf" containerName="revoke-keys" Oct 31 01:22:35.229859 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:35.229834 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7c48k" Oct 31 01:22:35.283570 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:35.283531 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7c48k"] Oct 31 01:22:35.332298 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:35.332267 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:22:35.332298 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:35.332266 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:22:35.332577 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:35.332268 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:22:35.332577 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:35.332270 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-29cjc\"" Oct 31 01:22:35.333005 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:35.332984 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96672df6-a63f-4313-b703-67469597b7e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-7c48k\" (UID: \"96672df6-a63f-4313-b703-67469597b7e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7c48k" Oct 31 01:22:35.333091 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:35.333053 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjh5d\" (UniqueName: \"kubernetes.io/projected/96672df6-a63f-4313-b703-67469597b7e7-kube-api-access-hjh5d\") pod \"maas-api-revoke-sub-0165d7d2-7c48k\" (UID: \"96672df6-a63f-4313-b703-67469597b7e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7c48k" Oct 31 01:22:35.333158 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:35.333124 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96672df6-a63f-4313-b703-67469597b7e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-7c48k\" (UID: \"96672df6-a63f-4313-b703-67469597b7e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7c48k" Oct 31 01:22:35.434145 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:35.434094 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96672df6-a63f-4313-b703-67469597b7e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-7c48k\" (UID: \"96672df6-a63f-4313-b703-67469597b7e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7c48k" Oct 31 01:22:35.434343 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:35.434228 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96672df6-a63f-4313-b703-67469597b7e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-7c48k\" (UID: \"96672df6-a63f-4313-b703-67469597b7e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7c48k" Oct 31 01:22:35.434343 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:35.434308 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjh5d\" (UniqueName: \"kubernetes.io/projected/96672df6-a63f-4313-b703-67469597b7e7-kube-api-access-hjh5d\") pod \"maas-api-revoke-sub-0165d7d2-7c48k\" (UID: \"96672df6-a63f-4313-b703-67469597b7e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7c48k" Oct 31 01:22:35.434820 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:35.434799 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96672df6-a63f-4313-b703-67469597b7e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-7c48k\" (UID: \"96672df6-a63f-4313-b703-67469597b7e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7c48k" Oct 31 01:22:35.436751 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:35.436727 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96672df6-a63f-4313-b703-67469597b7e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-7c48k\" (UID: \"96672df6-a63f-4313-b703-67469597b7e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7c48k" Oct 31 01:22:35.497106 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:35.497033 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjh5d\" (UniqueName: \"kubernetes.io/projected/96672df6-a63f-4313-b703-67469597b7e7-kube-api-access-hjh5d\") pod \"maas-api-revoke-sub-0165d7d2-7c48k\" (UID: \"96672df6-a63f-4313-b703-67469597b7e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7c48k" Oct 31 01:22:35.543028 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:35.542993 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7c48k" Oct 31 01:22:35.890223 ip-10-0-137-56 kubenswrapper[5222]: E1031 01:22:35.890178 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 01:22:35.916905 ip-10-0-137-56 kubenswrapper[5222]: W1031 01:22:35.916868 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96672df6_a63f_4313_b703_67469597b7e7.slice/crio-e3e47d49d1359f83e2bf095d1bca3f2ff85040a02cb82acb6a8f44fd11bdbb57 WatchSource:0}: Error finding container e3e47d49d1359f83e2bf095d1bca3f2ff85040a02cb82acb6a8f44fd11bdbb57: Status 404 returned error can't find the container with id e3e47d49d1359f83e2bf095d1bca3f2ff85040a02cb82acb6a8f44fd11bdbb57 Oct 31 01:22:35.918464 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:35.918439 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7c48k"] Oct 31 01:22:36.810146 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:36.810109 5222 generic.go:356] "Generic (PLEG): container finished" podID="96672df6-a63f-4313-b703-67469597b7e7" containerID="d459cecd753f1524e761dc060c69ff50a9bd9a77b5666671e4a2236cf76207b2" exitCode=0 Oct 31 01:22:36.810527 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:36.810197 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7c48k" event={"ID":"96672df6-a63f-4313-b703-67469597b7e7","Type":"ContainerDied","Data":"d459cecd753f1524e761dc060c69ff50a9bd9a77b5666671e4a2236cf76207b2"} Oct 31 01:22:36.810527 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:36.810252 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7c48k" event={"ID":"96672df6-a63f-4313-b703-67469597b7e7","Type":"ContainerStarted","Data":"e3e47d49d1359f83e2bf095d1bca3f2ff85040a02cb82acb6a8f44fd11bdbb57"} Oct 31 01:22:37.953885 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:37.953853 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7c48k" Oct 31 01:22:38.062967 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:38.062926 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96672df6-a63f-4313-b703-67469597b7e7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96672df6-a63f-4313-b703-67469597b7e7-maas-api-openshift-ca\") pod \"96672df6-a63f-4313-b703-67469597b7e7\" (UID: \"96672df6-a63f-4313-b703-67469597b7e7\") " Oct 31 01:22:38.063181 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:38.063029 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96672df6-a63f-4313-b703-67469597b7e7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96672df6-a63f-4313-b703-67469597b7e7-maas-api-xks-ca\") pod \"96672df6-a63f-4313-b703-67469597b7e7\" (UID: \"96672df6-a63f-4313-b703-67469597b7e7\") " Oct 31 01:22:38.063181 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:38.063061 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96672df6-a63f-4313-b703-67469597b7e7-kube-api-access-hjh5d\" (UniqueName: \"kubernetes.io/projected/96672df6-a63f-4313-b703-67469597b7e7-kube-api-access-hjh5d\") pod \"96672df6-a63f-4313-b703-67469597b7e7\" (UID: \"96672df6-a63f-4313-b703-67469597b7e7\") " Oct 31 01:22:38.063363 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:38.063332 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96672df6-a63f-4313-b703-67469597b7e7-maas-api-openshift-ca" pod "96672df6-a63f-4313-b703-67469597b7e7" (UID: "96672df6-a63f-4313-b703-67469597b7e7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:22:38.063496 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:38.063476 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96672df6-a63f-4313-b703-67469597b7e7-maas-api-openshift-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:22:38.065337 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:38.065283 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96672df6-a63f-4313-b703-67469597b7e7-maas-api-xks-ca" pod "96672df6-a63f-4313-b703-67469597b7e7" (UID: "96672df6-a63f-4313-b703-67469597b7e7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:22:38.065426 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:38.065383 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96672df6-a63f-4313-b703-67469597b7e7-kube-api-access-hjh5d" pod "96672df6-a63f-4313-b703-67469597b7e7" (UID: "96672df6-a63f-4313-b703-67469597b7e7"). InnerVolumeSpecName "kube-api-access-hjh5d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:22:38.164639 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:38.164600 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96672df6-a63f-4313-b703-67469597b7e7-maas-api-xks-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:22:38.164639 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:38.164633 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjh5d\" (UniqueName: \"kubernetes.io/projected/96672df6-a63f-4313-b703-67469597b7e7-kube-api-access-hjh5d\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:22:38.820308 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:38.820272 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7c48k" Oct 31 01:22:38.820536 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:38.820307 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7c48k" event={"ID":"96672df6-a63f-4313-b703-67469597b7e7","Type":"ContainerDied","Data":"e3e47d49d1359f83e2bf095d1bca3f2ff85040a02cb82acb6a8f44fd11bdbb57"} Oct 31 01:22:38.820536 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:38.820362 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e3e47d49d1359f83e2bf095d1bca3f2ff85040a02cb82acb6a8f44fd11bdbb57" Oct 31 01:22:47.090530 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:47.090441 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-pkzv9"] Oct 31 01:22:47.091101 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:47.090872 5222 cpu_manager.go:414] "containerMap: removing container" podUID="96672df6-a63f-4313-b703-67469597b7e7" containerName="revoke-keys" Oct 31 01:22:47.091101 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:47.090885 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96672df6-a63f-4313-b703-67469597b7e7" containerName="revoke-keys" Oct 31 01:22:47.091101 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:47.090970 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="96672df6-a63f-4313-b703-67469597b7e7" containerName="revoke-keys" Oct 31 01:22:47.094130 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:47.094111 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-pkzv9" Oct 31 01:22:47.148794 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:47.148758 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-pkzv9"] Oct 31 01:22:47.198399 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:47.198291 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:22:47.198399 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:47.198319 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:22:47.198894 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:47.198875 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:22:47.198934 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:47.198875 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-29cjc\"" Oct 31 01:22:47.255974 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:47.255935 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfb397c7-da2c-4b99-83ef-d3f9c819f089-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-pkzv9\" (UID: \"cfb397c7-da2c-4b99-83ef-d3f9c819f089\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-pkzv9" Oct 31 01:22:47.256201 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:47.256062 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9rl6\" (UniqueName: \"kubernetes.io/projected/cfb397c7-da2c-4b99-83ef-d3f9c819f089-kube-api-access-x9rl6\") pod \"maas-api-revoke-sub-6b19338f-pkzv9\" (UID: \"cfb397c7-da2c-4b99-83ef-d3f9c819f089\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-pkzv9" Oct 31 01:22:47.256201 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:47.256129 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfb397c7-da2c-4b99-83ef-d3f9c819f089-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-pkzv9\" (UID: \"cfb397c7-da2c-4b99-83ef-d3f9c819f089\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-pkzv9" Oct 31 01:22:47.357764 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:47.357669 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfb397c7-da2c-4b99-83ef-d3f9c819f089-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-pkzv9\" (UID: \"cfb397c7-da2c-4b99-83ef-d3f9c819f089\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-pkzv9" Oct 31 01:22:47.357764 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:47.357739 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9rl6\" (UniqueName: \"kubernetes.io/projected/cfb397c7-da2c-4b99-83ef-d3f9c819f089-kube-api-access-x9rl6\") pod \"maas-api-revoke-sub-6b19338f-pkzv9\" (UID: \"cfb397c7-da2c-4b99-83ef-d3f9c819f089\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-pkzv9" Oct 31 01:22:47.358019 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:47.357790 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfb397c7-da2c-4b99-83ef-d3f9c819f089-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-pkzv9\" (UID: \"cfb397c7-da2c-4b99-83ef-d3f9c819f089\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-pkzv9" Oct 31 01:22:47.358533 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:47.358505 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfb397c7-da2c-4b99-83ef-d3f9c819f089-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-pkzv9\" (UID: \"cfb397c7-da2c-4b99-83ef-d3f9c819f089\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-pkzv9" Oct 31 01:22:47.360201 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:47.360174 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfb397c7-da2c-4b99-83ef-d3f9c819f089-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-pkzv9\" (UID: \"cfb397c7-da2c-4b99-83ef-d3f9c819f089\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-pkzv9" Oct 31 01:22:47.420223 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:47.420186 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9rl6\" (UniqueName: \"kubernetes.io/projected/cfb397c7-da2c-4b99-83ef-d3f9c819f089-kube-api-access-x9rl6\") pod \"maas-api-revoke-sub-6b19338f-pkzv9\" (UID: \"cfb397c7-da2c-4b99-83ef-d3f9c819f089\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-pkzv9" Oct 31 01:22:47.708357 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:47.708259 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-pkzv9" Oct 31 01:22:48.080302 ip-10-0-137-56 kubenswrapper[5222]: W1031 01:22:48.080269 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfb397c7_da2c_4b99_83ef_d3f9c819f089.slice/crio-32db6d9742b11feaab1f974a4844eb8475c02d6ceec69b5103f77fd1f38e3d78 WatchSource:0}: Error finding container 32db6d9742b11feaab1f974a4844eb8475c02d6ceec69b5103f77fd1f38e3d78: Status 404 returned error can't find the container with id 32db6d9742b11feaab1f974a4844eb8475c02d6ceec69b5103f77fd1f38e3d78 Oct 31 01:22:48.082248 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:48.082226 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-pkzv9"] Oct 31 01:22:48.868133 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:48.868091 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-pkzv9" event={"ID":"cfb397c7-da2c-4b99-83ef-d3f9c819f089","Type":"ContainerStarted","Data":"52b74685a79d0423404fb28e3d023fcf5e4e07bff7970d015e14f9ceb2701a19"} Oct 31 01:22:48.868605 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:48.868154 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-pkzv9" event={"ID":"cfb397c7-da2c-4b99-83ef-d3f9c819f089","Type":"ContainerStarted","Data":"32db6d9742b11feaab1f974a4844eb8475c02d6ceec69b5103f77fd1f38e3d78"} Oct 31 01:22:48.988080 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:48.988023 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-pkzv9" podStartSLOduration=1.98800827 podStartE2EDuration="1.98800827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:22:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:22:48.986824263 +0000 UTC m=+4061.457994011" watchObservedRunningTime="2026-10-31 01:22:48.98800827 +0000 UTC m=+4061.459178061" Oct 31 01:22:49.874278 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:49.874236 5222 generic.go:356] "Generic (PLEG): container finished" podID="cfb397c7-da2c-4b99-83ef-d3f9c819f089" containerID="52b74685a79d0423404fb28e3d023fcf5e4e07bff7970d015e14f9ceb2701a19" exitCode=0 Oct 31 01:22:49.874714 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:49.874325 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-pkzv9" event={"ID":"cfb397c7-da2c-4b99-83ef-d3f9c819f089","Type":"ContainerDied","Data":"52b74685a79d0423404fb28e3d023fcf5e4e07bff7970d015e14f9ceb2701a19"} Oct 31 01:22:51.011975 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:51.011944 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-pkzv9" Oct 31 01:22:51.094493 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:51.094449 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfb397c7-da2c-4b99-83ef-d3f9c819f089-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfb397c7-da2c-4b99-83ef-d3f9c819f089-maas-api-openshift-ca\") pod \"cfb397c7-da2c-4b99-83ef-d3f9c819f089\" (UID: \"cfb397c7-da2c-4b99-83ef-d3f9c819f089\") " Oct 31 01:22:51.094683 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:51.094519 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cfb397c7-da2c-4b99-83ef-d3f9c819f089-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfb397c7-da2c-4b99-83ef-d3f9c819f089-maas-api-xks-ca\") pod \"cfb397c7-da2c-4b99-83ef-d3f9c819f089\" (UID: \"cfb397c7-da2c-4b99-83ef-d3f9c819f089\") " Oct 31 01:22:51.094683 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:51.094576 5222 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfb397c7-da2c-4b99-83ef-d3f9c819f089-kube-api-access-x9rl6\" (UniqueName: \"kubernetes.io/projected/cfb397c7-da2c-4b99-83ef-d3f9c819f089-kube-api-access-x9rl6\") pod \"cfb397c7-da2c-4b99-83ef-d3f9c819f089\" (UID: \"cfb397c7-da2c-4b99-83ef-d3f9c819f089\") " Oct 31 01:22:51.094897 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:51.094872 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfb397c7-da2c-4b99-83ef-d3f9c819f089-maas-api-openshift-ca" pod "cfb397c7-da2c-4b99-83ef-d3f9c819f089" (UID: "cfb397c7-da2c-4b99-83ef-d3f9c819f089"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:22:51.096813 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:51.096784 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfb397c7-da2c-4b99-83ef-d3f9c819f089-maas-api-xks-ca" pod "cfb397c7-da2c-4b99-83ef-d3f9c819f089" (UID: "cfb397c7-da2c-4b99-83ef-d3f9c819f089"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:22:51.096940 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:51.096913 5222 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfb397c7-da2c-4b99-83ef-d3f9c819f089-kube-api-access-x9rl6" pod "cfb397c7-da2c-4b99-83ef-d3f9c819f089" (UID: "cfb397c7-da2c-4b99-83ef-d3f9c819f089"). InnerVolumeSpecName "kube-api-access-x9rl6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:22:51.196205 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:51.196108 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfb397c7-da2c-4b99-83ef-d3f9c819f089-maas-api-openshift-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:22:51.196205 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:51.196144 5222 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfb397c7-da2c-4b99-83ef-d3f9c819f089-maas-api-xks-ca\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:22:51.196205 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:51.196154 5222 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9rl6\" (UniqueName: \"kubernetes.io/projected/cfb397c7-da2c-4b99-83ef-d3f9c819f089-kube-api-access-x9rl6\") on node \"ip-10-0-137-56.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:22:51.884352 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:51.884319 5222 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-pkzv9" Oct 31 01:22:51.884556 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:51.884355 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-pkzv9" event={"ID":"cfb397c7-da2c-4b99-83ef-d3f9c819f089","Type":"ContainerDied","Data":"32db6d9742b11feaab1f974a4844eb8475c02d6ceec69b5103f77fd1f38e3d78"} Oct 31 01:22:51.884556 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:22:51.884420 5222 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="32db6d9742b11feaab1f974a4844eb8475c02d6ceec69b5103f77fd1f38e3d78" Oct 31 01:23:00.022338 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:23:00.022297 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-79tk8"] Oct 31 01:23:03.049675 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:23:03.049635 5222 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9f6da73-kpj95"] Oct 31 01:23:03.100792 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:23:03.100747 5222 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9f6da73-kpj95"] Oct 31 01:23:03.654968 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:23:03.654929 5222 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e361b6c-3b9a-4871-b138-28d6c7c5f47e" path="/var/lib/kubelet/pods/3e361b6c-3b9a-4871-b138-28d6c7c5f47e/volumes" Oct 31 01:23:05.889301 ip-10-0-137-56 kubenswrapper[5222]: E1031 01:23:05.889260 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 01:23:11.135711 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:23:11.135677 5222 scope.go:122] "RemoveContainer" containerID="d03e6a4e6c17e79109e6686ba35e494b0de851a5b66d0a3afe147df3f366a2fb" Oct 31 01:23:12.861112 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:23:12.861077 5222 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-cc757_c31f203d-f391-446f-82c2-76dd9697bd43/manager/0.log" Oct 31 01:23:14.509926 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:23:14.509895 5222 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-7c48k_96672df6-a63f-4313-b703-67469597b7e7/revoke-keys/0.log" Oct 31 01:23:14.735915 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:23:14.735880 5222 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-pkzv9_cfb397c7-da2c-4b99-83ef-d3f9c819f089/revoke-keys/0.log" Oct 31 01:23:14.949364 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:23:14.949330 5222 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-wrnxz_99c2a864-cb2f-4169-8852-d443a7d102cf/revoke-keys/0.log" Oct 31 01:23:18.239144 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:23:18.239114 5222 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-79tk8_66c5518e-c6af-4298-aa66-5e1b6d6bc645/limitador/0.log" Oct 31 01:23:19.572874 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:23:19.572840 5222 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-8d22z_e7f7ac90-6d52-40a2-873c-b878687b02ca/discovery/0.log" Oct 31 01:23:20.070885 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:23:20.070853 5222 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-66d4777458-wxv4x_a3d36c34-a283-4b19-8ac9-754fe86ff7d3/kube-auth-proxy/0.log" Oct 31 01:23:20.313406 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:23:20.313360 5222 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-256br_a723d3c6-4cd4-4e23-bf6b-6f2dfb4a38f4/istio-proxy/0.log" Oct 31 01:23:21.407743 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:23:21.407710 5222 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-jd2w5_d2ec2aa7-9ec8-41bd-b10f-e5ef7ecbd803/istio-proxy/0.log" Oct 31 01:23:21.934063 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:23:21.934032 5222 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-qbn7v_7304a135-5e25-401d-b3cc-4503f1c0a794/istio-proxy/0.log" Oct 31 01:23:22.687015 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:23:22.686980 5222 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-zq2lm_3d1be84f-73f0-4f54-ae9f-6d5f38367862/istio-proxy/0.log" Oct 31 01:23:26.185722 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:23:26.185691 5222 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-4hfsn_00f550bd-1442-4662-bdc3-5714f0afda9d/main/0.log" Oct 31 01:23:26.415174 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:23:26.415139 5222 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-nvtsz_c357cb32-7e30-442c-b612-f46e4a7dbc5a/main/0.log" Oct 31 01:23:35.890300 ip-10-0-137-56 kubenswrapper[5222]: E1031 01:23:35.890260 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 01:23:39.079479 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:23:39.079444 5222 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-cc757_c31f203d-f391-446f-82c2-76dd9697bd43/manager/0.log" Oct 31 01:23:40.517679 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:23:40.517649 5222 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-7c48k_96672df6-a63f-4313-b703-67469597b7e7/revoke-keys/0.log" Oct 31 01:23:40.748950 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:23:40.748922 5222 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-pkzv9_cfb397c7-da2c-4b99-83ef-d3f9c819f089/revoke-keys/0.log" Oct 31 01:23:40.981887 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:23:40.981807 5222 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-wrnxz_99c2a864-cb2f-4169-8852-d443a7d102cf/revoke-keys/0.log" Oct 31 01:23:43.944799 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:23:43.944771 5222 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-79tk8_66c5518e-c6af-4298-aa66-5e1b6d6bc645/limitador/0.log" Oct 31 01:23:45.019797 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:23:45.019767 5222 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-8d22z_e7f7ac90-6d52-40a2-873c-b878687b02ca/discovery/0.log" Oct 31 01:23:45.487985 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:23:45.487905 5222 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-66d4777458-wxv4x_a3d36c34-a283-4b19-8ac9-754fe86ff7d3/kube-auth-proxy/0.log" Oct 31 01:23:45.713344 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:23:45.713311 5222 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-256br_a723d3c6-4cd4-4e23-bf6b-6f2dfb4a38f4/istio-proxy/0.log" Oct 31 01:23:46.792126 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:23:46.792093 5222 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-jd2w5_d2ec2aa7-9ec8-41bd-b10f-e5ef7ecbd803/istio-proxy/0.log" Oct 31 01:23:47.319534 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:23:47.319502 5222 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-qbn7v_7304a135-5e25-401d-b3cc-4503f1c0a794/istio-proxy/0.log" Oct 31 01:23:48.017074 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:23:48.017039 5222 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-zq2lm_3d1be84f-73f0-4f54-ae9f-6d5f38367862/istio-proxy/0.log" Oct 31 01:23:51.030596 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:23:51.030568 5222 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-4hfsn_00f550bd-1442-4662-bdc3-5714f0afda9d/main/0.log" Oct 31 01:23:51.265744 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:23:51.265703 5222 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-nvtsz_c357cb32-7e30-442c-b612-f46e4a7dbc5a/main/0.log" Oct 31 01:24:05.889243 ip-10-0-137-56 kubenswrapper[5222]: E1031 01:24:05.889199 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 01:24:11.157278 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:24:11.157238 5222 scope.go:122] "RemoveContainer" containerID="07c35875ca71145f6649993960092fc81fe82d67934a256a21db08fff88b15b6" Oct 31 01:24:11.165887 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:24:11.165867 5222 scope.go:122] "RemoveContainer" containerID="8f63015e21b711d05755ff16a9ffab6cae70684cd725d8e87e9ce1eea7c7fdd4" Oct 31 01:24:35.889307 ip-10-0-137-56 kubenswrapper[5222]: E1031 01:24:35.889213 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 01:25:04.469795 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:25:04.469760 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-bw6xw/must-gather-gqck6"] Oct 31 01:25:04.471954 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:25:04.470138 5222 cpu_manager.go:414] "containerMap: removing container" podUID="cfb397c7-da2c-4b99-83ef-d3f9c819f089" containerName="revoke-keys" Oct 31 01:25:04.471954 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:25:04.470149 5222 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfb397c7-da2c-4b99-83ef-d3f9c819f089" containerName="revoke-keys" Oct 31 01:25:04.471954 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:25:04.470241 5222 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfb397c7-da2c-4b99-83ef-d3f9c819f089" containerName="revoke-keys" Oct 31 01:25:04.472999 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:25:04.472980 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bw6xw/must-gather-gqck6" Oct 31 01:25:04.547598 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:25:04.547565 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/90d76055-d94e-49cc-8c71-0cc9c41ebbb9-must-gather-output\") pod \"must-gather-gqck6\" (UID: \"90d76055-d94e-49cc-8c71-0cc9c41ebbb9\") " pod="openshift-must-gather-bw6xw/must-gather-gqck6" Oct 31 01:25:04.547797 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:25:04.547623 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lzxc\" (UniqueName: \"kubernetes.io/projected/90d76055-d94e-49cc-8c71-0cc9c41ebbb9-kube-api-access-6lzxc\") pod \"must-gather-gqck6\" (UID: \"90d76055-d94e-49cc-8c71-0cc9c41ebbb9\") " pod="openshift-must-gather-bw6xw/must-gather-gqck6" Oct 31 01:25:04.581282 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:25:04.581239 5222 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-bw6xw\"/\"default-dockercfg-vkhnz\"" Oct 31 01:25:04.581282 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:25:04.581239 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bw6xw\"/\"kube-root-ca.crt\"" Oct 31 01:25:04.581282 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:25:04.581239 5222 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bw6xw\"/\"openshift-service-ca.crt\"" Oct 31 01:25:04.583034 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:25:04.583016 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bw6xw/must-gather-gqck6"] Oct 31 01:25:04.649267 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:25:04.649221 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lzxc\" (UniqueName: \"kubernetes.io/projected/90d76055-d94e-49cc-8c71-0cc9c41ebbb9-kube-api-access-6lzxc\") pod \"must-gather-gqck6\" (UID: \"90d76055-d94e-49cc-8c71-0cc9c41ebbb9\") " pod="openshift-must-gather-bw6xw/must-gather-gqck6" Oct 31 01:25:04.649494 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:25:04.649339 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/90d76055-d94e-49cc-8c71-0cc9c41ebbb9-must-gather-output\") pod \"must-gather-gqck6\" (UID: \"90d76055-d94e-49cc-8c71-0cc9c41ebbb9\") " pod="openshift-must-gather-bw6xw/must-gather-gqck6" Oct 31 01:25:04.649754 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:25:04.649731 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/90d76055-d94e-49cc-8c71-0cc9c41ebbb9-must-gather-output\") pod \"must-gather-gqck6\" (UID: \"90d76055-d94e-49cc-8c71-0cc9c41ebbb9\") " pod="openshift-must-gather-bw6xw/must-gather-gqck6" Oct 31 01:25:04.714965 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:25:04.714921 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lzxc\" (UniqueName: \"kubernetes.io/projected/90d76055-d94e-49cc-8c71-0cc9c41ebbb9-kube-api-access-6lzxc\") pod \"must-gather-gqck6\" (UID: \"90d76055-d94e-49cc-8c71-0cc9c41ebbb9\") " pod="openshift-must-gather-bw6xw/must-gather-gqck6" Oct 31 01:25:04.786975 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:25:04.786918 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bw6xw/must-gather-gqck6" Oct 31 01:25:05.158178 ip-10-0-137-56 kubenswrapper[5222]: W1031 01:25:05.158147 5222 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod90d76055_d94e_49cc_8c71_0cc9c41ebbb9.slice/crio-e57e657396d42c888b124dc2adb421899444ae5a26ea2e55865ab589b8261c2c WatchSource:0}: Error finding container e57e657396d42c888b124dc2adb421899444ae5a26ea2e55865ab589b8261c2c: Status 404 returned error can't find the container with id e57e657396d42c888b124dc2adb421899444ae5a26ea2e55865ab589b8261c2c Oct 31 01:25:05.159640 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:25:05.159617 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bw6xw/must-gather-gqck6"] Oct 31 01:25:05.160660 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:25:05.160641 5222 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 01:25:05.554332 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:25:05.554294 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bw6xw/must-gather-gqck6" event={"ID":"90d76055-d94e-49cc-8c71-0cc9c41ebbb9","Type":"ContainerStarted","Data":"e57e657396d42c888b124dc2adb421899444ae5a26ea2e55865ab589b8261c2c"} Oct 31 01:25:05.890436 ip-10-0-137-56 kubenswrapper[5222]: E1031 01:25:05.890332 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 01:25:07.565524 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:25:07.565487 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bw6xw/must-gather-gqck6" event={"ID":"90d76055-d94e-49cc-8c71-0cc9c41ebbb9","Type":"ContainerStarted","Data":"de50102372f803a533cdbddf55cdafb40f7aeeadab817ef33febaaa7dd810f8c"} Oct 31 01:25:07.565970 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:25:07.565535 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bw6xw/must-gather-gqck6" event={"ID":"90d76055-d94e-49cc-8c71-0cc9c41ebbb9","Type":"ContainerStarted","Data":"db89264d6af028ffa5052f93e7db638c4ad114a13f669e7749b5a497bfc7dbbb"} Oct 31 01:25:07.686128 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:25:07.685180 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-bw6xw/must-gather-gqck6" podStartSLOduration=2.084354263 podStartE2EDuration="3.685164356s" totalImagesPullingTime="1.600810093s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:25:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:25:07.684202772 +0000 UTC m=+4200.155372522" watchObservedRunningTime="2026-10-31 01:25:07.685164356 +0000 UTC m=+4200.156334104" Oct 31 01:25:08.377674 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:25:08.377646 5222 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h9hsf_0f8c3c03-a1fc-4376-98c6-feeeefa08068/ovn-acl-logging/0.log" Oct 31 01:25:08.411193 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:25:08.411151 5222 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h9hsf_0f8c3c03-a1fc-4376-98c6-feeeefa08068/ovn-acl-logging/0.log" Oct 31 01:25:11.188831 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:25:11.188798 5222 scope.go:122] "RemoveContainer" containerID="415519ba45e228b6bda7f80eecb4795c4f3ef12669e6ea9338260a7148864d1c" Oct 31 01:25:14.912461 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:25:14.911845 5222 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-fwgtx_d02ae6b5-4860-45b3-8d16-4a49531b77d4/global-pull-secret-syncer/0.log" Oct 31 01:25:18.030274 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:25:18.030232 5222 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-lvp2h_2ed1e615-cbca-482c-89cc-0fe7d8e7e597/konnectivity-agent/0.log" Oct 31 01:25:19.271837 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:25:19.271760 5222 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-56.us-west-2.compute.internal_e15cd53a0cf50b999ecad893fad25259/haproxy/0.log" Oct 31 01:25:24.121653 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:25:24.121615 5222 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-bw6xw/perf-node-gather-daemonset-hns5q"] Oct 31 01:25:24.127260 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:25:24.127232 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bw6xw/perf-node-gather-daemonset-hns5q" Oct 31 01:25:24.206765 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:25:24.206728 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bw6xw/perf-node-gather-daemonset-hns5q"] Oct 31 01:25:24.272745 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:25:24.272707 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/52908e61-4518-42de-9833-ef0faa4ee361-sys\") pod \"perf-node-gather-daemonset-hns5q\" (UID: \"52908e61-4518-42de-9833-ef0faa4ee361\") " pod="openshift-must-gather-bw6xw/perf-node-gather-daemonset-hns5q" Oct 31 01:25:24.272745 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:25:24.272750 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/52908e61-4518-42de-9833-ef0faa4ee361-podres\") pod \"perf-node-gather-daemonset-hns5q\" (UID: \"52908e61-4518-42de-9833-ef0faa4ee361\") " pod="openshift-must-gather-bw6xw/perf-node-gather-daemonset-hns5q" Oct 31 01:25:24.273008 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:25:24.272776 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/52908e61-4518-42de-9833-ef0faa4ee361-lib-modules\") pod \"perf-node-gather-daemonset-hns5q\" (UID: \"52908e61-4518-42de-9833-ef0faa4ee361\") " pod="openshift-must-gather-bw6xw/perf-node-gather-daemonset-hns5q" Oct 31 01:25:24.273008 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:25:24.272902 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7cxv\" (UniqueName: \"kubernetes.io/projected/52908e61-4518-42de-9833-ef0faa4ee361-kube-api-access-r7cxv\") pod \"perf-node-gather-daemonset-hns5q\" (UID: \"52908e61-4518-42de-9833-ef0faa4ee361\") " pod="openshift-must-gather-bw6xw/perf-node-gather-daemonset-hns5q" Oct 31 01:25:24.273111 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:25:24.273018 5222 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/52908e61-4518-42de-9833-ef0faa4ee361-proc\") pod \"perf-node-gather-daemonset-hns5q\" (UID: \"52908e61-4518-42de-9833-ef0faa4ee361\") " pod="openshift-must-gather-bw6xw/perf-node-gather-daemonset-hns5q" Oct 31 01:25:24.374298 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:25:24.374216 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/52908e61-4518-42de-9833-ef0faa4ee361-sys\") pod \"perf-node-gather-daemonset-hns5q\" (UID: \"52908e61-4518-42de-9833-ef0faa4ee361\") " pod="openshift-must-gather-bw6xw/perf-node-gather-daemonset-hns5q" Oct 31 01:25:24.374298 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:25:24.374261 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/52908e61-4518-42de-9833-ef0faa4ee361-podres\") pod \"perf-node-gather-daemonset-hns5q\" (UID: \"52908e61-4518-42de-9833-ef0faa4ee361\") " pod="openshift-must-gather-bw6xw/perf-node-gather-daemonset-hns5q" Oct 31 01:25:24.374538 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:25:24.374344 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/52908e61-4518-42de-9833-ef0faa4ee361-sys\") pod \"perf-node-gather-daemonset-hns5q\" (UID: \"52908e61-4518-42de-9833-ef0faa4ee361\") " pod="openshift-must-gather-bw6xw/perf-node-gather-daemonset-hns5q" Oct 31 01:25:24.374538 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:25:24.374404 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/52908e61-4518-42de-9833-ef0faa4ee361-podres\") pod \"perf-node-gather-daemonset-hns5q\" (UID: \"52908e61-4518-42de-9833-ef0faa4ee361\") " pod="openshift-must-gather-bw6xw/perf-node-gather-daemonset-hns5q" Oct 31 01:25:24.374538 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:25:24.374432 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/52908e61-4518-42de-9833-ef0faa4ee361-lib-modules\") pod \"perf-node-gather-daemonset-hns5q\" (UID: \"52908e61-4518-42de-9833-ef0faa4ee361\") " pod="openshift-must-gather-bw6xw/perf-node-gather-daemonset-hns5q" Oct 31 01:25:24.374538 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:25:24.374467 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7cxv\" (UniqueName: \"kubernetes.io/projected/52908e61-4518-42de-9833-ef0faa4ee361-kube-api-access-r7cxv\") pod \"perf-node-gather-daemonset-hns5q\" (UID: \"52908e61-4518-42de-9833-ef0faa4ee361\") " pod="openshift-must-gather-bw6xw/perf-node-gather-daemonset-hns5q" Oct 31 01:25:24.374696 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:25:24.374540 5222 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/52908e61-4518-42de-9833-ef0faa4ee361-proc\") pod \"perf-node-gather-daemonset-hns5q\" (UID: \"52908e61-4518-42de-9833-ef0faa4ee361\") " pod="openshift-must-gather-bw6xw/perf-node-gather-daemonset-hns5q" Oct 31 01:25:24.374696 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:25:24.374552 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/52908e61-4518-42de-9833-ef0faa4ee361-lib-modules\") pod \"perf-node-gather-daemonset-hns5q\" (UID: \"52908e61-4518-42de-9833-ef0faa4ee361\") " pod="openshift-must-gather-bw6xw/perf-node-gather-daemonset-hns5q" Oct 31 01:25:24.374696 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:25:24.374678 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/52908e61-4518-42de-9833-ef0faa4ee361-proc\") pod \"perf-node-gather-daemonset-hns5q\" (UID: \"52908e61-4518-42de-9833-ef0faa4ee361\") " pod="openshift-must-gather-bw6xw/perf-node-gather-daemonset-hns5q" Oct 31 01:25:24.437662 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:25:24.437626 5222 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7cxv\" (UniqueName: \"kubernetes.io/projected/52908e61-4518-42de-9833-ef0faa4ee361-kube-api-access-r7cxv\") pod \"perf-node-gather-daemonset-hns5q\" (UID: \"52908e61-4518-42de-9833-ef0faa4ee361\") " pod="openshift-must-gather-bw6xw/perf-node-gather-daemonset-hns5q" Oct 31 01:25:24.438514 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:25:24.438495 5222 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bw6xw/perf-node-gather-daemonset-hns5q" Oct 31 01:25:24.822436 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:25:24.819804 5222 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bw6xw/perf-node-gather-daemonset-hns5q"] Oct 31 01:25:25.634187 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:25:25.634156 5222 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-79tk8_66c5518e-c6af-4298-aa66-5e1b6d6bc645/limitador/0.log" Oct 31 01:25:25.713807 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:25:25.713776 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bw6xw/perf-node-gather-daemonset-hns5q" event={"ID":"52908e61-4518-42de-9833-ef0faa4ee361","Type":"ContainerStarted","Data":"2a1557839ba9ca9337999a0ae79691422f04d1c66ad3dfaa6f3e39d208d3e07b"} Oct 31 01:25:25.714025 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:25:25.713830 5222 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bw6xw/perf-node-gather-daemonset-hns5q" event={"ID":"52908e61-4518-42de-9833-ef0faa4ee361","Type":"ContainerStarted","Data":"65067fd328833c64a465b1afd9588169986e8c3a748f099cb61690d580037c96"} Oct 31 01:25:25.714025 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:25:25.713899 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-bw6xw/perf-node-gather-daemonset-hns5q" Oct 31 01:25:25.833727 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:25:25.833669 5222 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-bw6xw/perf-node-gather-daemonset-hns5q" podStartSLOduration=1.833654705 podStartE2EDuration="1.833654705s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 01:25:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:25:25.831797054 +0000 UTC m=+4218.302966800" watchObservedRunningTime="2026-10-31 01:25:25.833654705 +0000 UTC m=+4218.304824432" Oct 31 01:25:28.750578 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:25:28.750547 5222 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_65985e4d-b22b-4392-a735-6353b05d9f77/alertmanager/0.log" Oct 31 01:25:29.164608 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:25:29.164578 5222 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_65985e4d-b22b-4392-a735-6353b05d9f77/config-reloader/0.log" Oct 31 01:25:29.580484 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:25:29.580454 5222 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_65985e4d-b22b-4392-a735-6353b05d9f77/kube-rbac-proxy-web/0.log" Oct 31 01:25:29.995486 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:25:29.995409 5222 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_65985e4d-b22b-4392-a735-6353b05d9f77/kube-rbac-proxy/0.log" Oct 31 01:25:30.410521 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:25:30.410492 5222 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_65985e4d-b22b-4392-a735-6353b05d9f77/kube-rbac-proxy-metric/0.log" Oct 31 01:25:30.827003 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:25:30.826975 5222 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_65985e4d-b22b-4392-a735-6353b05d9f77/prom-label-proxy/0.log" Oct 31 01:25:31.242183 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:25:31.242097 5222 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_65985e4d-b22b-4392-a735-6353b05d9f77/init-config-reloader/0.log" Oct 31 01:25:31.729334 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:25:31.729304 5222 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-bw6xw/perf-node-gather-daemonset-hns5q" Oct 31 01:25:32.090803 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:25:32.090764 5222 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-rsf6d_ea9a5160-4696-4b75-8a10-0fbc54b2ad04/kube-state-metrics/0.log" Oct 31 01:25:32.503109 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:25:32.503029 5222 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-rsf6d_ea9a5160-4696-4b75-8a10-0fbc54b2ad04/kube-rbac-proxy-main/0.log" Oct 31 01:25:32.920204 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:25:32.920176 5222 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-rsf6d_ea9a5160-4696-4b75-8a10-0fbc54b2ad04/kube-rbac-proxy-self/0.log" Oct 31 01:25:33.339669 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:25:33.339643 5222 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-68dfc74784-dx6d6_2cc453ef-8580-4745-b57b-03f7516056a7/metrics-server/0.log" Oct 31 01:25:34.193983 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:25:34.193944 5222 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dbcsc_1b88d3b9-589f-4f4e-b8a6-872f4fb5536b/node-exporter/0.log" Oct 31 01:25:34.607585 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:25:34.607557 5222 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dbcsc_1b88d3b9-589f-4f4e-b8a6-872f4fb5536b/kube-rbac-proxy/0.log" Oct 31 01:25:35.033681 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:25:35.033650 5222 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dbcsc_1b88d3b9-589f-4f4e-b8a6-872f4fb5536b/init-textfile/0.log" Oct 31 01:25:35.890196 ip-10-0-137-56 kubenswrapper[5222]: E1031 01:25:35.890161 5222 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nq56p" containerName="extractor" Oct 31 01:25:38.073824 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:25:38.073798 5222 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-xd6zl_652defc0-301f-4679-8e49-6729381259dd/kube-rbac-proxy-main/0.log" Oct 31 01:25:38.489048 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:25:38.488976 5222 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-xd6zl_652defc0-301f-4679-8e49-6729381259dd/kube-rbac-proxy-self/0.log" Oct 31 01:25:38.904620 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:25:38.904579 5222 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-xd6zl_652defc0-301f-4679-8e49-6729381259dd/openshift-state-metrics/0.log" Oct 31 01:25:43.772123 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:25:43.772092 5222 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5bc6f9fd94-d2fk4_f7a951f0-22de-4adc-94d5-5bd1962e545d/telemeter-client/0.log" Oct 31 01:25:44.187921 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:25:44.187834 5222 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5bc6f9fd94-d2fk4_f7a951f0-22de-4adc-94d5-5bd1962e545d/reload/0.log" Oct 31 01:25:44.602325 ip-10-0-137-56 kubenswrapper[5222]: I1031 01:25:44.602285 5222 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5bc6f9fd94-d2fk4_f7a951f0-22de-4adc-94d5-5bd1962e545d/kube-rbac-proxy/0.log"